Toward quality discourse: Measuring the effect of user identity in commenting forums

Abstract

Via a content analysis of 4,800 comments from online commenting forums of top news sites, this research examines the overall quality of the comments. Expanding the scope of previous research in this area and guided by the theory of deliberative democracy, the normative conditions for quality discourse were measured with six parameters: civility, reciprocity, reflexivity, rationality, diversity, and relevance. In measuring the quality of the comments, two conditions were the identity of the commenter.

Keywords

anonymity deliberative democracy news sites online commenting forums quality dialogue

The arrival of online commenting forums on news sites about 15 years ago was generally met with uncertainty by newsrooms wary of giving the public an unrestrained platform for anonymous discussion. Researchers similarly questioned the usefulness of the forums’ discourse in advancing productive discussion as many commenters appeared to demonstrate a lack of respectful listening (Dahlberg, 2001) in spaces rife with incivility (Davis, 1999). Despite its turbulent beginnings, commenting forums on news sites endure, albeit with an uncertain future. This research continues the ongoing examination of the utility of the forums and builds on existing literature in this area as newsrooms continue to struggle with raising the quality of the dialogue.

Research shows that 35% of Americans read news comments, and 14% have commented on the news (Stroud, van Duyn, & Peacock, 2015), providing proof that commenting forums continue to be worthy of study. Overall, the effects of such forums on the public are well-documented. Comments from readers, for example, can skew the perception of a story (Anderson, Brossard, Scheufele, Xenos, & Ladwig, 2014; Prochazka, Weber, & Schweiger, 2018), alter the credibility of the news story (Conlin & Roberts, 2016; Thorson, Vraga, & Ekdale, 2010), influence prejudiced thoughts (Hsueh, Yogeeswaran, & Malinen, 2015), and lead to a distorted view of public opinion (Lee, 2012). Journalists at news organizations that host commenting forums generally dislike user anonymity (Ksiazek, 2016), yet value virtual conversations about the news if they embody civil discussion and provide news tips (Graham & Wright, 2015; Nielsen, 2012; Santana, 2011) even as they believe user-generated content can undermine journalistic norms and values (Prochazka et al., 2018; Singer, 2010; Usher, 2017). One key change to forums in recent years has been disallowing anonymity, usually by asking users to log in to a third-party platform such as Facebook.

Indeed, as a key characteristic of any online experience, anonymity has been addressed in research for its myriad effects in online communication (Berg, 2016; Boczkowski, 1999; Burkell, 2006; Dubrovsky, Kiesler, & Sethna, 1991; Fredheim, Moore, & Naughton, 2015; Graf, Erba, & Harn, 2017; Hardaker, 2010; Nielsen, 2014; Nissenbaum, 1999; Reader, 2012; Santana, 2013; Singer, 1996; Sproull & Kiesler, 1986; Suler, 2004). The differing types of comments from online users are the focus of this research, which examines the extent to which commenting forums of online news outlets provide a space for constructive dialogue. Already established as an area of scholarly research, commenting forums are nevertheless still an area of much consternation among news organizations as they try to raise the quality of the dialogue. At the heart of the quandary is the extent to which anonymity plays a role that has an impact on whether a comment has discursive value.

This research measures the quality of online discourse by examining 4,800 comments from the forums of 30 online news sites on the topic of the 2016 U.S. presidential election. Expanding the scope of previous research in this area and oriented in the theory of deliberative democracy, the normative conditions for quality discourse were measured with six parameters: civility, reciprocity, reflexivity, rationality, diversity, and relevance. A seventh category measures whether comments carry an emotional component. This research contributes to literature in the area of the normative conditions for quality discourse (Camaj & Santana, 2015; Coe, Kenski, & Rains, 2014; Collins & Nerlich, 2015; Dahlberg, 2004; Fishkin, 2009; Fredheim et al., 2015; Graham, 2002; Graham & Witschge, 2003; Jensen, 2003; Rowe, 2015; Schneider, 1997; Steenbergen, Bächtiger, Spörndli, & Steiner, 2003; Stromer-Galley, 2007; Wilhelm, 1999) by adding two conditions: anonymity and non-anonymity. Results of this research will be useful for journalism practitioners and scholars in their ongoing examination of ways to raise the quality of the dialogue in online forums.

Literature Review

Deliberative Democracy

Long before the onset of online communication, scholars placed great emphasis on public participation in civic life as a central tenet of democracy. Some have stressed that a standard requirement for a strong democracy includes effective participation, rational understanding of the political process, and equal opportunities in decision making (Dahl, 1989). Deliberative democracy theory posits that group deliberation aids the development of citizens’ political sophistication (Gastil & Dillard, 1999). The theory is often tested or applied in the context of political apathy (Ackerman & Fishkin, 2005) that occurs when people are not engaged in discussing political issues; rather than merely voting, citizens, to be true participants in a democracy, ought to exchange views (Goodin, 2008).

Deliberation is a specific form of communication where participants consider information and weigh different arguments against their own. In this way, the quality of the deliberation is the basis of democratic legitimacy (Dryzek, 2000; Dryzek & Niemeyer, 2008; Gastil & Dillard, 1999; Parkinson, 2003). Deliberation can thus be defined as “an unconstrained exchange of arguments that involves practical reasoning and . . . potentially leads to a transformation of preferences” (Cooke, 2000, p. 948). Gastil and Black (2008) put it this way: “When people deliberate, they carefully examine a problem and arrive at a well-reasoned solution after a period of inclusive, respectful consideration of diverse points of view” (p. 2). These ideas include consideration for the spaces where such deliberation occurs. Habermas (1984/1987, 1988, 1996) believed that offering the public a space for unrestricted and rational communication was a cornerstone of a healthy democracy. Habermas (2006) defined the “public sphere” as a domain of social life where public opinion can be formed and access granted to all citizens.

Online Discourse Research

Research into the quality of online discourse often focuses on the civility of the dialogue (Blom, Carpenter, Bowe, & Lange, 2014; Hmielowski, Hutchens, & Cicchirillo, 2014; Stroud, van Duyn, Alizor, Alibhai, & Lang, 2017), including as it pertains to controversial topics, such as immigration (Santana, 2015), politics (Dillon, Neo, & Seely, 2015; Hmielowski et al., 2014; Papacharissi, 2004; Sobieraj & Berry, 2011), and abortion (Chen & Lu, 2017). Accepting the notion that expressing disagreement in online commenting forums is an expected component of productive dialogue, Chen and Lu (2017) found that both civil and uncivil disagreement caused negative emotions and aggressive intentions. Other research found that the presence of incivility in online political communication limited the deliberative potential of online interactions (Gervais, 2015), and uncivil discussants were perceived as less credible (Ng & Detenber, 2005). Similarly, people perceive civil comments as more persuasive (Chen & Ng, 2016) and trustworthy than uncivil comments (Graf et al., 2017). Incivility in the forums also serves to antagonize (Hwang, Kim, & Kim, 2018) and silence (Dalisay, 2012) newsreaders and leads to lower expectations about public deliberation (Hwang, Kim, & Huh, 2014). Diakopoulos and Naaman (2011) found that the main reasons people cited for not wanting to read online comments in the Sacramento Bee included comments being off-topic, predictable, and argumentative as well as commenters being uninformed, mean spirited, judgmental, and prone to making personal attacks. Civility in discourse alone, however, does not necessarily translate into more deliberative discussion (Rowe, 2015).

Many theorists agree that public forums must meet certain criteria to achieve quality discussion (Ryfe, 2005), and in the past decade, many studies have been conducted to evaluate online discussions in an ideal notion of a public sphere. Schneider (1997) listed four dimensions: equality, diversity, reciprocity, and quality. Wilhelm (1999) suggested topography, topicality, inclusiveness, design, and deliberation. Graham (2002) argued that deliberative discourse occurs when participants’ discourse abides rational-critical debate, reciprocity, reflexivity, sincerity, equality, and freedom. Jensen (2003) selected six criteria: form, dialogue, openness, tone, argumentation, and reciprocity. Steenbergen et al. (2003) listed the following: participation, level of justification, content of justification, respect, and constructive politics. Graham and Witschge (2003) developed a method for examining the extent to which Internet forums meet the normative requirements of rational-critical debate with reciprocity and reflexivity. Informed by the work of Habermas, Dahlberg (2004) also identified six conditions, which include thematization and reasoned critique of problematic validity claims (reciprocity and justification), reflexivity, ideal role taking, sincerity, inclusion and discursive equality, and autonomy from state or economic power. Six elements were essential for political deliberation, according to Stromer-Galley (2007): reasoned opinion expression, references to external sources, expressions of disagreement, equal levels of participation, coherence with regard to the structure and topic of deliberation, and engagement. Fishkin (2009) argued that deliberation requires that discussions be informed, conscientious, balanced, comprehensive, and substantive. Coe et al. (2014) relied on engagement and evidence to assess argument quality, and Collins and Nerlich (2015) defined deliberative discussion as requiring reciprocity, topicality, and argumentation. To measure discursive quality and the effect of anonymity on the quality of the dialogue in commenting forums, other researchers relied on a list of antagonistic words (Fredheim et al., 2015). Marchionni (2015) argued that a true test of deliberation is the extent to which the discourse was a conversation and thus tested six variables: social presence, friendliness, informality, co-orientation, homophily, and interactivity. Finally, Rowe (2015) argued that among the indicators of quality deliberation were if commenters stayed on topic; expressed an opinion; justified it with sources, evidence, or personal experience; provided a solution to the problem; asked or answered a question; and acknowledged the opinions of others.

Conditions for Quality Discourse

Overall, most content-based research exploring the quality of online deliberation concentrate on criteria that often echo each other. In developing normative conditions for an online public sphere, Dahlberg’s (2004) first and widely accepted criterion, for example, emphasizes the extent to which “positions are put forward and critically debated, with all positions being supported by reasons” (p. 33). In utilizing the standards set forth by other scholars, quality dialogue, as defined in this article, includes being diverse in which contrarian points of view are present; reciprocal in which participants acknowledge the point of view of others; reflexive in which the commenter is self-aware via personal and/or firsthand observations; on-topic and focused on the issue/news at hand; and grounded in rationality, expressed through reasoned argumentation. A sixth measure put forward by many researchers deals with need for civility in the discussion. The literature review addresses each of these.

Discursive Civility

Civility in online discourse has been examined for its role in facilitating constructive deliberation, including its effect on people’s willingness to consider and even adopt another point of view (Ksiazek, Peer, & Zivic, 2014). Discursive incivility induces defensive reactions instead of deliberative ones and often leads to a collapse in deliberative dialogue (Hwang et al., 2018). Examined another way, free expression and exposure to differing views can hold deliberative potential only when participants were respectful toward each other, including being empathetic, egalitarian, and open-minded (Benhabib, 1996; Chambers, 1996; Gutmann & Thompson, 1996; Warren, 1996). Incivility can also be defined as attacks that go beyond differences in opinion and that devolve into name-calling, contempt, and derision (Brooks & Geer, 2007). For some, common markers that a comment was considered uncivil were if it contained personal attacks, vulgarities, abusive language, racist sentiments (Santana, 2013), name-calling, capital letters (to indicate a raised voice), and an attack on another commenter’s viewpoints (Chen & Lu, 2017). Other research defined incivility as “an unnecessarily disrespectful tone toward the discussion forum, its participants, or its topics” (Coe et al., 2014 p. 660). For others, a comment was considered uncivil if the commenter verbalized a threat to democracy; assigned stereotypes; threatened other people’s rights (Dillon et al., 2015; Papacharissi, 2004); featured profanity, racial slurs, and attacks of other participants’ character, news media, or reporter (Blom et al., 2014). Online newspapers also have clear definitions. The Philadelphia Inquirer, for example, notifies readers that comments that are unlawful, threatening, abusive, libelous, defamatory, obscene, vulgar, pornographic, profane, indecent, or otherwise objectionable are not allowed, and researchers have used such rules as benchmarks for measuring quality dialogue (Ruiz et al., 2011).

Discursive Diversity

Some scholars argue that while civility may be necessary to have a constructive debate, so too disagreement must exist (Mouffe, 2000). Dahl (1989) noted that citizens must have access to information from a diverse group of sources in order for a democratic vision to be realized. Wojcieszak and Mutz (2009) pointed out that if people are confronted with views contrary to their own, they have the benefit of taking those views into account and, possibly, re-evaluating their own opinions. Similarly, MacKuen (1990) emphasized the importance of a diversity of views to produce dialogue in which individuals engage in discussions with others who hold different viewpoints, and research has shown that exposure to disagreement contributes to people’s ability to generate reason (Price, Capella, & Nir, 2002). In essence, diversity requires a fair and open presentation of competing viewpoints (Schneider, 1997), thus contributing to a diversity and competition of ideas. The existence of such a dimension as a prerequisite to quality discourse equates to the absence of a silencing effect in which people who do not feel their views are acceptable in public discourse are apt to remain silent (Noelle-Neumann, 1984). Hargittai (2018) argued that this ideal, however, is problematic as not all people have equal access to platforms for opinion expression. Younger people and those of higher socioeconomic status, for example, are more likely to be on several platforms, suggesting that the content derived from social media tends to be the views of more privileged people. That privilege can shape discourse and ultimately interfere with deliberation (Robinson, 2018).

Reciprocity

Capturing the degree to which a conversation is deliberative (Janssen & Kies, 2005), reciprocity—the act of interacting with one another—is considered to be a crucial element of deliberation (Schneider, 1997; Wilhelm, 1999). To this end, Graham (2008) asserted that people need to listen and respond to others’ remarks after they have reflected upon their own positions. In previous research, reciprocity has been operationalized in content analysis that codes for the number of replies a message gets (Janssen & Kies, 2005; Jensen, 2003; Schneider, 1997) or questions posed (Rowe, 2015). Whereas some research has shown that many people do not feel connected to other commenters in news commenting forums (Stroud et al., 2017), other research has documented high interactivity between participants on political discussions on social networking sites (Ancu & Cozma, 2009). Political discussion on Facebook groups supporting the 2008 presidential candidates, for example, was dominant among voter-to-voter interactions (Fernandes, Giurcanu, Bowers, & Neely, 2010). Robertson, Vatrapu, and Medina (2010) also found that people who posted on multi-candidate Facebook walls “were not just talking about themselves but also addressing the comments of others” (p. 29), indicating a high level of reciprocity and respect.

Reasoned/Rational Argument

Unlike emotional appeals, informational appeals contextualize an issue by providing factual information. The use of this kind of argument can come in a variety of forms, such as citing news, statistics, issue-to-issue comparisons, and research (Blom et al., 2014; Main, Argo, & Huhmann, 2004). Quality deliberation requires that the exchange of claims be accompanied by reasoned justification (Dahlberg, 2001). As Graham (2008) pointed out, political talk “requires that participants provide reasoned claims, which are critically reflected upon, and that an adequate level of coherence and continuity is maintained” (p. 20). Researchers found that frequent posters on 2008 presidential candidates’ Facebook walls were apt to provide outside links “to provide bolstering evidence for a claim, rebut another comment, or call others to action” (Robertson et al., 2010, p. 29). Other research found that factual responses to uncivil comments indirectly increased participation rates by suggesting a deliberative discussion atmosphere (Ziegele & Jost, 2016). Overall, the use of evidence is often cited as a component of quality deliberation (Coe et al., 2014; Gastil, 2008; Rowe, 2015; Stromer-Galley, 2007).

Reflexivity

In sociology, reflexivity means an act of self-reference or being self-aware. Self-reference occurs in natural or formal languages when a sentence, idea, or formula refers to itself. Dahlberg (2001) defined reflexivity as a scenario in which participants critically examine their cultural values, assumptions, and interests, as well as the larger social context. Graham (2002) described it as “the rethinking of one’s own validity claims and arguments in light of another’s validity claim and/or argument” (p. 46). In philosophy, reflexivity refers to the ability of a subject to speak of or refer to himself or herself—to have the kind of thought expressed by the first-person pronoun “I.” Reflexivity often manifests itself as a story or personal anecdote, although some scholars argue that recognizing the condition might be complicated to recognize as it is such a personal process (Dahlberg, 2004).

Relevance

The notion that online commenters should be expected to stay on-topic is a common expectation of news sites. Among its five guidelines for entering the comments’ section, the Los Angeles Times, for example, lists “Be Relevant” as its first guideline. The Seattle Times similarly asks readers to “keep the conversation focused on the topic at hand,” and the Baltimore Sun asks commenters to keep comments “courteous and on-topic.” Such pre-commenting requirements perhaps presume that the user is reading the accompanying news article, or at least part of it, before registering a relevant comment. Research has shown, however, that, especially for young people, the news consumption experience on social media is largely incidental (Boczkowski, Mitchelstein, & Matassi, 2018). Even if some readers are apt to browse the news rather than delve into deep reading, academic research has also examined story relevance in examining the quality of user comments (Collins & Nerlich, 2015; Rowe, 2015; Stromer-Galley, 2007; Stroud, Scacco, Muddiman, & Curry, 2014).

Anonymity and Non-Anonymity

A commonly discussed subtopic of the topic of online forums is anonymity and its use in non-verbal communication (Boczkowski, 1999). One of the most popular research approaches is the reduced cues model. The model posits that when a person’s identity is removed, he or she is apt to behave and communicate in ways that are different from the way he or she would conduct themselves if his or her identity were intact (Sproull & Kiesler, 1986). Proponents of this model argue that online anonymity is associated with a propensity for anonymous individuals to suspend their awareness of consequences and thus adopt a sense of being less inhibited in their expressions (Sproull & Kiesler, 1986). Anonymity in computer-mediated communication was also found to have an equalizing effect; when people made decisions in face-to-face meetings, the high-status member dominated discussions, but when the same person communicated over email, status dissolved and participation became more inclusive (Dubrovsky et al., 1991). Other research found that prohibiting anonymity in commenting forums was related to less hostile conversation (Ksiazek, 2015; Santana, 2013).

Supporting this idea is the so-called online disinhibition effect, which delves deeper into the varied reasons why people tend to lose their inhibitions while anonymous online (Suler, 2004). Benign disinhibition takes place when people share personal information about themselves and demonstrate acts of compassion and kindness. Toxic disinhibition, on the contrary, takes place when people engage in rude language, harsh criticism, anger, hatred, and threats. In general, anonymity can foster a sense of impunity, loss of self-awareness, and a likelihood of acting upon normally inhibited impulses in ways inconsistent with a person’s offline self. In this way, “with the protection of anonymity and distance, online users can exercise aggression against other people with little risk of being identified or held accountable for their actions” (Hardaker, 2010, p. 238).

Affective Catharsis

The prevalence of toxic disinhibition in online dialogue has given rise to the notion that uninhibited expression may not be without value. Suler (2004) described its use as “a blind catharsis” (p. 321). Some researchers have examined catharsis in online communication and discovered that posting comments in an online discussion board had powerful cathartic consequences for posters’ feelings. Such practices allowed posters “to vent their anger and shame, and in this way discursively release the tensions associated with ongoing challenges to their identity” (da Cunha & Orlikowski, 2008, p. 150). In revisiting the theory of aggression and the “positive effects of aggression,” researchers found that participants who were aggressive against the source of negative feedback had sharp decreases in anger when compared with participants who aggressed against non-source neutral controls (Bresin & Gordon, 2013, p. 401).

Hypotheses

In summary, guided by the suggestions set forth by other scholars (Camaj & Santana, 2015; Coe et al., 2014; Collins & Nerlich, 2015; Dahlberg, 2004; Fishkin, 2009; Fredheim et al., 2015; Graham, 2002; Graham & Witschge, 2003; Jensen, 2003; Rowe, 2015; Schneider, 1997; Steenbergen et al., 2003; Stromer-Galley, 2007; Wilhelm, 1999) in an attempt to operationalize the criteria necessary for online quality discourse, this research was conducted after constructing its own similar measurement. This article argues that these variables—civility, reciprocity, reflexivity, rationality, diversity of opinion, and relevance—represent an ample overview of what is required for a comment to be considered valuable to a discussion. These variables were compared with the conditions of the commenter—anonymous and non-anonymous—which, research has shown, have a distinct effect on user behavior. Specifically, anonymous online users are more apt to lower their inhibitions (Sproull & Kiesler, 1986; Suler, 2004) and thus are more likely to express themselves more openly, freely and, by extension, coarsely. Even if not engaged in quality discourse, anonymous commenters may find more value than non-anonymous commenters in registering their raw, emotional opinions on a public forum. Thus, the following hypotheses are posed:

H1:Non-anonymous commenters in online news commenting forums will be more likely to meet the conditions for quality deliberation than anonymous commenters.

H2:Anonymous commenters in online news commenting forums will be more likely to express an emotional comment than non-anonymous commenters.

Method

The websites of all U.S. daily newspapers with a circulation greater than 50,000 daily (n = 137) were visited. Furthermore, 13 other news sites, including broadcast and online-only news sites determined to be the largest, were included for analysis. They included: Fox News, MSNBC, Huffington Post, the Mail (the United Kingdom), the Guardian (the United Kingdom), CNN, NBC News, BBC News, Yahoo News, ABC News, CBS News, and the wire services Reuters, UPI, and the Associated Press. In all, 150 news sites were visited. In each, several news stories were accessed to determine if the site hosted forums. If they did, the researcher determined if the site allowed or disallowed anonymous commenting.

In the vast majority of cases, when a news organization disallowed anonymity, it did so by requiring a commenter to register with his or her Facebook account, which then revealed the commenter’s name, picture, and sometimes occupation and location, alongside the comment. A smaller portion allowed a Google+ or LinkedIn login. A comment was considered anonymous if the commenter’s name and/or picture was unknown to other commenters. A comment was considered non-anonymous if the commenter’s name and/or picture was known to other commenters.

Among the nation’s top 35 cities (per population), 24 in 17 states were identified as having news organizations that offered unmoderated/post-moderated anonymous and non-anonymous forums. (Virtually all forums are post-moderated in which users can flag an existing comment for consideration for removal by a moderator. This is often highly subjective; some comments require a certain number of flags to be removed, and thus few comments are deleted. See Diakopoulos & Naaman, 2011.) In each of the 24 cities, 24 daily newspapers were identified as hosting both anonymous and non-anonymous forums. The 12 newspapers with non-anonymous forums included the Arizona Republic, Boston Herald, El Paso Times, Detroit News, Charlotte Observer, Columbus Dispatch, Indianapolis Star, Nashville Tennessean, Las Vegas Review-Journal, Kansas City Star, Fresno Bee and the Sacramento Bee. The 12 newspapers with anonymous commenting forums were the Washington Post, Dallas Morning News, San Antonio Express-News, Houston Chronicle, Los Angeles Times, Chicago Tribune, Philadelphia Inquirer, Seattle Times, San Diego Union-Tribune, Denver Post, Austin American Statesman, and the Florida Times-Union. Finally, comments from national broadcast news networks and online-only news websites were also captured. Fox News, ABC News, CBS News, NBC News, and Yahoo News offered anonymous forums while the Huffington Post offered non-anonymous forums. Thirty news organizations in the nation’s most populous cities offered forums from which comments were drawn.

Collected comments came from online news stories on the U.S. presidential election. The topic was chosen because it was one that was most likely to engender robust discussion and debate. Coders entered the “below-the-line” commenting forums of all 30 news sites on designated dates most likely to generate the greatest number of comments. The dates of collection, beginning on September 27 and ending on November 10, surrounded the first, second, and third presidential debates; the vice-presidential debate; and the presidential election; thus, the dates of comment collection were September 27, 28; October 5, 6; October 10, 11; October 20, 21; and November 9, 10. Coders were instructed to collect comments from stories that were specifically about the election, including comments that were “nested” as secondary replies to a “parent” comment. Coders were instructed to collect the same number of comments from news stories from each site—between 300 and 400 comments for each of the 30 news sites during the 10-day collection period. This was done by selecting and copying the comments from the webpage, then pasting and numbering each in a Word document. In all, 51,575 comments were collected. After each comment was assigned a number, an online random number generator randomized the comments, and 9.3% of the sample (4,800 comments) was chosen for inspection. The sample included 2,400 anonymous and 2,400 non-anonymous comments. Then, all 4,800 comments were coded for content. The individual comment was the unit of analysis.

This coding schema was guided by existing empirical, normative, and theoretical studies of deliberation (Stromer-Galley, 2007).

Civility

A comment was considered “uncivil” if it had at least one of the following: personal or inflammatory attacks, threats, vulgarities, language that was obscene or hateful, epithets or ethnic slurs, sentiments that were racist or bigoted, disparaging on the basis of race/ethnicity or that assigned stereotypes. This type of comment sometimes took the form of cruel potshots, sharp one-line zingers, snarky rejoinders and ad hominem attacks (i.e., “You suck!”), or computer-based acronyms (i.e., “stfu”). A comment was considered “civil” if it had none of the “uncivil” characteristics. A third category was considered “neither/nor” to classify comments that were deemed neither especially uncivil nor especially civil. This type of comment did not have any of the “uncivil” characteristics but was impolite and/or ill-mannered. It could be spirited but lacked decorum (i.e., “Oh, boo hoo, cry me a river. Build a wall and get over it.”) Comments that lacked context, that appeared apropos of nothing (i.e., “y’all need jesus”), or were otherwise unintelligible were coded in this latter category.

Reciprocity

An affirmative determination in this category was made when the commenter responded to other people’s remarks by addressing a previous commenter, either by (pseudo)name or with a hashtag, before or after weighing in with his or her own opinion (i.e., “@NickG Were you as upset at Trump interrupting Hillary and the moderator?”). To the extent that it could be determined, if a comment replied to a parent message within a cascading thread (nested replies), even without addressing the original commenter, it was coded in the affirmative. Overall, coders were instructed to determine if the comment asked or answered a question and/or responded or addressed another commenter, including the journalist/news organization of the news story commented on.

Reflexivity

Coders were instructed that reflexivity often manifests itself as a personal anecdote, most clearly identified with the use of the pronouns “I,” “me,” or “my.” However, to be coded affirmatively in this category, the comment needed to be more than mere self-referenced rejoinder (i.e., “I doubt it.”); the comment needed to possess language that demonstrated some reasoned self-reflection, even a casual one (i.e., “It is hard for me to watch all of this since I think she’s lying, but until some magical truth serum is invented, I don’t think that anything is going to change.”).

Rationality

Coders were instructed that this category indicated a discussion supported by rational reasoning and/or evidence. One method for determining if an argument was reasoned was to see if the statement under analysis included a clause that began with “because,” indicating the potential presence of elucidation. Coders were instructed that evidence could consist of citations to news stories, polls, research, statistics, records, and so forth. If there was a figure/data in the sentence, it qualified as evidence (i.e., “We already have roughly 18,000 of the 21,000 Border Patrol agents stationed along the US-Mexico border. If the ‘wall’ is simply a metaphor for increased border security, what are we going to build?”). Well supported argumentation was the key factor, not the accuracy of the citation. Ultimately, a rational comment was one that was based in accordance with reason or logic.

Diversity

A comment fell into this category if it expressed some disagreement with another comment or what the commenter perceived to be the frame/bias of the news story. Coders were instructed that the comment had to have language that acknowledged the point of view of another comment and/or news story—which was coded in the affirmative if it was part of a nested thread (i.e., as a response)—and offered an alternative point of view, contributing to a competition of ideas. The comment had to engage in an exchange of dissimilar perspectives, even if via inference and absent exposition (i.e., “How can you think that? That’s the most ridiculous thing I ever heard.”). If the comment presented a contrarian point of view to any aspect of another comment or news story, it was coded affirmatively in this category.

Relevance

For this category, coders were instructed to determine if the comment was related to the accompanying news item. Coders were instructed that it was not necessary to read the entire news story but to glean what the overarching topic was about. They were instructed that even if “Trump,” “Clinton,” “Pence,” or “Kaine” were not mentioned, they were to use their best judgment when making the determination if the comment was about a candidate, a political topic, or some aspect of the political process, including, for example, stories about the electoral college or swing states.

Affective Catharsis

A comment was considered emotional if it appealed to non-factual arguments or reasoning. Coders were instructed that this type of comment contained no empirical evidence to support an opinion. Instead, it was characterized by intense feeling (i.e., anger, rage, despair, or elation) and was devoid of rationality or perspective. This type of comment might be seen as lacking any of the previous quality criteria and often overlapped as an “uncivil” comment but not always (i.e., “Never Forget Benghazi!”). Comments in all caps, indicating a raised voice, were also coded in the affirmative in this category.

After a process of refining categories and clarifying definitions during pretests, coders were trained with these detailed definitions. Coders coded 10% of the sample and reached agreement on all criteria, Krippendorff’s alpha (α) ≥ .80. Coders then coded the entire sample of 4,800 comments.

Results

Again, the first hypothesis in this research predicted that non-anonymous commentary would be more civil than anonymous commentary. A chi-square test of independence was calculated comparing the civility of the comment with the identity status of the users (anonymous or non-anonymous). A significant interaction was found, χ²(3) = 348.3, p < .001 (see Table 1). Anonymous commenters were significantly more likely to register their opinion with an uncivil comment than non-anonymous commenters. Just over 55% of the anonymous comments were uncivil, while nearly 33% of non-anonymous comments were uncivil. Looked at another way, about 34% of the anonymous comments were civil, while 61% of the non-anonymous comments were civil. The remainder in both categories were either neither/nor or unclear.

Table 1

Variables by Condition

	Anonymous, %	Non-anonymous, %	χ²
Civility			348.3**
Civil	34.2	61
Uncivil	54.9	32.8
Neither/nor	11	6.2
Reciprocity			17.4**
Present	54.5	60.1
Not present	43.5	37.6
Neither/nor	2.0	2.3
Reflexivity			95.5**
Present	18.6	30.7
Not present	81.3	69.1
Neither/nor	0.1	0.2
Rationality			17.9**
Present	19.4	24.1
Not present	80.6	75.8
Neither/nor	0	0.1
Diversity			8.72*
Present	57.9	54.6
Not present	40.1	43.9
Neither/nor	2	1.4
Relevance			206.1**
Present	62.8	80.5
Not present	35.1	17
Neither/nor	2	2.5
Emotional			162.3**
Present	45.7	28.0
Not present	51	68.1
Neither/nor	3.3	3.8

p ≤ .05. **p ≤ .001.

Similar results were found with the other variables. Anonymous commenters were significantly more likely to register their opinion without reciprocity than non-anonymous commenters, χ²(2) = 17.4, p < .001. Reciprocity appeared in 60% of non-anonymous comments compared with 54% of anonymous comments. Anonymous commenters were also significantly more likely to register their opinion without reflexivity than non-anonymous commenters, χ²(2) = 95.5, p < .001. In more than 81% of anonymous comments, reflexivity was absent compared with its absence in 69% of the times for non-anonymous comments. While there was also a significant difference between the conditions, χ²(2) = 17.9, p < .001, rationality in the comments of top news websites was absent in more than 80% of anonymous comments and 75% of non-anonymous comments. The case was the same with diversity in comments, χ²(2) = 8.72, p < .05, where the condition was present in 58% of anonymous comments and just over 54% of non-anonymous comments. Finally, non-anonymous commenters were significantly more likely to stay on-topic than non-anonymous commenters, χ²(2) = 206.1, p < .001, with more than 80% of non-anonymous comments on-topic compared with 63% of anonymous comments. Considering these findings, H1 was supported.

Again, the second hypothesis predicted that commentary in anonymous online news commenting forums would be used primarily as places for expressing vociferous emotions. Anonymous commenters found this use of commenting forums significantly more useful than non-anonymous commenters, χ²(3) = 162.3, p < .001, with more than 45% of anonymous commenters expressing some level of forceful emotional expression compared with 28% of non-anonymous commenters. Thus, H2 was tentatively supported.

Conclusion

On the morning of November 12, 2016, with the U.S. presidential election on his mind, a retired chef from Boston entered a Boston Herald commenting forum following a news story on the election and weighed in: “Just as economists predicted, Trump will put the country into a recession,” he began. About an hour later, a man from College Park, Maryland, offered a counter response, addressing the first commenter by name. “Yeah the bond markets are spooked because Trump might add $3.5 T to the debt over ten years but weren’t spooked by the $10 T added by Obama.” Soon, a third Maryland man joined the conversation, addressing the second man by name. What then followed was a reasoned and civil debate about issues facing the nation upon the election of a new president. All three commenters registered their opinions using their Facebook page, presumably with their real identities. That same day, across the country in a commenting forum in the Los Angeles Times, two anonymous commenters were busy hurling insults at each other. One called the other a “pig” and a “liar” while the other responded by saying he was an “idiot” and an “ass****.” The back-and-forth among the commenters was bitterly acerbic, mostly personal attacks filled with hateful language.

Through an examination of a sample of news comments, this study offers insights into the psyche of online commenters like these in an attempt to understand if news forums reflect the ideals of the normative conditions for quality discourse in a public sphere. What it found was, on one hand, people angry and at stark odds with one another, expressing themselves with combative language and with more monologue than dialogue, often based in a dichotomous (liberal vs. conservative) point of view. It should be noted that, considering the topic of the news, this result should be met with some caution; research has shown that on matters of politics, people are apt to be especially aggressive in their expression (Dillon et al., 2015; Hmielowski et al., 2014). On the other hand, responding to issues raised by Trump, Clinton, Pence, and Kaine in pre-election debate news coverage, this research found that other commenters were not altogether infrequently engaged in meaningful, civil, and deliberative (albeit spirited) debate with each other.

One key ingredient in the differences in the comments is the element of anonymity. This research has shown that anonymous commenters were not only more likely to be uncivil but also more likely to not engage in any of what the academic literature outlines as requirements for quality dialogue. The reduced cues model in social science and the online disinhibition effect (Sproull & Kiesler, 1986; Suler, 2004) might offer the best explanation. By removing their identity, people are apt to lower their inhibition, which often manifests itself as incivility, although it is important to be wary of claiming causation. Still, in all, anonymous commenting boards hosted on news sites were more likely to be fraught with incivility, while non-anonymous commenting forums were more likely to contain civil sentiments. This is consistent with the findings of previous research that compared the civility of comments embedded on a news site versus Facebook (Dillon et al., 2015; Fredheim et al., 2015; Rowe, 2014, 2015; Su et al., 2018), although Facebook commenters do not necessarily perceive the platform as a better alternative. Facebook news commenters, for example, did not perceive their audiences to be any more reasonable, intelligent, or responsive than did commenters on news organizations’ websites. They also did not perceive comments to be of any greater quality than did commenters on news organizations’ websites (Kim, Lewis, & Watson, 2018).

Increasingly prevalent in the academic literature are findings about the extent to which such conditions of the commenter plays a role in not just the civility of the comments but in their overall discursive quality. But the differences, while statistically significant, did not necessarily mean that being non-anonymous represents an ideal condition for quality dialogue; it merely shows that it is a better condition for approaching quality. Just over 75% of comments from non-anonymous commenters, for example, did not incorporate the use of a rational argument, and nearly 33% were uncivil in their expression. Still, non-anonymous comments were better than anonymous comments in every quality measure, including rationality. This study’s findings can be directly relevant to commenting-hosting news sites and their commenters who, anecdotally observed in this research, often express exacerbation of the low quality of the dialogue. (This research found that 93% of the top 150 news sites hosted comments. Of those, 78 [52%] allowed anonymous commenting, 62 [41.3%] disallowed it, and 10 [6.6%] did not have the forums.) Some scholars have argued that the content that some users create can be seen as a type of citizen journalism insomuch as they are augmenting news from traditional journalists with relevant and rational information. If one agrees with this argument, a new kind of participatory journalism is produced when news workers seek the public’s positive participation in the news production process (Westlund & Ekström, 2018).

Another area explored in this study is the idea that even comments devoid of quality measures may nevertheless not be without some value. A seventh category of this research measured the extent to which a comment carried an emotional component, perhaps giving rise to the notion that catharsis via forceful self-expression may be a useful benefit that some commenters enjoy. This finding should be viewed as tentative; catharsis may be one psychological outcome of raw expression, but this is not well supported here. Allowing commenters anonymity may afford that expression while also ensuring broader participation in commenting forums (Reader, 2012), and many commenters want news sites to continue to allow anonymity (Stroud et al., 2017) because disallowing it makes them less apt to enter the fray (Fredheim et al., 2015). Overall, anonymous commenters were more likely to express themselves with an emotional appeal, whereas the majority of non-anonymous commenters avoided it. It may be that venting anger online may serve to mitigate offline anger (Bresin & Gordon, 2013; Bushman, 2002), but new research in this area, including cross-disciplinary collaborations with psychology scholars, is needed. Despite their promotion by news websites as spaces of deliberation, commenting forums should also be further studied as spaces that provide an opinion-sharing gratification (Sundar & Limperos, 2013).

New research in the areas of civility, reciprocity, diversity, rationality, reflexivity, and relevance should be informed by this research, and the development of theory in the area of online quality discourse should include consideration of the effects of anonymity. Future quantitative and qualitative research should examine the behavior, including motivations, of anonymous commenters as newsrooms continue to give them considerable latitude for uncivil and unproductive expression. Future research also should study smaller news media, as this research studied only the very largest, and therefore a very small percentage of, U.S. daily newspapers (and no non-daily newspapers) and only national/international news sources of other types.

Footnotes

Acknowledgements

The author would like to thank research assistants Meagan Leon, Richa Mishra, and Rajdeep Yadav for their assistance on this project. This project was made possible through a research grant from San Diego State University.

References

Ackerman

Fishkin

J. S.

(2005). Deliberation day. New Haven, CT: Yale University Press.

Ancu

Cozma

(2009). Myspace politics: Uses and gratifications of befriending candidates. Journal of Broadcasting & Electronic Media, 53, 567-583.

Anderson

A. A.

Brossard

Scheufele

D. A.

Xenos

M. A.

Ladwig

(2014). The “nasty effect”: Online incivility and risk perceptions of emerging technologies. Journal of Computer-Mediated Communication, 19, 373-387.

Benhabib

(Ed.). (1996). Democracy and difference: Contesting the boundaries of the political. Princeton, NJ: Princeton University Press.

Berg

(2016). The impact of anonymity and issue controversiality on the quality of online discussion. Journal of Information Technology & Politics, 13, 37-51.

Blom

Carpenter

Bowe

B. J.

Lange

(2014). Frequent contributors within U.S. newspaper comment forums: An examination of their civility and information value. American Behavioral Scientist, 58, 1314-1328.

Boczkowski

(1999). Understanding the development of online newspapers: Using computer-mediated communication theorizing to study internet publishing. New Media & Society, 1, 101-126.

Boczkowski

Mitchelstein

Matassi

(2018). “News comes across when I’m in a moment of leisure”: Understanding the practices of incidental news consumption on social media. New Media & Society, 20, 3523-3539.

Bresin

Gordon

K. H.

(2013). Aggression as affect regulation: Extending catharsis theory to evaluate aggression and experiential anger in the laboratory and daily life. Journal of Social & Clinical Psychology, 32, 400-423.

10.

Brooks

D. J.

Geer

J. G.

(2007). Beyond negativity: The effects of incivility on the electorate. American Journal of Political Science, 51, 1-16.

11.

Burkell

(2006). Anonymity in behavioural research: Not being unnamed, but being unknown. University of Ottawa Law & Technology Journal, 3, 189-203.

12.

Bushman

B. J.

(2002). Does venting anger feed or extinguish the flame? Catharsis, rumination, distraction, anger and aggressive responding. Personality and Social Psychology Bulletin, 28, 724-731.

13.

Camaj

Santana

A. D.

(2015). Political deliberation on Facebook during electoral campaigns: Exploring the relevance of moderator’s technical role and political ideology. Journal of Information Technology & Politics, 12, 325-341.

14.

Chambers

(1996). Reasonable democracy: Jürgen Habermas and the politics of discourse. Ithaca, NY: Cornell University Press.

15.

Chen

G. M.

(2017). Online political discourse: Exploring differences in effects of civil and uncivil disagreement in news website comments. Journal of Broadcasting & Electronic Media, 6, 108-125.

16.

Chen

G. M.

Y. M. M.

(2016). Third-person perception of online comments: Civil ones persuade you more than me. Computers in Human Behavior, 55(Part B), 736-742.

17.

Coe

Kenski

Rains

S. A.

(2014). Online and uncivil? Patterns and determinants of incivility in newspaper website comments. Journal of Communication, 64, 658-679.

18.

Collins

Nerlich

(2015). Examining user comments for deliberative democracy: A corpus-driven analysis of the climate change debate online. Environmental Communication, 9, 189-207.

19.

Conlin

Roberts

(2016). Presence of online reader comments lowers news site credibility. Newspaper Research Journal, 37, 365-376.

20.

Cooke

(2000). Five arguments for deliberative democracy. Political Studies, 48, 947-969.

21.

Da Cunha

J. V.

Orlikowski

W. J

. (2008). Performing catharsis: The use of online discussion forums in organizational change. Information and Organization, 18, 132-156.

22.

Dahl

R. A.

(1989). Democracy and its critics. New Haven, CT: Yale University Press.

23.

Dahlberg

(2001). Computer-mediated communication and the public sphere: A critical analysis. Journal of Computer-Mediated Communication, 7(1), JCMC174.

24.

Dahlberg

(2004). Net-public sphere research: Beyond the “first phase.” Javnost—The Public, 11, 27-44.

25.

Dalisay

F. S.

(2012). The spiral of silence and conflict avoidance: Examining antecedents of opinion expression concerning the U.S. military buildup in the Pacific island of Guam. Communication Quarterly, 60, 481-503.

26.

Davis

(1999). Web of politics: The Internet’s impact on the American political system. New York, NY: Oxford University Press.

27.

Diakopoulos

Naaman

(2011, March 19-23). Towards quality discourse in online news comments. Proceedings of the ACM 2011 conference on computer supported cooperative work, Hangzhou, China.

28.

Dillon

K. P.

Neo

R. L.

Seely

(2015, October 21-24). Civil keystrokes: Examining anonymity, politeness, and civility in online newspaper forums. Proceedings of the 16th annual meeting of the Association of Internet Researchers, Phoenix, AZ.

29.

Dryzek

J. S.

(2000). Deliberative democracy and beyond: Liberals, critics, contestations. Oxford, UK: Oxford University Press.

30.

Dryzek

J. S.

Niemeyer

(2008). Discursive representation. American Political Science Review, 102, 481-493.

31.

Dubrovsky

V. J.

Kiesler

Sethna

B. N.

(1991). The equalization phenomenon: Status effects in computer-mediated and face-to-face decision-making groups. Human-Computer Interaction, 6, 119-146.

32.

Fernandes

Giurcanu

Bowers

K. W.

Neely

J. C.

(2010). The writing on the wall: A content analysis of college students’ Facebook groups for the 2008 presidential election. Mass Communication and Society, 13, 653-675.

33.

Fishkin

J. S.

(2009). When the people speak: Deliberative democracy and public consultation. Oxford, UK: Oxford University Press.

34.

Fredheim

Moore

Naughton

(2015, June). Anonymity and online commenting: The broken windows effect and the end of drive-by commenting. Proceedings of the ACM Web Science Conference (WebSci’15), Oxford, UK.

35.

Gastil

(2008). Political communication and deliberation. Los Angeles, CA: SAGE.

36.

Gastil

Black

L. W.

(2008). Public deliberation as the organizing principle of political communication research. Journal of Public Deliberation, 4, 1-47.

37.

Gastil

Dillard

J. P.

(1999). Increasing political sophistication through public deliberation. Political Communication, 16, 3-23.

38.

Gervais

B. T.

(2015). Incivility online: Affective and behavioral reactions to uncivil political posts in a web-based experiment. Journal of Information Technology & Politics, 12, 167-185.

39.

Goodin

R. E.

(2008). Innovating democracy: Democratic theory and practice after the deliberative turn. New York, NY: Oxford University Press.

40.

Graf

Erba

Harn

R.-W.

(2017). The role of civility and anonymity on perceptions of online comments. Mass Communication and Society, 20, 526-549.

41.

Graham

(2002). The public sphere needs you. Deliberating in online forums: New hope for the public sphere? (Master’s thesis). Amsterdam School of Communications Research, University of Amsterdam, The Netherlands.

42.

Graham

(2008). Needles in a haystack: A new approach for identifying and assessing political talk in nonpolitical discussion forums. Javnost—The Public, 15, 17-36.

43.

Graham

Witschge

(2003). In search of online deliberation: Towards a new method of examining the quality of online discussions. Communications, 28, 173-204.

44.

Graham

Wright

(2015). A tale of two stories from “below the line”: Comment fields at the Guardian. The International Journal of Press/Politics, 20, 317-338.

45.

Gutmann

Thompson

(1996). Democracy and disagreement. Cambridge, MA: Belknap Press of Harvard University Press.

46.

Habermas

(1987). The theory of communicative action (Vols. 1 & 2). Boston, MA: Beacon Press. (Original work published 1984)

47.

Habermas

(1988). The structural transformation of the public sphere. Cambridge, MA: MIT Press.

48.

Habermas

(1996). Between facts and norms: Contributions to a discourse theory of law and democracy. Cambridge, MA: MIT Press.

49.

Habermas

(2006). The public sphere. In Goodin

R. E.

Pettit

(Eds.), Contemporary political philosophy: An anthology (pp. 103-107). Malden, MA: Blackwell.

50.

Hardaker

(2010). Trolling in asynchronous computer-mediated communication: From user discussions to academic definitions. Journal of Politeness Research, 6, 215-242.

51.

Hargittai

(2018). Potential biases in big data: Omitted voices on social media. Social Science Computer Review. Advance online publication. doi:10.1177/0894439318788322

52.

Hmielowski

J. D.

Hutchens

M. J.

Cicchirillo

V. J.

(2014). Living in an age of online incivility: Examining the conditional indirect effects of online discussion on political flaming. Information, Communication & Society, 17, 1196-1211.

53.

Hsueh

Yogeeswaran

Malinen

(2015). “Leave your comments below”: Can biased online comments influence our own prejudicial attitudes and behaviors? Human Communication Research, 41, 557-576.

54.

Hwang

Kim

Huh

C. U.

(2014). Seeing is believing: Effects of uncivil online debate on political polarization and expectations of deliberation. Journal of Broadcasting & Electronic Media, 58, 621-633.

55.

Hwang

Kim

(2018). Influence of discussion incivility on deliberation: An examination of the mediating role of moral indignation. Communication Research, 45, 213-240.

56.

Janssen

Kies

(2005). Online forums and deliberative democracy. Acta Politica, 40, 317-335.

57.

Jensen

J. L.

(2003). Public spheres on the internet: Anarchic or government-sponsored—A comparison. Scandinavian Political Studies, 26, 349-374.

58.

Kim

Lewis

S. C.

Watson

B. R.

(2018). The imagined audience for and perceived quality of news comments: Exploring the perceptions of commenters on news sites and on Facebook. Social Media + Society, 4, 1-12.

59.

Ksiazek

T. B.

(2015). Civil interactivity: How news organizations’ commenting policies explain civility and hostility in user comments. Journal of Broadcasting & Electronic Media, 59, 556-573.

60.

Ksiazek

T. B.

(2016). Commenting on the news: Explaining the degree and quality of user comments on news websites. Journalism Studies, 19, 650-673.

61.

Ksiazek

T. B.

Peer

Zivic

(2014). Discussing the news: Civility and hostility in user comments. Digital Journalism, 3, 850-870.

62.

Lee

E.-J.

(2012). That’s not the way it is: How user-generated comments on the news affect perceived media bias. Journal of Computer-Mediated Communication, 18, 32-45.

63.

MacKuen

(1990). Speaking of politics: Individual conversational choice, public opinion and the prospects for deliberative democracy. In Ferejohn

J. A.

Kuklinski

J. H.

(eds.), Information and democratic processes (pp. 59-99). Urbana: University of Illinois Press.

64.

Main

K. J.

Argo

J. J.

Huhmann

B. A.

(2004). Pharmaceutical advertising in the USA: Information or influence? International Journal of Advertising, 23, 119-142.

65.

Marchionni

(2015). Online story commenting: An experimental test of conversational journalism and trust. Journalism Practice, 9, 230-249.

66.

Mouffe

(2000). The democratic paradox. London, England: Verso.

67.

E. W. J.

Detenber

B. H.

(2005). The impact of synchronicity and civility in online political discussion on perceptions and intentions to participate. Journal of Computer-Mediated Communication, 10(3), JCMC1033.

68.

Nielsen

C. E.

(2012). Newspaper journalists support online comments. Newspaper Research Journal, 33, 86-100.

69.

Nielsen

C. E.

(2014). Coproduction or cohabitation: Are anonymous online comments on newspaper websites shaping news content? New Media & Society, 16, 470-487.

70.

Nissenbaum

(1999). The meaning of anonymity in an information age. The Information Society: An International Journal, 15, 141-144.

71.

Noelle-Neumann

(1984). The spiral of silence: Public opinion—Our social skin. Chicago, IL: University of Chicago Press.

72.

Papacharissi

(2004). Democracy online: Civility, politeness, and the democratic potential of online political discussion groups. New Media & Society, 6, 259-283.

73.

Parkinson

(2003). Legitimacy problems in deliberative democracy. Political Studies, 51, 180-196.

74.

Price

Capella

J. N.

Nir

(2002). Does disagreement contribute to more deliberative opinion? Political Communication, 19, 95-112.

75.

Prochazka

Weber

Schweiger

(2018). Effects of civility and reasoning in user comments on perceived journalistic quality. Journalism Studies, 19, 62-78.

76.

Reader

(2012). Free press vs. free speech? The rhetoric of “civility” in regard to anonymous online comments. Journalism & Mass Communication Quarterly, 89, 495-513.

77.

Robertson

S. P.

Vatrapu

R. K.

Medina

(2010). Off the wall political discourse: Facebook use in the 2008 U.S. presidential elections. Information Polity, 15, 11-31.

78.

Robinson

(2018). Networked news, racial divide: How power & privilege shape public discourse in progressive communities. Cambridge, UK: Cambridge University Press.

79.

Rowe

(2014). Civility 2.0: A comparative analysis of incivility in online political discussion. Information, Communication & Society, 18, 121-138.

80.

Rowe

(2015). Deliberation 2.0: Comparing the deliberative quality of online news user comments across platforms. Journal of Broadcasting & Electronic Media, 59, 539-555.

81.

Ruiz

Domingo

Micó

J. L.

Diaz-Noci

Meso

Masip

(2011). Public sphere 2.0? The democratic qualities of citizen debates in online newspapers. The International Journal of Press/Politics, 16, 463-487.

82.

Ryfe

D. M.

(2005). Does deliberative democracy work? Annual Review of Political Science, 8, 49-71.

83.

Santana

A. D.

(2011). Online readers’ comments represent new opinion pipeline. Newspaper Research Journal, 32, 66-81.

84.

Santana

A. D.

(2013). Virtuous or vitriolic: The effect of anonymity on civility in online newspaper reader comment boards. Journalism Practice, 8, 18-33.

85.

Santana

A. D.

(2015). Incivility dominates online comments on immigration. Newspaper Research Journal, 36, 92-107.

86.

Schneider

S. M.

(1997). Expanding the public sphere through computer-mediated communication: Political discussion about abortion in a Usenet newsgroup. Boston: Massachusetts Institute of Technology. Retrieved from http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.198.8577&rep=rep1&type=pdf

87.

Singer

J. B.

(1996). “Virtual anonymity”: Online accountability and the virtuous virtual journalist. Journal of Mass Media Ethics, 11, 95-106.

88.

Singer

J. B.

(2010). Quality control: Perceived effects of user-generated content on newsroom norms, values and routines. Journalism Practice, 4, 127-142.

89.

Sobieraj

Berry

J. M.

(2011). From incivility to outrage: Political discourse in blogs, talk radio, and cable news. Political Communication, 28, 19-41.

90.

Sproull

Kiesler

(1986). Reducing social context cues: Electronic mail in organizational communications. Management Science, 32, 1492-1512.

91.

Steenbergen

Bächtiger

Spörndli

Steiner

(2003). Measuring political deliberation: A discourse quality index. Comparative European Politics, 1, 21-48.

92.

Stromer-Galley

(2007). Measuring deliberation’s content: A coding scheme. Journal of Public Deliberation, 3, 1-35.

93.

Stroud

N. J.

Scacco

J. M.

Muddiman

Curry

A. L.

(2014). Changing deliberative norms on news organization’s Facebook sites. Journal of Computer-Mediated Communication, 20, 188-203.

94.

Stroud

N. J.

van Duyn

Alizor

Alibhai

Lang

(2017). Comment section survey across 20 news sites. Engaging New Project. Retrieved from https://mediaengagement.org/wp-content/uploads/2017/01/Comment-Section-Survey-Across-20-News-Sites.pdf

95.

Stroud

N. J.

van Duyn

Peacock

(2015). News commenters and news comment readers. Engaging New Project. Retrieved from https://mediaengagement.org/wp-content/uploads/2016/03/ENP-News-Commenters-and-Comment-Readers1.pdf

96.

L. Y.-F.

Xenos

M. A.

Rose

K. M.

Wirz

Scheufele

D. A.

Brossard

(2018). Uncivil and personal? Comparing patterns of incivility in comments on the Facebook pages of news outlets. New Media & Society, 20, 3678-3699.

97.

Suler

(2004). The online disinhibition effect. Cyber Psychology & Behavior, 7, 321-326.

98.

Sundar

S. S.

Limperos

A. M.

(2013). Uses and Grats 2.0: New gratifications for new media. Journal of Broadcasting & Electronic Media, 57, 504-525.

99.

Thorson

Vraga

Ekdale

(2010). Credibility in context: How uncivil online commentary affects news credibility. Mass Communication and Society, 13, 289-313.

100.

Usher

(2017). The appropriation/amplification model of citizen journalism: An account of structural limitations and the political economy of participatory content creation. Journalism Practice, 11, 247-265.

101.

Warren

M. E.

(1996). Deliberative democracy and authority. American Political Science Review, 90, 46-60.

102.

Westlund

Ekström

(2018). News and participation through and beyond proprietary platforms in an age of social media. Media and Communication, 6(4), 1-10.

103.

Wilhelm

A. G.

(1999). Virtual sounding boards: How deliberative is online political discussion? In Loader

B. D.

Hague

B. H.

(Eds.), Digital democracy: Discourse and decision making in the information age (pp. 154-178). London, England: Routledge.

104.

Wojcieszak

M. E.

Mutz

D. C.

(2009). Online groups and political discourse: Do online discussion spaces facilitate exposure to political disagreement? Journal of Communication, 59, 40-56.

105.

Ziegele

Jost

P. B.

(2016). Not funny? The effects of factual versus sarcastic journalistic responses to uncivil user comments. Communication Research. Advance online publication. doi:10.1177/0093650216671854