The Sociomateriality of Rating and Ranking Devices on Social Media: A Case Study of Reddit’s Voting Practices

Abstract

Rating and ranking devices are everywhere on social media. While these devices may seem like objective tools to measure value and rank content, research shows how they profoundly shape social interaction and emotional expression and are central to platform moderation. Yet, very little is known about how users themselves talk about these devices, much less what this can tell us about how these devices co-constitute social reality on platforms. To explore this gap, we examine Reddit’s rating and ranking device, known as upvoting and downvoting, through a textual analysis of over half a million user comments that contain keywords such as “upvote” and “downvote” and their variants. We find that Redditors (Reddit users) rarely use or talk about voting in the way the platform intends. For the most part, Redditors not only disregard the rules about voting but also make, and enforce, their own rules, norms, and ethics around it. We uncover a rich set of voting practices that we present as the following four themes in a conceptual framework: (1) platform culture, (2) prescriptive device, (3) materialization of value, and (4) ontology of self. Drawing on a sociomaterial lens, we reposition voting as a material-discursive practice that is inseparable to Reddit culture. This provides compelling evidence that rating and ranking devices on social media intervene in and perform sociality and we invite future research to apply our conceptual framework to other rating and ranking devices on social media.

Keywords

Reddit voting ratings rankings sociomateriality ontology

Introduction

Ratings are ubiquitous design features on social media platforms. On Facebook, users can “Like” posts, “react” to comments, and “flag” inappropriate or offensive content. Twitter’s “Like” button enables users to express approval, or perhaps tag for later reading tweets and replies on tweet threads. On Reddit, the upvote/downvote “buttons” form the backbone of how content is evaluated and subsequently ranked algorithmically, ostensibly to ensure that the “best” content rises to the top and becomes the most visible on the platform.

While ratings quantify the value of content and render it commensurable within a broader population (Graham, 2018), rankings sort and classify such value, curating it within a competitive attention economy (Ciampaglia et al., 2015). This may occur in an interactive or participative way through sorting mechanisms (e.g., a “sort by” button), but it can also occur through opaque algorithms (Pasquale, 2015) that aggregate ratings scores, perform calculations, and ultimately produce a ranking that (re)orders the visibility of content.

Rating and ranking devices on social media are governed by the interests of both platform operators and users (Ziewitz, 2013), mediated by platform architectures and algorithms (Gillespie, 2015), and powered by the crowd (Orlikowski & Scott, 2015a). As such, rating and ranking devices on social media circulate with the social values, ethics, rationalities, and meaning-making of both the platform designers who bring them into play (Flanagan et al., 2008), as well as the users who put them into practice. In this light, we appraise rating and ranking devices not simply as static representations of value, but as highly dynamic practices that are illustrative of sociomateriality. Sociomateriality recognizes “that the social and material are constitutively entangled in everyday life” (Orlikowski, 2007, p. 1437; emphasis original), and conceptualizing rating and ranking devices on social media as sociomaterial draws attention to how they are co-constitutively performed (Barad, 2003) and enacted (Suchman, 2007) with users.

The sociomaterial nature of rating and ranking devices on social media can be conceptualized as a material-discursive practice (Orlikowski & Scott, 2015b). According to Barad (2003), discursive practices and material phenomena are ontologically entangled. What this means for rating and ranking devices on social media is that these “buttons” are not merely unbiased digital artifacts that afford particular social or material outcomes. Rather, through practices, whether material (e.g., by clicking the “Like” button and contributing to a “Like” count) or discursive (e.g., talking about how the “Like” button ought [not] to be used), rating and ranking devices actively co-shape sociality as they contribute to the production of content on platforms. In this way, sociality is understood as inherently enmeshed with material realities.

Using Reddit as a platform case study, this research investigates the material-discursive practices of rating and ranking devices on social media. For Reddit, the binary rating device is an upvote/downvote option and it constitutes the life-blood of the platform’s business model and culture. Perhaps, supreme among its intended effects is the goal of creating the “front page of the internet” (Reddit.com). The platform achieves this by harnessing, or exploiting, practices of peer information aggregation (Leavitt & Robinson, 2017) to make the “best” content submitted to the platform rise to the top of its site. Indeed, the Reddiquette—Reddit’s “informal” guidelines—stipulates that the intended purpose of voting is “If you think something contributes to conversation, upvote it. If you think it does not contribute to the subreddit it is posted in or is off-topic in a particular community, downvote it” (Reddiquette, 2021). While existing scholarship has conceptualized voting on Reddit broadly in two ways—a metric to understand user engagement on the platform (Eschler & Menking, 2018; Glenski et al., 2017; Topinka, 2018) and/or an affordance of Reddit’s platform design (Davis & Graham, 2021; Gibson, 2019; Kilgo et al., 2018; Myles et al., 2020; Panek et al., 2018; Squirrell, 2019)—a gap remains to explore how users of Reddit, called Redditors, talk about voting in practice. By applying a sociomaterial lens, this exploratory article answers the research question, how do voting practices on Reddit materialize through discourse?

To answer this question, we sampled one year of comments on Reddit across 10 purposively sampled subreddits (theme-based community forums) and followed the keyword-based method of Maloney et al. (2019), to examine comments that mention Reddit’s upvote/downvote rating and ranking device—namely, “upvote,” “downvote,” and their variants, as detailed in the “Methods” section. Our analysis involved the following three stages: (1) topic modeling, (2) labeling and qualitative interpretation of topics through textual analysis, and (3) development of a conceptual framework. This conceptual framework is developed from what Redditors’ comments about upvoting/downvoting tell us about voting practices, in terms of the overarching themes of four clusters of topics. These four themes—platform culture, prescriptive device, materialization of value, and ontology of self—characterize voting as a material-discursive practice. We argue that this conceptual framework is theoretically generalizable (Davis & Love, 2019) to other rating and ranking devices on social media. Moreover, through our analysis, we discover that the upvote/downvote binary is constituted as two separate practices with different intended outcomes—an occurrence that is contrary to the guidelines of the platform mentioned earlier. In this article, we bring together social media studies with scholarship on sociomateriality and rating and ranking devices to map the material-discursive practices of upvoting/downvoting on Reddit, adding further evidence that rating and ranking devices on social media intervene in and perform sociality.

The Sociomateriality of Rating and Ranking Devices

Ratings and rankings on social media, such as Reddit’s upvote/downvote, are not passive features of a platform’s design but devices that actively intervene in creating sociality online. Rating and ranking devices on social media are mutually entailed (Barad, 2007) in their circulation with users and platform designers—in a word, they are sociomaterial. Sociomateriality, as the term suggests, bridges the divide between the social and material world by recognizing that materiality is created, interpreted, and used in social context so much as “all social action is possible because of some materiality” (Leonardi, 2012, p. 33). A rating provides a rubric to evaluate entities through a unary mechanism that gauges the level of popularity or engagement (e.g., number of “Likes”), a binary device that affords users to specify either positive or negative value (e.g., Reddit’s upvote/downvote) or a nominal scale (e.g., Amazon’s 5-star system). Whereas, a ranking orders a collection of individuals, organizations, artifacts, or content based on an evaluation of others or a proprietary algorithm.

When ratings and rankings are combined, they enact in practice (e)valuation, in that a metric or position (i.e., a valuation) is attributed based on a rubric (a process of evaluation; Jarke, 2017). Rating and rankings are often positioned through an economic lens of rational decision-making (Ziewitz, 2013). They offer commensuration to different entities (Pollock, 2012) and thus are an “objectified transferable commodity” (Gherardi, 2000, p. 13). However, as Jarke (2017) highlights, while this is useful in making “transient, social relations tangible and translates local and situated accounts into standardised objects” (p. 372) and thus comparable, the hallmark of rating and rankings as objective tools ignores that they are sociomaterial (Orlikowski & Scott, 2015b).

Recognizing ratings and rankings as sociomaterial is to consider how they co-shape sociality and cannot be considered disjointly from, or in the absence of, users and algorithms they circulate with. Like other sociomaterial devices, such as standardized visualization tools in finance and economics (Pollock, 2012, p. 94), rating and ranking devices on social media do more than merely evaluate; they also intervene and perform the very thing they are purported to measure. These devices intervene through what Espeland and Sauder (2007, p. 21) call “mechanisms of reactivity” because they encourage people to reflect on the relationship between, for example, the number (rating) or position (ranking) and what this mirrors back in terms of meaning. Espeland and Sauder (2007) highlight that commensuration—the process of imposing criteria on a set of things so to compare them—changes the way that information circulates and is interpreted. The effect is that rating and ranking devices can transform the things being compared to resemble the criteria used. In this way, rating and ranking devices, particularly on social media, need to be conceptualized as sociomaterial because their materiality is bound up with, and intervenes in, sociality.

This research contributes to a growing body of literature that critiques the apparatuses, contexts, and effects of rating and ranking devices on social media through sociomaterial approaches. Woolgar and Neyland (2013) unpack the ways in which everyday sociality is governed through “very unremarkable objects and technologies” (p. 2). Arguably, rating and ranking devices on social media are unremarkable objects to users because they are ordinary, everyday, and pervasive. Whereas for the platform operators, rating—and by extension ranking—devices such as Facebook’s “Like” button participates in a political economy that inherently sustains the platform (Gerlitz & Helmond, 2013). Indeed, Couldry and van Dijck (2015, p. 3) critique that social media purport a “myth of us” (i.e., the sociality within social media) by harnessing the value generated from user interactions with the platform’s mechanisms of “counting and valuing”—the rating and ranking devices.

In particular, Orlikowski and Scott (2014, 2015a) consider how un-unremarkable rating and ranking devices are to social media users and explore how they intervene and shape sociality. Orlikowski and Scott employ sociomaterial approaches to rating and rankings on TripAdvisor and provide insight on the production of commensurability when performed online (Scott & Orlikowski, 2012), the practices and implications of online valuation (Orlikowski & Scott, 2014), and the algorithmic configuration produced by crowd-sourced rating and ranking mechanisms (Orlikowski & Scott, 2015a).

Our study extends Orlikowski and Scott’s (2015b) work by continuing to consider the sociomateriality of rating and ranking on social media, through examining the material-discursive practices of such devices on the social media platform Reddit. We respecify upvoting and downvoting on Reddit as a material-discursive practice by analyzing how Redditors themselves talk about and mention it organically in their everyday communication (i.e., commenting on posts and replying to each other). Whereas with TripAdvisor and other consumer-oriented comparison websites, the relations are clear (e.g., guests evaluating hotels), in the case of Reddit, the relations are performed and contextual. Thus, a major contribution of this article is to provide overwhelming evidence that the process of valuation does far more than simply sort and rank the best content. Rather, through the sociomaterial perspective, we contend that practices of rating and ranking are constitutive of the social world.

“The Front Page of the Web”: The Sociomateriality of Reddit’s Voting Practices

Reddit is a link-sharing, news aggregating social media platform that relies on Redditors to rate posts and comments on the site, ostensibly for the purpose of rendering, the “best” content visible. Reddit is a crowd-sourced bulletin board that comprises diverse cultures (Myles et al., 2020; Squirrell, 2019) and practices (Kilgo et al., 2018). Users submit links, images, and videos to the site and participate in theme-based discussions organized into localized communities called “subreddits.” To parse and rank the massive volume of user-generated content, Reddit relies on Redditors to upvote or downvote each post and comment, to sort and push the most “interesting” content to the top of each reply thread and webpage, while pushing the most “irrelevant” content into the platform’s abyss. Given its central role on the platform, voting practices on Reddit comprises the focus of this study.

Previous social media scholarship has described voting on Reddit as a feature (Squirrell, 2019), a function (Kilgo et al., 2018; Massanari, 2016), a sorting mechanism (Panek et al., 2018), a mode to self-moderate (Gibson, 2019), and a method to gain visibility (Rafail & Freitas, 2019). From a purely functional perspective, we consider Reddit’s upvote/downvote buttons as a binary rating device that produces two outputs. The first output is a ranking that intervenes in the visibility of the comment or post within a given thread—which is structured as hierarchical through a reply-tree—as well as the threads position on any of Reddit’s webpages. To help sort the most “relevant” content to the top of the reply-tree and webpage, Redditors can either “upvote” or “downvote” each post and reply once, based on a simple rubric. Namely, the Reddiquette mentioned earlier; upvote what should be more visible and downvote what should be less visible. In essence, upvoting and downvoting are intended to be two sides of the same coin. But this binary crowd-sourced ranking process also relies on a proprietary algorithm to make sense of the user-generated data (Orlikowski & Scott, 2015a).

Clicking either the upvote or downvote button produces a two-fold rating output called “points” and “karma.” “Points” is the numerical value attributed to each post or reply: +1 point is given per upvote and –1 point per downvote. The numerical value displayed on each post or reply is a “fuzzed” number of votes. Stoddard (2015) explains that “vote fuzzing” is a “measure to combat spam and manipulation” (p. 425). It is a ratio of the aggregate number of upvotes minus downvotes as ascribed by other Redditors. Whereas, “karma” is an individual rating that, according to Reddit, “reflects how much a user has contributed to the Reddit community” (Reddiquette, 2021). Redditors receive karma based on other users upvoting or downvoting their posts and replies. While karma is closer to a one-to-one ratio compared to points, karma is still an approximate value of upvotes minus downvotes. Regardless, a high-karma score is intended to symbolize that the Redditor contributes “valuable,” “interesting,” or “insightful” content to the platform (Reddiquette, 2021). In this way, karma and points are a positive evaluation that incentivizes and normalizes particular practices through gaming yet disciplinary means (Sauder & Espeland, 2009). Thus, voting on Reddit is not merely a form of commensuration to help other Redditors navigate the colossal information on the platform but a means to govern practices.

Previous research demonstrates that users of voting devices act based on the outputs. Muchnik et al.’s (2013) study on a “social news aggregation website” showed that the visibility of prior upvotes and downvotes “created significant bias in individual rating behaviour” (p. 647). Davis and Graham (2021) found that Redditors’ emotional expression is often influenced by whether their previous content was upvoted or downvoted, and moreover that downvoted content statistically receives more attention as measured by number of replies. Anecdotally, we get a sense that Redditors can act based on the output of upvoting or downvoting through the tactic of “brigading” where Redditors coordinate an attack of mass downvotes. Brigading is a form of harassment—demonstrative of Massanari’s (2016, p. 333) observation of Reddit’s “toxic technocultures” which are issue-based publics who utilize platform architecture to implicitly or explicitly harass others. Massanari (2016) describes how Reddit’s design, algorithm, governance structure, and platform policies support “toxic technocultures” where “retrograde ideas of gender, sexual identity, sexuality, and race . . . push against issues of diversity, multiculturalism, and progressivism” (p. 333). In particular, Massanari (2016) problematizes how the voting on Reddit can be used as a means of providing visibility (upvoting) to anti-feminist and misogynistic activity while brigading (downvoting) any counter activity. Importantly, this behavior directly omits the Reddiquette, described earlier (Reddiquette, 2021).

While Reddit has specific guidelines about how to use and interpret the upvoting/downvoting buttons, users are by no means beholden to follow the Reddiquette, to use the rating and ranking device in any particular way, or even at all. Brigading serves as a useful example of sociomateriality and the way that the discursive and the material are tangled up together; individual Redditors need to interpret the post or comment in question and internalize the standing of points (rating) to then make a choice to either join the downvoting brigade or counteract it by upvoting (ranking); then, Redditors necessarily must talk about practices of upvoting and downvoting on the thread in question or in other subreddits, to coordinate a (counter)response. It does not make sense to consider users and vote buttons separately—in practice, they are co-constitutive.

This enmeshment of the social and material—that is, ratings, rankings, Redditors, cultural practices, and algorithms—calls us to reconsider how voting practices on Reddit materialize through discourse. This is the core question that we explore in this article through an analysis of how Redditors mention and discuss the platform’s rating and ranking device in everyday vernacular on the site (i.e., comments). The next section details our data collection and analysis procedure implemented to investigate this research problem.

Methods

Data Collection

To answer the research questions specified earlier, we drew from Reddit comments data,¹ that is, the comments and replies that users submit on Reddit posts, along with relevant meta-data (vote score and subreddit posted in). At the time of writing, the complete Reddit data archive comprised over 600 gigabytes and hundreds of millions of comments (Baumgartner et al., 2020). As such, the following methodological approach is one way to analyze large-scale textual data on Reddit and explore how voting practices are enacted materially through discourse.

To make our analysis both feasible and analytically focussed, we sampled one year (all of 2018) of comments across 10 purposively sampled subreddits (see Table 1). We chose these particular subreddits to explore voting practices within broad comment environments. The first subreddits chosen were selected from previous literature on Reddit. These included r/ShitRedditSays (Massanari, 2016), r/MensRights (Rafail & Freitas, 2019), and r/ImGoingToHellForThis (Topinka, 2018). We chose to also include the subreddits r/AskFeminists and r/conspiracy based on preliminary manual observations. Not wanting to only examine discourse on voting practices in subreddits where disputation possibly reigns, we also sought out what we classify as mundane popular subreddits. These are subreddits that were listed (at the time of writing) within the top 10 “safe-for-work” subreddits (based on number of subscribers) as stipulated by redditlist.com: a subreddit itself, that lists the current trending subreddits. These subreddits were purposely chosen from the list of 10 and included (listed in order of most subscribers at the time of writing): r/funny, r/AskReddit, r/gaming, r/pics, and r/science.

Table 1.

Summary Statistics of Subreddits Analyzed in This Study.

Subreddit	Number of comments (containing vote-related keywords)	Number of subscribers^a
r/AskFeminists	948	31,173
r/AskReddit	262,746	25,075,514
r/conspiracy	29,423	1,014,541
r/funny	80,587	27,021,921
r/gaming	72,100	24,039,082
r/ImGoingToHellForThis	4,173	1,511,666
r/MensRights	5,378	240,806
r/pics	70,685	23,108,970
r/science	160	22,651,814
r/ShitRedditSays	715	133,423

As of 7 November 2019.

Following the keyword-based method of Maloney et al. (2019), we examined only the comments that specifically include keywords related to voting practices on Reddit. Through an iterative process, we refined the keywords to include only mentions of voting that specifically relate to the “up” and “down” voting using Reddit’s binary rating device, namely, “upvote,” “up vote,” “up-vote,” “upvoting,” “up-voting,” and “up voting,” and reciprocally “downvote,” “down vote,” “down-vote,” “downvoting,” “down-voting,” and “down voting.” We evaluated this through a manual analysis of a small random sample of comments that include one or more of these vote-related keywords and found that the resulting dataset captured the intended phenomenon, namely comments that explicitly reference voting practices on Reddit.

Data Analysis

Our study combines computational and qualitative analytical approaches. Given that the dataset contains over half a million text comments, we used a computational approach known as Latent Dirichlet Allocation (LDA) topic modeling (Blei et al., 2003) to help us make sense of the underlying themes and summarize these into a small number of “topics” that would provide the basis for our qualitative analysis and interpretation.

Informally, LDA models summarize large text collections by organizing the documents into a small number of topics that help us extract and make sense of the underlying themes in the data. Documents (in this case, Reddit comments) are quantified by the probability that they are “generated” by each topic in the model. Each document is a probability distribution over topics, that is, a mix of topics with probabilities that sum to 1. Likewise, the topics in LDA models are a probability distribution of words.²

The output of the LDA model—a set of unlabelled topics that are now associated with particular words, and a set of documents that are now associated with particular topics—provided the basis to qualitatively interpret each topic and to label and describe the topics that underpin the text collection of Reddit comments. To assign labels to each topic, we extracted example comments that had a high probability of belonging to that topic (greater than 0.6). We also extracted the top 20 words associated with each topic. Next, we undertook a textual analysis of the example comments and top terms, cross-referencing the analysis by looking directly at the Reddit threads where the comments were situated in context. This process resulted in a label for each topic and a brief description summarizing it.

Finally, we used a dimensionality reduction technique to visualize the similarities between topics and group them into a small number of clusters. To do this, we synthesized the qualitative analysis of the topics with a quantitative visualization approach known as the Intertopic Distance Map (Sievert & Shirley, 2014). The overarching themes derived from the qualitative analysis of the 30 topics were mapped onto their respective positions on the Intertopic Distance Map (see Figure 1). We then revisited the example comments within each topic and analyzed the patterns occurring among clustered topics. In this second iteration of textual analysis, it became clear that despite one well-defined theme in each topic, outlier comments that fit within neighboring topics existed within the corpus of each topic. This finding encouraged us to rethink the Intertopic Distance Map not in terms of the quadrant axis but as clusters, like those highlighted on the map (see Figure 1). This, in turn, produced a bird’s-eye perspective to analyze the themes and develop a conceptual framework about rating and ranking devices on social media, based on four thematic clusters which include platform culture, prescriptive device, materialization of value, and ontology of self.

Figure 1.

Intertopic distance map showing the thematic clusters of the conceptual framework.

Conceptual Framework

The four themes of the conceptual framework and their respective topics are listed in Table 2. The topics listed under each theme of the conceptual framework describe voting practices on Reddit. Analyzing the comments within each topic and developing this conceptual framework allowed us to answer the research question of this article and thereby explore how “specific materialisations of discourse make a difference in practice . . . and with what performative consequences” (Orlikowski & Scott, 2015b, p. 704) in respect to voting on Reddit.

Table 2.

Topic Labels Sorted by the Four Thematic Clusters of the Conceptual Framework.

Platform culture
#4 Politics and “truth”	#6 Deliberation how to vote 1	#13 Knowledge production	#17 Toxic technocultures (Massanari, 2016)
#18 Emotionally valent	#20 Voting as validation	#21 Deliberation how to vote 2	#28 Geek masculinity (Braithwaite, 2016)
Prescriptive device
#1 Acquiring capital	#5 Deserving votes	#7 Deterministic expression	#8 A means to govern (bots)
#10 Wanting to give and wishing for more, upvotes	#12 Indeterminate about voting	#23 Questioning voting outcome	#27 Conspicuous voting
Materialization of value
#2 Governing subreddits	#3 Temporal dynamics of Reddit	#14 Age, dates, numbers, statistics, and time	#22 Popularity
#24 Karma farming	#25 Giving and getting votes	#26 Content being seen (reposts, ranking)
Ontology of self
#9 Not adhering to the Reddiquette	#11 Practices of personal voting	#15 Practice of self	#16 Tool for self-governance
#19 Personal ethics of voting	#29 Second-order observation (Esposito & Stark, 2019)	#30 Voting as relational

Platform Culture: How Voting Practices Perform Reddit Culture

The platform culture cluster (Figure 1) captures the various ways that voting co-constitutes and performs cultural practices on the site. Voting emerges as inseparable to how Redditors interact with one another and make meaning. We observe how the upvote button affords a highly visible and quantified marker of who and what is “right” in comment threads, while at the same time, generating and magnifying arguments as the upvotes pile up and users weigh in to vocalize their disagreement and outrage with how “the people [Redditors] have spoken” (Striphas, 2010 cited in Orlikowski & Scott, 2015a, p. 21). Our analysis shows that Redditors talk about upvoting and downvoting as a tool to leverage an outcome and attempt to govern the conduct of other Redditors. For example, we observe a common discourse that positions upvoting as a bargaining chip: “Here’s an upvote. Now go away and never come back, please” (r/AskReddit, Topic #4), and downvoting as a marker of the validity of a claim: “ . . . The more downvotes I get, the more I know I was right” (r/pics, Topic #4).

Our findings support previous research on practices of downvoting, as it is used as a suppression tool, to “brigade” (Massanari, 2016) viewpoints that challenge valorized geek masculinities (Braithwaite, 2016) and localized norms on subreddits. The comments in Topic #17 supports Massanari’s (2016) argument that Reddit’s voting practices support toxic technocultures. Many of these are comments in response to brigading, for instance,

Gross Edit: of course I get downvotes because Reddit is full of man children. (r/pics; Topic #17)

You’re being downvoted by a contingent of men who don’t like it when women take control of their own sexuality. (r/funny; Topic #17)

Yet, to a lesser extent, we also see how downvoting practices enact resistance to toxic masculinities on the site (Maloney et al., 2019), whereby Redditors express concern with how problematic discourses are suppressed and counteracted in a material-discursive manner through downvoting. For example, the comments below from Topic #17 show how misogynistic comments are sometimes met with resistance in the form of mass vote suppression, or what we might consider a sort of ethical inversal of brigading that challenges rather than supports toxic technocultures:

You have pretty face. And a plump body. Are you irish? Why the downvote, cunts? (r/gaming; Topic #17)

don’t worry about the downvotes its a well accepted fact that most men find women less attractive if they already have kids. (r/pics; Topic #17)

Topic #18 (emotionally valent) further reveals how voting practices on Reddit is more than merely a means to sort content; upvoting/downvoting is a device that produces an affect on the social world (Pollock, 2012). We observe comments that explicitly read as an emotional response, as well as employing offensive language and themes in their argumentation. For example, one Redditor commented that they were “embarrassed at how hard I laughed at that. Take your upvote” (r/funny), while another commented, “ALSO FUCK YOU TO WHOEVER DOWNVOTED YOU KNOW THIS POST IS SHIT” (r/pics). The emotional valence within the comments of this topic is illustrative of how the platform’s “features are objects of intense feelings” (Bucher & Helmond, 2017, p. 2). The very tool that is imagined to produce consensus and rank content is also a device that shapes and drives disputes on the site.

To illustrate this further, we highlight Topic #14, which contains comments about politics and “truth.” Thematically, we observe comments about US politics (mentions of former US Presidents Barack Obama and Donald J. Trump) as well as commentary about being “right” and “speaking the truth.” In particular, commentary about being “right” and “true” is positioned as something in tension with being downvoted. For example, “Reddit proves itself over and over. Truth is downvoted more often” (r/AskReddit) and “You’re right. And the fact that you dot [sic] downvoted just proves that you’re right” (r/AskReddit).

As we have described earlier, the Reddiquette instructs Redditors to merely downvote uninteresting content and restrain from using the rating and ranking device as a dislike mechanism. Yet, in our analysis, this instruction is found to be largely absent in practice as Redditors comment on either receiving or anticipating to receive downvotes due to the crowd disliking the comment in question. Thus, in addition to its role in brigading practices, we find that this is symptomatic of ontology; downvoting practices are inextricably tied up with what is permitted to pass as “truth” in discourse. The key takeaway is that Reddit culture is also voting culture, or rather, the two co-constitute one another and thus produce performative consequences such as brigading.

Prescriptive Device: Mechanistic Expression, Bots, and Reddiquette

Given the explicit role of Reddiquette guidelines for platform governance, we would expect to find a proportion of comments that speak to the intended purpose of the upvoting/downvoting on the platform. Indeed, we find that the prescriptive device theme captures a comparatively small set of topics and comments that are broadly in line with Reddit’s intended purpose of voting that prescribes it as a functional device for content moderation. Interestingly, many of the comments for topics in this theme are duplicated and repetitive, often reflecting mechanistic expressions that encode performance of gender identity on the site, such as many variations of,

I’m a simple man, I see something I like and I upvote. (r/AskReddit, Topic #7)

I’m a simple man. I see a downvote dogpile, I jump on. (r/gaming, Topic #7)

But more so, this is evident through the high-volume comment activity of bot accounts (i.e., automated, software-controlled accounts) that play various functional roles on the site such as auto-moderation and providing timestamped links to embedded videos:

[ **Jump to 01:59 @** Fergie Performs The U.S. National Anthem / 2018 NBA All-Star Game] . . . ^^Downvote ^^me ^^to ^^delete ^^malformed^^comments . . . . (r/funny, Topic #7)

Here’s a sneak peek of /r/MGTOW using the [top posts] . . . ^^I’m ^^a ^^bot, ^^beep ^^boop ^^| ^^Downvote ^^to ^^remove ^^..|. (r/MensRights, Topic #8)

These accounts write comments that include boilerplate instructions such as “downvote to remove,” suggesting an unproblematic, functional perspective of voting in its capacity to rank and moderate content through the crowd (Orlikowski & Scott, 2015a). In some ways, this is the closest reflection of the Reddiquette guidelines we could find in our analysis (i.e., if the bot is not contributing to the discussion, downvote it), although it represents only a small proportion of comments in the overall dataset.

Perhaps, the irony of Reddiquette is that it mostly comes into effect only for non-human participants on the site (i.e., bot moderators) and mechanistic expression that arguably contributes little to the discussion (contrary to the prescribed Reddiquette that encourages the reward of comments that contribute to the discussion). What we do see in effect of the Reddiquette is Redditors’ internalize the prescriptive nature of its rating and ranking device. For example, “without me, my upvote is useless. Without my upvote, I am useless” (r/funny, Topic #10). Indeed, another example of the sociomaterial relationship between voting and Reddit culture.

Materialization of Value: The Outcome of Voting Practices

Reddit operates within a competitive attention economy (Ciampaglia et al., 2015; Gerlitz & Helmond, 2013) where the threat of invisibility on the platform is modulated by the accruement (popularity) and timing of voting practices. This is embodied in the four emblematic topics in the bottom-right cluster on the map (see Figure 1), namely Topic #3 (temporality), Topic #22 (popularity), Topic #24 (karma farming), and Topic #26 (content being seen). We argue that this cluster is constitutive of the performative consequences that result from the materialization of value—the numerical values (“points” and “karama”) and ranking produced from voting practices. Through Topic #3, we observe an overarching concern about inexplicable discrepancies in what content gets seen and upvoted, which in turn centers around produsage (Bruns, 2006) and the socio-cultural memory on the platform:

I’m so fucking salty, I posted this a month ago and got 0 upvotes and you do it and is one of the top post [sic]. (r/AskReddit)

I had a similar comment in a similar AskReddit thread the other day and got downvotes. This is a weird place. (r/AskReddit)

Related to this is a concern with the attention economy for “reposts,” where identical content keeps turning up on the front page of subreddits and sometimes even on /r/all (Reddit’s front page). The karma economy of Reddit follows a power law distribution, where most content attracts few (or no) upvotes and it is often recycled content (i.e., reposts) that receives massive engagement through preferential attachment effects (i.e., “the rich get richer”; c.f. Barabási & Albert, 1999). Indeed, we identify an entire topic (Topic #26) about “being seen” due the outcome of voting practices. What emerges from the analysis of this thematic cluster is discourse by Redditors that describes the materialization of value produced through voting practice, which is to say, a ranking position on a reply-tree or webpage that renders the post highly visible. For example,

This is the fifth time it has been posted in 9 days. Most recently it was posted 5 days ago and got nearly 5000 upvotes. This is more than a little egregious. (r/gaming, Topic #3)

WHY DOES THIS POST GET SO MANY UPVOTES WHEN IT IS BEING REPOSTED FOR THE BILLIONTH TIME. (r/gaming, Topic #26)

This brings to bear Redditors’ awareness of the relationship between the valuation produced from voting practices, which has ties to temporality (Topic #3) and popularity (Topic #22). Reposting before anyone else and getting in early on comment threads is crucial to attracting upvotes—an implicit awareness, perhaps, of preferential attachment effects on the platform. For comments, participating late on a thread or being swiftly downvoted means that the comment gets “buried” and thus rendered practically invisible within the long tail at the bottom of the thread. The material value generated through voting practices therefore is not only consequential, but is performative of Reddit’s culture (Espeland & Sauder, 2007; Gillespie, 2015; Pollock, 2012). For example, we observe Redditors deliberating about why particular posts garner the votes they do, such as “low effort” content or “trash” receiving an inexplicably high number of upvotes:

I posted literally an image of the letter J and it got over 4.9k upvotes. (r/AskReddit; Topic #3)

The really infuriating part is that the trash somehow gets upvoted. (r/AskReddit; Topic #22)

This discourse would not be possible without the materialization of value that the rating and ranking device (upvoting/downvoting) produces. In this way, we can think about the outputs produced from voting practices, as something that performs culture, moderation, ethics, and self, rather than something that purely measures. Similar to the above examples, we observe instances of self-reflexive reposting being rewarded with upvotes, as the following comment shows: “New karma farming strat: 1: Get upvotes for repost. 2: Get upvotes for calling own post a repost” (r/AskReddit, Topic #24). This further illustrates the sociomateriality of voting practices, particularly the “mechanisms of reactivity” (Espeland & Sauder, 2007, p. 21), whereby Redditors change their behavior in response to being evaluated, which in turn shapes the materiality of the platform architecture, and both are co-constituted through this infinite recursion. Redditors’ discourse about the materialization of value produced from voting practices demonstrates how deeply interconnected voting is to the culture of the platform. When the upvote/downvote (the value) is materialized, it is only made real through its co-constitutive ontology with Reddit culture.

Ontology of Self: Voting, Redditors, and Ethical Self-Conduct

We have so far shown that voting on Reddit is not a simple, objective rubric to rank content—on the contrary, it is a material-discursive practice that performs localized cultures and meaning-making on the site. Taking this further, the ontology of self theme (Figure 1) captures a set of topics that are primarily concerned with the relationship between voting practices and Redditors’ ethical self-conduct and sense of self. Topic #19 (personal ethics of voting) is important to our study because it is the most prevalent topic in the dataset. It reveals how voting not only mediates but also intervenes with how Redditors relate to themselves and others, and in this way, navigate and deliberate ethical conduct. For example, in a comment thread about mental health, one Redditor comments that people should only use the term “OCD” (obsessive compulsive disorder) as a self-description if they have had a formal diagnosis:

I cannot upvote this enough. I’m a therapist and I cannot stand it when people say stuff like this. We all have quirks and preferences and routines, but unless you’ve been diagnosed with OCD, Bipolar, etc please use a different term! (r/AskReddit)

In this prototypical example of Topic #19, the author declares that they “cannot upvote this enough” (the comment that they agree with) because it accords with their own values about the issue at stake. The fact that the author of this comment cannot upvote it enough shows an awareness that voting matters in a material sense, and that the practice of voting is constitutive of ethical self-conduct. This accords with Esposito and Stark’s (2019) proposition that ratings devices offer a useful framework to orientate oneself in a complex and uncertain world.

Of course, this same logic applies to downvoting, as we observe from the following example comment (Topic #19), where the author calls out another Redditor for spreading misinformation about the human immunodeficiency virus (HIV):

You do not get HIV from saliva. Seriously this person needs to be downvoted to prevent more people being misinformed. (r/AskReddit)

In effect, the author of this comment is calling for a low-scale “brigade” (Massanari, 2016), where they appeal to other Redditors in the thread to downvote the comment because they believe it is the right thing to do: “to prevent more people being misinformed.” Again, we observe a kind of ethics of voting at play. In this example, the user draws upon the downvote button as a tool of ethical conduct, as one tool among many to shape and manage subjectivity. It is a call to action that involves clicking the downvote button—something to be taken seriously, according to the author of this comment. Crucially, this also highlights how it is not always content, that is, evaluated by voting practices, but the people who produce the content—Redditors themselves are constituted through evaluative procedures. Once again, the “mechanisms of reactivity” (Espeland & Sauder, 2007, p. 21) are brought to bear; this reactivity to being evaluated induces a power effect whereby Redditors must act in certain ways to avoid (attract) downvotes (upvotes). Thus, Redditors appear attuned to the fact that it is a real person on the other end of a voting practice, an individual with their own values, aspirations, and self-identities. For example, the author of the following comment explicitly disambiguates the content of a comment with the person who sent it:

I just want you to know, I’m downvoting just for being such a lazy writer. I’m not downvoting you for your comment, which I didn’t bother to read. As you go through life, you’ll learn that it’s on you to persuade others to take you seriously and want to listen to you, not on them to indulge your laziness. (r/funny; Topic #11)

In doing so, the author of this comment draws on the downvote button as a technique to negatively evaluate the person as distinct from the comment (notably, which they “didn’t bother to read”) in an effort to shape their sense of self. They explicitly address the person qua Redditor (“I just want you to know”) and ascribe them an identity category (“a lazy writer”), then proceed to make a normative judgment about how the person should conduct themselves in the future. In this example, broadly typical of many comments in Topic #11, voting repeatedly emerges as a highly personal and ethical affair—far from the Reddiquette imaginary. The ontology of self theme of our conceptual framework thus offers a key takeaway; on Reddit at least, voting practices are bound up with identity and ethical conduct is materialized in practice through rating and ranking devices.

Reddit’s Upvote/Downvote Practices: Two Sides of Different Coins

The official Reddiquette guidelines stipulate that Redditors ought to employ voting based on a subjective but simple rubric: does this content contribute to the subreddit? Ostensibly, the guidelines configure users within a binary choice setting: yes (upvote) or no (downvote), and in this way, upvoting and downvoting are often regarded as two sides of the same coin. However, among the 30 topics, we only found a small portion of commentary (Topic #9) that validates the Reddiquette guideline. The remaining 29 topics, which were subsequently clustered and conceptually organized into the four themes as discussed previously, comprise a diverse vernacular that trivializes the platform guidelines—making upvoting and downvoting two sides of different coins.

For the most part, Redditors not only disregard the rules about voting but also make, and enforce, their own rules, norms, and ethics around it. Our analysis of the relationship between Redditors and voting therefore supports the idea that the two cannot be made sense of apart from one another. Reddit communication is a sociomaterial practice rather than a human-based activity, whereby voting—respecified as a material-discursive practice—performs the very thing it is purported to measure. As Gillespie (2015) argues, “platforms intervene, and the public culture that emerges from them is, in important ways, the outcome” (p. 2; emphasis added). Taking this argument a step further, the voting practices and Reddit culture are mutually entailed on an ontological level (Orlikowski & Scott, 2015b), that in practice means that they “lack an independent, self-contained existence” (Barad, 2007, p. ix). The reality of Reddit is thus the reality of material-discursive voting practices.

Conclusion

This article examined voting practices on Reddit through a textual analysis of over half a million user-generated comments on the platform. To answer the research question, we investigated voting practices on Reddit in terms of how they materialize through Redditors’ discourse, that is, explicit references to it “in the wild” within comment threads. We found that Redditors appraise voting in a radically different and more complex way than its intended use by the platform operators. More importantly, our analysis comprehensively showed how Reddit culture and voting are discursively inseparable from one another. We advanced this thesis through the development of a conceptual framework that explores the sociomateriality of rating and ranking devices on social media through the following four themes: platform culture, prescriptive device, materialization of value, and ontology of self. Through this framework, we showed how Reddit culture and Redditors’ self-identities are “materially enacted in practice” (Orlikowski & Scott, 2015b, p. 700) through their entanglement with the platform’s rating and ranking device.

Like all research, this study has limitations along with departure points for future work. By focussing on a single platform, the findings of this study are restricted in their generalizability. It is not clear to what extent the empirical insights of this work extend to other platforms, particularly where binary ratings devices are not as central or even absent from the platform architecture. Despite this, we believe that this work obtains what Davis and Love (2019) term “theoretical generalizability” in the formal theory tradition. The conceptual framework developed in this article offers a theoretical proposition that tests and significantly extends our understanding of how ratings and ranking devices are an inextricable part of discourse on social media platforms. We encourage further work to investigate whether, and to what extent, the theoretical propositions of this article apply to other platforms and different kinds of ratings and rankings devices. We also acknowledge that this study did not include all of Reddit—it focussed on 10 purposively selected subreddits relevant to scholarship and to address sample bias due to the heavily skewed popularity for particular subreddits and types of content. However, future work could scale up the data collection to further explore the material-discursive practices of voting on Reddit and how it differs for particular topics, communities, and/or subreddits of interest. We also highlight the limitations of focussing on user comments as a data source. Although the research question of this article was geared toward such data, we invite future research to examine the role of algorithms and habitual practices, as well as other relevant methodologies such as online ethnographies.

To imagine ratings and rankings devices on social media as objective tools is to ignore their ontological entanglement with online culture and selfhood. They do not measure the world so much as they perform it. Recognizing the sociomateriality of these technologies is to apprehend how they co-shape who we are, how we relate to each other, and where we are going. As ever, it is critical that we design our social media spaces knowing the ways that they, too, design us.

Footnotes

Authors’ note

Of note, authorship was equal and names are listed alphabetically.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Aleesha Rodriguez

Notes

Author biographies

Timothy Graham is Senior Lecturer in the School of Communication, Queensland University of Technology. His research examines platform architectures and online networks, with a particular interest in online mis- and disinformation and ratings and rankings devices.

Aleesha Rodriguez is a PhD candidate at the Digital Media Research Centre (DMRC), Queensland University of Technology. Her research examines public communication on social media platforms, with a particular interest in applying Science and Technology Studies (STS) approaches to digital media studies.

References

Barabási

Albert

(1999). Emergence of scaling in random networks. Science, 286(5439), 509–512. https://doi.org/10.1126/science.286.5439.509

Barad

(2003). Posthumanist performativity: Toward an understanding of how matter comes to matter. Signs: Journal of Women in Culture and Society, 28(3), 801–831. https://doi.org/10.1086/345321

Barad

(2007). Meeting the universe halfway: Quantum physics and the entanglement of matter and meaning (2nd Printing ed.). Duke University Press Books.

Baumgartner

Zannettou

Keegan

Squire

Blackburn

(2020, May). The pushshift Reddit dataset. In Proceedings of the International AAAI Conference on Web and Social Media (Vol. 14, pp. 830–839). https://ojs.aaai.org/index.php/ICWSM/article/view/7347/7201

Blei

D. M.

A. Y.

Jordan

M. I.

(2003). Latent Dirichlet allocation. Journal of Machine Learning Research, 3, 993–1022.

Braithwaite

(2016). It’s about ethics in games journalism? Gamergaters and geek masculinity. Social Media + Society, 2(4), 1–10. https://doi.org/10.1177/2056305116672484

Bruns

(2006). Towards produsage: Futures for user-led content production. In Ess

Sudweeks

Hrachovec

(Eds.) Proceeding of the 5th International Conference on Cultural Attitudes towards Technology and Communication (pp. 275–284). Queensland University of Technology.

Bucher

Helmond

(2017). The affordances of social media platforms. In Burgess

Marwick

Poell

(Eds.), The Sage handbook of social media (pp. 223–253). SAGE.

Ciampaglia

G. L.

Flammini

Menczer

(2015). The production of information in the attention economy. Scientific Reports, 5, 9452. https://doi.org/10.1038/srep09452

10.

Couldry

Van Dijck

(2015). Researching social media as if the social mattered. Social Media+ Society, 1(2), 1–7. https://doi.org/10.1177/2056305115604174

11.

Davis

J. L.

Graham

(2021). Emotional consequences and attention rewards: The social effects of ratings on Reddit. Information, Communication & Society, 24(5), 649–666. https://doi.org/10.1080/1369118X.2021.1874476

12.

Davis

J. L.

Love

(2019). Generalizing from social media data: A formal theory approach. Information, Communication & Society, 22(5), 637–647. https://doi.org/10.1080/1369118x.2018.1555610

13.

Eschler

Menking

(2018). “No prejudice here”: Examining social identity work in starter pack memes. Social Media + Society, 4(2), 1–13. https://doi.org/10.1177/2056305118768811

14.

Espeland

Sauder

(2007). Rankings and reactivity: How public measures recreate social worlds. American Journal of Sociology, 113(1), 1–40. https://doi.org/10.1086/517897

15.

Esposito

Stark

(2019). What’s observed in a rating? Rankings as orientation in the face of uncertainty. Theory, Culture & Society, 36(4), 3–26. https://doi.org/10.1177/0263276419826276

16.

Flanagan

Howe

D. C.

Nissenbaum

(2008). Embodying values in technology: Theory and practice. In van den Hoven

Weckert

(Eds.), Information technology and moral philosophy (pp. 322–415). Cambridge University Press.

17.

Gerlitz

Helmond

(2013). The like economy: Social buttons and the data-intensive web. New Media & Society, 15(8), 1348–1365. https://doi.org/10.1177/1461444812472322

18.

Gherardi

(2000). Practice-based theorizing on learning and knowing in organizations. Organization, 7(2), 211–223. https://doi.org/10.1177/135050840072001

19.

Gibson

(2019). Free speech and safe spaces: How moderation policies shape online discussion spaces. Social Media + Society, 5(1), 1–15. https://doi.org/10.1177/2056305119832588

20.

Gillespie

(2015). Platforms intervene. Social Media + Society, 1(1), 1–2. https://doi.org/10.1177/2056305115580479

21.

Glenski

Pennycuff

Weninger

(2017). Consumers and curators: Browsing and voting patterns on Reddit. IEEE Transactions on Computational Social Systems, 4(4), 196–206. https://doi.org/10.1109/tcss.2017.2742242

22.

Graham

(2018). Platforms and hyper-choice on the World Wide Web. Big Data & Society, 5(1), 1–12. https://doi.org/10.1177/2053951718765878

23.

Jarke

(2017). Community-based evaluation in online communities. Information Technology & People, 30(2), 371–395. https://doi.org/10.1108/itp-03-2015-0046

24.

Kilgo

D. K.

Y. M. M.

Riedl

M. J.

Lacasa-Mas

(2018). Reddit’s veil of anonymity: Predictors of engagement and participation in media environments with hostile reputations. Social Media + Society, 4(4), 1–9. https://doi.org/10.1177/2056305118810216

25.

Leavitt

Robinson

J. J.

(2017, February). The role of information visibility in network gatekeeping: Information aggregation on Reddit during crisis events. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing (pp. 1246–1261). https://doi.org/10.1145/2998181.2998299

26.

Leonardi

P. M.

(2012). Materiality, sociomateriality, and socio-technical systems: What do these terms mean? How are they different? Do we need them. In Leonardi

P. M.

Nardi

B. A.

Kallinikos

(Eds.), Materiality and organizing: Social interaction in a technological world (pp. 25–48). Oxford University Press.

27.

Maloney

Roberts

Graham

(2019). Gender, masculinity and video gaming: Analysing Reddit’s r/gaming community. Springer Nature. https://doi.org/10.1007/978-3-030-28262-2

28.

Massanari

(2016). #Gamergate and The Fappening: How Reddit’s algorithm, governance, and culture support toxic technocultures. New Media & Society, 19(3), 329–346. https://doi.org/10.1177/1461444815608807

29.

Muchnik

Aral

Taylor

S. J.

(2013). Social influence bias: A randomized experiment. Science, 341(6146), 647–651. https://doi.org/10.1126/science.1240466

30.

Myles

Benoit-Barné

Millerand

(2020). “Not your personal army!” Investigating the organizing property of retributive vigilantism in a Reddit collective of websleuths. Information, Communication & Society, 23(3), 317–336. https://doi.org/10.1080/1369118x.2018.1502336

31.

Orlikowski

W. J.

(2007). Sociomaterial practices: Exploring technology at work. Organization Studies, 28(9), 1435–1448. https://doi.org/10.1177/0170840607081138

32.

Orlikowski

W. J.

Scott

(2014). What happens when evaluation goes online? Exploring apparatuses of valuation in the travel sector. Organization Science, 25(3), 868–891. https://doi.org/10.1287/orsc.2013.0877

33.

Orlikowski

W. J.

Scott

S. V.

(2015a). The algorithm and the crowd: Considering the materiality of service innovation. Management Information Systems Quarterly, 39(1), 201–216. https://doi.org/10.25300/MISQ/2015/39.1.09

34.

Orlikowski

W. J.

Scott

S. V.

(2015b). Exploring material-discursive practices. Journal of Management Studies, 52(5), 697–705. https://doi.org/10.1111/joms.12114

35.

Panek

Hollenbach

Yang

Rhodes

(2018). The effects of group size and time on the formation of online communities: Evidence from Reddit. Social Media + Society, 4(4), 1–13. https://doi.org/10.1177/2056305118815908

36.

Pasquale

(2015). The black box society. Harvard University Press.

37.

Pollock

(2012). Ranking devices: The socio-materiality of ratings. In Leonardi

P. M.

Nardi

B. A.

Kallinikos

(Eds.), Materiality and organizing: Social interaction in a technological world (pp. 92–112). Oxford University Press.

38.

Rafail

Freitas

(2019). Grievance articulation and community reactions in the men’s rights movement online. Social Media + Society, 5(2), 1–11. https://doi.org/10.1177/2056305119841387

39.

Reddiquette. (2021). Reddit help. https://www.reddithelp.com/hc/en-us/articles/205926439

40.

Röder

Both

Hinneburg

(2015, February 2–6). Exploring the space of topic coherence measures. In Proceedings of the Eighth ACM International Conference on Web Search and Data Mining (pp. 399–408). https://doi.org/10.1145/2684822.2685324

41.

Sauder

Espeland

(2009). The discipline of rankings: Tight coupling and organizational change. American Sociological Review, 74(1), 63–82. https://doi.org/10.1177/000312240907400104

42.

Scott

S. V.

Orlikowski

W. J.

(2012). Great expectations: The materiality of commensurability in social media. In Leonardi

P. M.

Nardi

B. A.

Kallinikos

(Eds.), Materiality and organizing: Social interaction in a technological world (pp. 114–133). Oxford University Press.

43.

Sievert

Shirley

(2014, June 27). LDAvis: A method for visualizing and interpreting topics. In Proceedings of the Workshop on Interactive Language Learning, Visualization, and Interfaces (pp. 63–70). https://doi.org/10.3115/v1/W14-3110

44.

Squirrell

(2019). Platform dialectics: The relationships between volunteer moderators and end users on Reddit. New Media & Society, 21(9), 1910–1927. https://doi.org/10.1177/1461444819834317

45.

Stoddard

(2015, May 26–29). Popularity dynamics and intrinsic quality in Reddit and hacker news. In Ninth International AAAI Conference on Web and Social Media (pp. 416–425). https://ojs.aaai.org/index.php/ICWSM/article/view/14636

46.

Striphas

(2010, June 14). How to have culture in an algorithmic age. The Late Age of Print Blog. http://www.thelateageofprint.org/2010/06/14/how-to-have-culture-in-an-algorithmic-age/

47.

Suchman

L. A.

(2007). Human-machine reconfigurations: Plans and situated actions. Cambridge University Press. https://doi.org/10.1017/CBO9780511808418

48.

Topinka

(2018). Politically incorrect participatory media: Racist nationalism on r/ImGoingToHellForThis. New Media & Society, 20(5), 2050–2069. https://doi.org/10.1177/1461444817712516

49.

Woolgar

Neyland

(2013). Mundane governance: Ontology and accountability. Oxford University Press.

50.

Ziewitz

(2013). Evaluation as governance: The practical politics of reviewing, rating and ranking on the web [Doctoral thesis]. University of Oxford.