Abstract
As online platforms grow, they find themselves increasingly trying to balance two competing priorities: individual rights and public health. This has coincided with the professionalization of platforms’ trust and safety operations—what we call the “customer service” model of online governance. As professional trust and safety teams attempt to balance individual rights and public health, platforms face a crisis of legitimacy, with decisions in the name of individual rights or public health scrutinized and criticized as corrupt, arbitrary, and irresponsible by stakeholders of all stripes. We review early accounts of online governance to consider whether the customer service model has obscured a promising earlier model where members of the affected community were significant, if not always primary, decision-makers. This community governance approach has deep roots in the academic computing community and has re-emerged in spaces like Reddit and special purpose social networks and in novel platform initiatives such as the Oversight Board and Community Notes. We argue that community governance could address persistent challenges of online governance, particularly online platforms’ crisis of legitimacy. In addition, we think community governance may offer valuable training in democratic participation for users.
Introduction
Since the earliest days of computing, people have used information technology to converse with one another. Four years before the internet, Noel Morris and Tom Van Vleck wrote both an electronic mail system and a real-time chat system for MIT’s Compatible Time-Sharing System (CTSS), allowing users who logged onto the single shared computer to leave messages for one another or send messages to another user’s terminal (Van Vleck, 2012). Within 3 years of the introduction of the internet, email became the primary use of a network initially established to let computer scientists run programs on remote machines (Sterling, 1993). France’s Minitel service, designed to give users access to an electronic telephone directory and the ability to make travel reservations online, quickly became dominated by chat services, particularly erotic chat (Tempest, 1989). People want to talk to one another and will find ways to do so as soon as they are technically capable of connecting to one another.
Unfortunately, as soon as people are able to talk to one another, they are also able to harm each other. Spam has undermined the utility of email and largely destroyed Usenet, the dominant community platform of the academic internet in the 1980s and early 1990s. Harassment and hate speech have become facts of life for users of many online systems, particularly for women, people of color, and LGBTQIA+ people. People often behave differently online than they would offline (Suler, 2004) and the impetus for humans to harass each other via digital tools is at least as strong as the impulse to connect.
The emergence of trust and safety as a professional discipline reflects the centrality of issues like content moderation, spam and fraud prevention, and efforts to combat child sexual abuse imagery (CSAM) to the operation of platforms that enable user-generated content and conversation. As Tarleton Gillespie (2018) notes in Custodians of the Internet, “Platforms are not platforms without moderation.” Recent efforts to recognize trust and safety as a profession, with the establishment of the Trust & Safety Professional Association in 2020 and the emergence of a Journal of Online Trust and Safety in 2021 are overdue, as the work of policing online spaces traces back at least to the 1980s, if not earlier.
One danger of losing the early history of online governance is a narrowing of possible futures, making it seem as if the contemporary model for governing online spaces, where professionals make decisions about what behavior is acceptable, with little input from members of the community, is the way it’s always been done. We refer to this model as the “customer service” model and contrast it to earlier models of online governance in which community members were significant, if not always primary, decision-makers about the online spaces they were a part of. This article examines three paradigms of online governance that preceded the contemporary customer service model and suggests that varying degrees of community governance may be a viable and socially beneficial option for many online spaces.
This article is far from an exhaustive history of early online governance or of the emergence of the customer service model, though both histories are needed. While there has been excellent work calling attention to the complexities of trust and safety (Gillespie, 2018; Gray & Suri, 2019), it has focused primarily on the “web 2.0” social media platforms that emerged in the mid-2000s—the shift toward the customer service model begins in the late 1980s and is cemented in place by the mid-1990s. This is also an opinionated and personal history, as one of the authors (Zuckerman) built the early content moderation department for Tripod.com, one of the web’s first user-generated content sites, from 1995 to 1999.
Before the Web
Well before the World Wide Web became the dominant use for computer networking, user-generated content was the dominant form of material shared online. A major, if not the main, attraction of dial-up internet services like AOL and Compuserve was the ability to interact with other users in online fora and chat rooms. Spaces native to the early academic internet—Usenet groups, MUDs, and MOOs (multiuser virtual worlds)—dealt entirely with user-generated content, as the academic internet based around NSFNET was not open to commercial uses before 1991 (RFC 1192, 1990).
These online communities were rife with bad behavior: much of the language we use today to discuss bad behavior online—trolling, flaming, and spam—have their origins in early internet and bulletin board culture. In most online spaces, a system of governance emerged to address antisocial behavior. At least three paradigms are worth consideration as antecedents to contemporary models for online governance: norms-based, community-based, and strong sysop.
Usenet—Norms-Based Governance
Usenet was created in 1979 by graduate students Tom Truscott and Jim Ellis at Duke University, using a simple protocol (UUCP) to copy files between Unix systems, enabling the sharing of software and text files. By the late 1990s, Usenet connected people in more than 200 countries and accounted for one third of the data transfer of the internet (M. Smith, 1999).
Usenet groups were initially unmoderated, with the capacity for moderating a newsgroup added only in 1984. Architecturally, Usenet was designed to put most control over moderation primarily in the hands of a reader—heavy Usenet users grew adept at the use of a “killfile” filled with the names of topics and posters one wished to filter out. (“Killfiles—The Cure for all That Ails You,” 1994)
In addition to the blocking power of killfiles, Usenet users had a great deal of affirmative power in proposing new newsgroups. Motivated and technically knowledgeable news users could propose a new group by posting a “request for discussion” to news.announce.newgroups, which included a rationale for the new group and a charter for what the group would be used to discuss. A minimum of a 3-week discussion period ensued, followed by a call for votes. Newsgroups were created if they received a two thirds “Yes” vote and at least 100 more “Yes” than “No” votes (Paolillo & Heald, 2002). Ultimately, the power to create newsgroups rested in the hands of server administrators, but strong norms about democratic participation meant users were powerful in steering the evolution of Usenet discussions.
The limits of Usenet’s architecture were tested when one of the internet’s most prolific trolls, Serdar Argic, began using automated posting software (i.e. a “bot”) to respond to any mention of Turkey or Armenia across Usenet with long screeds denying the Armenian genocide. Argic posted an average of 100 posts a day, many inappropriately targeted (for example, in response to a discussion of a Thanksgiving turkey). While administrators could have decided to “cancel” all messages coming from Argic’s account, a strong norm against “third-party cancellation” as a form of censorship prevailed on Usenet, making this an untenable solution. A Usenet administrator eventually created a new newsgroup—alt.cancel.bots—which would automatically cancel Argic’s posts, but only if you chose to subscribe to the newsgroup (DeVoto, 1994).
Because the technical affordances of Usenet put so much control in the hands of an individual user, much of the governance of Usenet happened through strong community norms. Many Usenet newsgroups included an FAQ (Frequently Answered Questions) document, which was intended to prevent new users from asking repetitive, previously answered questions. “Netiquette,” a portmanteau of “network” and “etiquette” was likely coined on Usenet as a way of talking about the social rules that made discussions online less hostile and more constructive (Merriam–Webster).
The dependence of Usenet on netiquette is reflected in the story of “the eternal September.” Usenet users often complained about the quality of discourse online in September, when new students came to campus, got access to the internet, and began annoying existing users with their ignorance of netiquette. When AOL began providing Usenet access to millions of its customers in September 1993, experienced Usenet user Dave Fischer declared it “the September that never ended” (Koebler, 2015; E. Smith, 2020). While elitist and condescending, the notion of “eternal September” captured something real and true about Usenet’s vulnerabilities: a space governed primarily by strong norms was vulnerable to the excesses of rulebreakers.
Shortly after AOL welcomed a much larger userbase to Usenet, spam became a serious problem. Law firm Canter and Siegel posted an ad offering immigration services to 5,500 newsgroups, signaling the vulnerability of the community to cross-posted advertisements, which cluttered individuals’ newsfeeds and taxed Usenet servers. Some Usenet system administrators, who had previously respected the community’s strong free speech norms, began campaigns to block prolific spammers, while others argued that Usenet norms prohibited such systemwide actions (Miller, 2021).
The power of sysadmins to unilaterally block spammers was a reminder that all technologically mediated spaces experience a tension between the intentions of whoever technically controls the space and the community that chooses to use the space. Usenet administrators gave individuals a great deal of responsibility and control over their behavior, and put significant voting structures in play, yet found themselves making unilateral decisions when norm violators began making the space unusable. Usenet became a less culturally significant space in the mid-1990s both due to the rise of the graphical Web and due to spam problems.
MOOs and MUDs—Community-Based Governance
A rite of passage for most programmers is the creation of a game. Many programmers in the 1970s and 1980s wrote games influenced by the books of JRR Tolkien and the roleplaying game Dungeons and Dragons. These games are called “dungeon crawlers” and went through a rapid process of iteration on shared access minicomputers at universities in the late 1970s. In 1978, a student at the University of Essex in the United Kingdom created a dungeon that could be explored by multiple users, titled “Multi User Dungeon,” or MUD (Kelly & Rheingold, 1993).
MUDs became enormously popular on university campuses in the 1980s, with some joking that the acronym stood for “Multi-Undergraduate Destroyer,” as some students became so immersed in the virtual worlds that they failed to attend classes (“When Studying MUDs [. . .],” 1998). Some MUDs remained close to their dungeon crawling roots, while others focused on social elements, allowing users to chat, interact, and build new features of a virtual world together. One of the most popular of these social MUDs was LambdaMOO, hosted at Xerox’s Palo Alto Research Center and built by researcher Pavel Curtis in 1990. 1 LambdaMOO became a popular gathering spot for those interested in novel social interactions online and was widely documented by technology authors including Howard Rheingold (1993) and Sherry Turkle (1995).
LambdaMOO (which still exists as of late 2022) resembles a set of connected chat rooms, with some powerful features lurking beneath the surface. Each room has a textual description as does each character; as journalist Julian Dibbell (1993) put it, LambdaMOO is “a very large and very busy rustic mansion built entirely of words.” In addition to talking to one another, users of LambdaMOO can build new parts of the environment, creating new spaces and objects which can be coded to carry out behaviors. For example, a room could have a trap door—when a user pulls the lever, users in the room “fall” into another room. The combination of chat, textual description, and coding makes LambdaMOO a powerfully immersive space, despite lacking graphics.
Power over decision-making on LambdaMOO was held by Curtis and a small team of “wizards,” who had special administrative and technical powers beyond those of normal users. In early 1993, Curtis announced to the community (Mnookin, 1996): I realize now that the LambdaMOO community has attained a level of complexity and diversity that I’ve actually been waiting and hoping for since four hackers and I first set out to build this place: this society has left the nest . . . So, as the last social decision we make for you, and whether or not you independent adults wish it, the wizards are pulling out of the discipline/manners/arbitration business; we’re handing the burden and freedom of that role to the society at large . . .
While stating his desire to hand over control to users, Curtis did not propose an actual governance mechanism, and for a few months, the community largely ignored Curtis’s mandate.
In March 1993, the LambdaMOO community was confronted by an egregious case of sexual abuse, in which a user—Mr. Bungle—used a piece of code (a “voodoo doll”) to make players engage in actions without their consent. He targeted female-presenting users and forced their characters into violent and sexual situations. In an article describing the attack and its aftermath titled “A Rape in Cyberspace,” journalist Julian Dibbell (1993) examined how the LambdaMOO community wrestled with the complexities of virtual rape and sexual assault. A long discussion on LambdaMOO’s main mailing list converged around a decision to remove Mr. Bungle from the system, and a wizard deleted his account.
The Mr. Bungle affair accelerated the formation of governance processes at LambdaMOO. Dibbell (1993) describes ongoing arguments between those who wanted a formal legal system to govern the space, and those who took anarchist or libertarian stances. Jennifer Mnookin (1996) describes the rapid development of a complex governance system, colloquially referred to as Lambda Law, which includes a process for petitioning for socio-technical changes to be made to the space, and a detailed mediation process for conflicts between users. By 1996, Lambda MOO users had approved 44 of these petitions, making permanent changes to the affordances of the system, including establishing the mediation process (Mnookin, 1996).
As with Usenet, control over the technical systems rested with the “wizards” capable of controlling the code and the database. But to an even greater extent than with Usenet, those with technical authority over the platform chose to make that power subject to the will of the broader community.
BBS and Online Services—The Strong Sysop
Kevin Driscoll (2022b) suggests that early histories of internet community often overfocus on communities like Usenet and MUDs/MOOs, which were popular with academic audiences, at the expense of communities hosted on bulletin boards (BBS)—systems run by individuals or small teams from home computers connected to small banks of modems. In these worlds, the operator of the bulletin board—the sysop—was king or queen. But few of these monarchs saw themselves as omnipotent: instead, they were deeply attuned to the needs and wants of their communities, lest their users decide to stop logging on.
While these systems were often operated as commercial enterprises, Driscoll argues that they had a vast array of business models, from sysops who treated their bulletin board systems as an expensive hobby, like owning a vintage car, to sysops who leaned heavily on their communities for financial support as well as participation. Driscoll (2022a) explains further: There were also BBSs organized like a social club. Members paid ‘dues’ to keep the hard drive spinning. Others formed nonprofit corporations, soliciting tax-exempt donations from their users. Even on the hobby boards, sysops sometimes passed the virtual hat, asking everybody for a few bucks to buy a new modem or knock out a big telephone bill.
This dependency on users for financial support created a feedback loop between users and sysops. A sysop who failed to listen to her users was unlikely to find support to expand her system. More likely, users would leave for another board in the same area code. As a result, even boards that were noncommercial, hobbyist projects spent time and energy listening to their users. Driscoll (2022b, p. 148) discusses a board in Terre Haute, Indiana, called TARDIS, run by four friends as a hobby, which worked to “surprise and delight its callers.” TARDIS created a community popular with women (for whom they provided a “Ladies Only” area, as a helpful corrective to male-dominated online spaces) and visually disabled users (they kept their software simple and screen-reader compatible). The rewards for listening to user needs were “a dedicated core group of users who became fiercely loyal and personally invested in the board’s culture” (Driscoll, 2022b, p. 151) and who worked to promote the board and recruit like-minded callers.
The model of the all-powerful sysop, listening attentively to his users, is one that ended up influencing even the largest commercial systems. Internet scholar Jonathan Zittrain traces his fascination with online spaces to his time as a sysop on Compuserve, working to pay down his substantial online collection charges. Compuserve, Zittrain explains (personal communication, November 22, 2022), outsourced community management duties to a set of third-party contractors, who were paid according to how much traffic their boards generated. Within the context of making their boards popular and highly trafficked, sysops had a great deal of flexibility in how they carried out their duties. Some hired deputies and paid them in free time on the system or with a share of their earnings. Others did as little work as possible.
Zittrain (1997) notes that many sysops found that the most productive ways to manage their communities involved constraining their authoritarian impulses: When intervention by a sysop takes place, it often takes place thoughtfully, with notions like due process or tolerance of distasteful speech generated on the fly or cobbled together from the Western cultural and legal landscape that still looms large on today’s Internet.
In this, sysops were anticipating the free speech paradigm that many large social media platforms initially professed. But they also emphasized the need for a justifiable, defensible system thanks to the dynamics of BBSs: an unfairly governed forum is one users will leave, costing the sysop earnings, or perhaps her job.
The relationship between a strong sysop and her community is not a democratic model. It is closer to a benevolent dictatorship. But much as Rebecca MacKinnon (2012) suggests in Consent of the Networked, a monarchy that recognizes a set of basic rights for citizens is healthier and more stable than one that seeks that its serfs be silent. The bureaucratic systems of governance that quickly emerged in the early years of the Web (1994–2000) often abandoned the recognition that users of online communities had a critical role in their creation, maintenance, and governance.
From Communities to Bureaucracies
The early web grew with a speed that was often shocking to those building its key infrastructures. Our World In Data (Roser et al., 2015) estimates the total population of internet users in 1990 (the heyday of Usenet) at 2.6 million, in 1995 (early days of the commercial web) at 44.4 million, and 2000 (the bursting of the first internet “bubble”) at 412.8 million.
Zuckerman was part of the founding team that built Tripod.com, a website that gained popularity by offering free internet homepages to registered users. Traffic to these homepages meant that Tripod was one of the 10 most trafficked websites in the late 1990s, as well as one of the first ad-supported user-generated content businesses. 2 Tripod was wholly unprepared for the challenges of managing an online community: it began as an edited, magazine-like content site aimed at recent college graduates and added homepage building as an afterthought. Discovering that users would use their webpages to share pornography, violent imagery, and pirated software was obvious in retrospect but surprising to those involved with Tripod as the site grew.
Zuckerman (who is not a lawyer) wrote the terms of service for Tripod in late 1995 both to signal what behaviors would be prohibited on Tripod and in the hopes of creating a form of due process before Tripod’s “abuse” team (a branch of the customer service department) began systematically removing user homepages and canceling accounts. Tripod had not anticipated problems like CSAM, and when CSAM was uploaded to the site, Tripod’s team had to figure out the mechanics of reporting it to the FBI. The FBI requested that CSAM not be emailed, as it raised thorny questions about of whether Tripod and FBI servers were being used to transmit CSAM. Instead, a Tripod staffer periodically drove stacks of floppy disks 150 miles to the nearest FBI field office.
Other early web companies were similarly unprepared for unintended uses of their software. Nicole Wong, who became vice president and deputy legal counsel at Google, told Robyn Caplan (Data & Society Podcast, 2020) that her first experiences with what is now known as trust and safety came from working for Craigslist, which was founded in 1995. The customer service department for Craigslist at that point was Craig Newmark, the company’s founder, who continued to answer the bulk of customer service emails for the next few years. When matters reached a level of complexity that Craig—essentially as sysop—felt uncomfortable handling, they escalated to Wong, as counsel for the company (Data & Society Podcast, 2020).
Wong also described the emergence of trust and safety at her next job, working for Google. Wong recalled (Caplan, 2018), “There was effectively no moderation. There were customer support people who answered questions, but there were no true moderation policies, per se, other than for copyright and child pornography.” Caplan (2018) continues, “Over time, [Wong] said, key events (such as the Yahoo v. LICRA case, regarding the availability of Nazi paraphernalia globally) played an important role in developing policies, with formalization occurring over time as the platform grew.”
The emergence of trust and safety as a professional department within an online platform, encompassing content moderation, legal compliance, and strategies to minimize abuse and harm, appears to happen primarily as an afterthought. Members of the trust and safety team at Zoom capture this pattern well in their description of Zoom’s creation of a trust and safety team during the Coronavirus pandemic (Maxim et al., 2022): Trust and Safety (T&S) teams are most often born in a crisis. Based on our discussions with other companies, it seems rare that technology executives wake up one day and think, “Next quarter we should start a Trust and Safety Team.” It’s what you do when something bad has already happened. Maybe you notice a lot of cryptocurrency scammers contacting your users, or fake reviews directing users off the platform to hand over login credentials, or your app has become the gathering place du jour for a community of zoophiles. You grab whoever you can to address the problem immediately, and that’s where T&S teams come from . . . Zoom rapidly scaled and formalized its T&S team in the spring of 2020, at a time when we were growing at a blistering pace and gaining a larger, global user base at the same time. Suddenly, people were using Zoom in ways far beyond the business use it was designed for.
This evolution from ad hoc decisions made by sysop-like founders and customer service teams to legally defensible and formalized policies makes sense. As userbases of services increase, the stakes get higher. Companies like Craigslist—which started as a local mailing list run as a hobby—become money-making enterprises with investors, employees, and other stakeholders. Content that violates rules can now reach hundreds of thousands of users, instead of hundreds.
What’s less clear is why early web platforms so quickly embraced a model of trust and safety that had its roots in customer service, rather than in community governance. While many of the people involved in building early Web platforms grew up in pre-web online spaces such as Usenet, MOOs and MUDs, and BBSs, models of community governance largely failed to make the transition from pre-Web spaces to the early commercial Web.
Here are some reasons why businesses like Tripod—which was initially built by programmers experienced with the community-governed world of MOOs and MUDs—unquestioningly adopted bureaucratic, customer-service-oriented models of governance for their new online spaces.
The scale of these new spaces was overwhelming. Pre-Web spaces like LambdaMOO had a few hundred regular participants. Active Usenet newsgroups might have a core of 50–200 active posters and a larger group of readers and occasional posters. Tools to control spam and trolling with these small audiences might not have worked with the thousands of new users flocking to early Web platforms. Asking users to help make the rules for a space growing by the thousands per day seemed like a luxury these companies could not afford.
Companies like Tripod that took venture capital funding found themselves answering questions about their ability to scale their operations. While businesses built on user-generated content were vastly cheaper to operate than those commissioning professionally generated content, directly involving users in governance might make user-generated content businesses financially infeasible. Demonstrating that communities could be governed efficiently with a small team—or contracted out—became part of the process of limiting investor risk.
Legal liability became a major concern for user-generated content companies. Companies like Tripod, which began hosting content before the 1996 Telecommunications Act, and the accompanying section 230 “safe harbor,” had little assurance that they would be insulated from liability for defamation or other complaints about damages caused by online content. As copyright infringement and DMCA takedown requests increased and as CSAM became a serious problem on user-generated content platforms, the need for professionals to respond quickly to infringing content became more pressing.
The community governance mechanisms of the pre-Web internet presumed small communities of like-minded users. Spaces like Usenet and bulletin boards presumed a common interest. Early internet users might have been able to imagine that “on the internet, no one knows you are a dog” (Steiner, 1993), but statistically speaking, a pre-1990 internet user could safely assume anyone she was interacting with was a student, recent graduate, or tech staffer at a university in a wealthy country in the global North. Was it realistic to believe that a “community” of anonymous users with no common background or history could be self-governing?
Whether or not these factors adequately explain the shift in paradigm from community governance to customer service, that shift had occurred by the time most contemporary internet users came online.
The Implications of Shifting From Community Governance to Customer Service
The models of governance that emerged on Usenet, Lam-bdaMOO, and bulletin board systems, where those with technical control over community systems ceded some of that control to their communities, or at least positioned themselves as listening to the needs of their communities, faded into internet history as platforms like Facebook, Twitter, and YouTube came to power. By 2012, Rebecca MacKinnon’s analogies comparing social media platforms to monarchies might have appeared dramatic, but not off-base. In 2006, when Facebook decided to alter users’ experience of the site by selecting a subset of posts to appear in their “newsfeed,” they conducted extensive testing, but did not hold a consultation with users or a vote (Hempel, 2016).
This shift from governance rooted in community participation to governance rooted in top-down bureaucracies has led to a crisis of legitimacy for online platforms. As a platform matures, it finds itself increasingly trying to balance two competing priorities—what Jonathan Zittrain (2019) terms “rights” and “public health.” Rights refers to the ability of end users to participate in online spaces without undue interference—most often this means freedom of speech. Public health asks platforms to: “[weigh] systemic benefits or harms . . . and to think about what systemic interventions might curtail its apparent excesses” (Zittrain, 2019)—most often this means moderating content. When a platform’s system of governance follows the customer service model, its decisions in the name of rights or public health are often scrutinized and criticized as corrupt, arbitrary, and irresponsible, largely because of the lack of community input.
US Senator Ted Cruz captures this sentiment well, referring to then Twitter CEO Jack Dorsey: “Mr. Dorsey, who the hell elected you and put you in charge of what the media are allowed to report and what the American people are allowed to hear . . .?” (CBS News, 2020).
How should online platforms respond to this crisis of legitimacy? We argue that participatory processes—a swing back toward earlier models of community governance—are what’s needed. Participatory processes understand that all views will not and cannot be reconciled. But ideally a participatory process leaves even those who lost a particular dispute or debate able to accept a decision as legitimate.
The governance models from the pre- Web internet gave users a sense of ownership and authority over their communities. In a literal sense, users were often powerless if the administrators of a site made a major change, as the power to change a systems’ rules rested in the hands of a technical elite. But some of that power had devolved to users, either through the establishment of governance mechanisms or through more indirect methods, and many users felt that their points of view had influence over how a community operated.
Users who feel ownership over the spaces in which they participate are often more willing to support them, and sometimes go to extraordinary lengths to do so. Reddit moderator Robert Peck (2019) explains that his willingness to moderate communities (subreddits) as a volunteer is based around his sense of ownership of the community and the rules that allow it to function: You couldn’t pay me to mod reddit.com. Imagine that job: 9 to 5 every day behind a screen, weeding out trolls, totally anonymous yet more vulnerable by the hour for every new racist or sexist you ban. No, I insist on doing it for free.
The rules Peck enforces on his subreddits are ones he helped create. Much as LambdaMOO found with their process of rulemaking through petition, creating your own rules gives them a legitimacy that it is difficult to dispute. Additionally, legal scholars Tracey Meares and Tom Tyler (2021) have found evidence that processes seen as legitimate are more likely to be followed. Examining Facebook and Twitter, they surveyed users who’d been suspended due to violating terms of service. Those who felt the process behind their suspension had been fair and legitimate were less likely to re-offend than those who felt the process was unfair or arbitrary. Tyler, Meares, and their team are now experimenting with increasing the perception of fairness and legitimacy through governance mechanisms that emphasize community participation.
In addition to the benefits that community participation in governance has for online spaces, participating in online governance may also help participants develop as citizens in a democracy. Robert Putnam argued that the retreat of Americans from local institutions, from social clubs to bowling leagues to community government, was likely to damage citizens’ ability to participate in civic life (Putnam, 2000, p. 410): “. . . associations and less formal networks of civic engagement instill in their members habits of cooperation and public-spiritedness, as well as the practical skills necessary to partake in public life.” Serving in a governance position in something as quotidian as a local bowling league might serve as training for broader civic participation. The habits we learn from holding productive meetings, resolving disagreements between people working on a common venture, articulating our point of view, and seeking common ground arguably are the skills we need to participate in a functioning democracy. It is possible to imagine developing these skills in online spaces as well as in offline ones. In losing governance of our online communities, we may be losing a valuable education in democratic citizenship. (Zuckerman, 2022).
Bringing the Community Back Into Platform Governance
The key to the next era of online governance lies not in settling on a set of affordances and policies that are correct or incorrect in one person or company’s view, but rather whether that set of affordances and policies are legitimate because of the “inclusive and deliberative . . . way in which they were settled” (Zittrain, 2019). Participatory processes offer the possibility of reconciling online platforms’ competing priorities in a way that leaves most stakeholders feeling that the decisions made by a platform are legitimate, even if they disagree with a particular outcome. We envision such processes taking the form of transparency, decentralization, and professional norms.
Transparency
There is a litany of worries about the digital public sphere. We worry that online spaces may be harming individuals, undermining people’s body image and self esteem, and pushing vulnerable people toward extremist ideologies. We worry that the internet is increasing political polarization, locking us into ideological echo chambers, and misleading us with mis/disinformation.
How valid are these concerns? It’s complicated, and not just because social science is complicated, but because it’s difficult for independent researchers to study what’s happening in online spaces. We know significantly more about some spaces than others—Twitter became the Drosophila of social media scholarship by making content public by default and giving academics access to APIs—but a recent study of social media scholars (Hansen Shapiro et al., 2021) found that none believed they had access to the data they needed to answer key questions about social media’s effects on individuals and society. This included researchers who were instrumental in designing academic/industry collaborations like Social Science One.
Transparency enables researchers, policymakers, and the public to hold online platforms accountable, understand the digital public sphere better, and spot opportunities for improvements more easily, while potentially legitimating platforms’ claims and practices. Platforms should release more data about content moderation, about how they direct traffic across the internet, and about the effectiveness of techniques used to reduce information disorder (DiResta et al., 2022). They should also give users more information and opportunities for redress when individual-level decisions are made, such as suspensions, content deletions, and content downranking. Legislation related to transparency, like the Platform Accountability and Transparency Act (“Coons, Portman, Klobuchar . . . ,” 2021) can support such goals.
Decentralization
Decentralizing control over online spaces can bring legitimacy to online platforms by including more voices in the construction and operation of the digital public sphere and by offering more choices to participants. We consider a variety of approaches to sharing power, not limited to technical approaches like federated protocols, but also including advisory boards, democratic processes, and experimentation.
Advisory Boards
Advisory boards that have binding power related to a platform and sufficient independence from the platform are one way to decentralize control.
Meta’s Oversight Board is the most prominent example. It is made up of former political leaders, human rights activists, and journalists from around the world who weigh in on Meta’s content decisions (Levy, 2022). The Oversight Board’s (n.d.) decisions to uphold or reverse Meta’s content decisions are binding, which means Meta has to implement them, unless doing so would violate the law. The Oversight Board can also make recommendations to Meta about its policies and how it enforces them. Meta is committed to publicly responding to those recommendations within 60 days (Meta, 2022). Elon Musk has expressed interest in forming a similar “content moderation council” for Twitter (Frenkel, 2022).
Advisory boards enable online platforms to incorporate the input of independent stakeholders into their governance. Ideally, this enables the platform to improve the governance of its space, legitimate some of its decisions, and reduce its operating load. The success of advisory boards is likely dependent on whether their input is binding, the process for selecting members, and the financial sustainability of the board. There are reasonable critiques of both the concept of an advisory board and Meta’s specific implementation, but a check to the otherwise absolute power of a company’s trust and safety decisions is a wise step toward decentralizing power.
Democratic Processes
Democratic processes are a way for online platforms to incorporate input from the community into their decision-making.
Aviv Ovadya (2021) discusses the promise of democratic processes in a paper about “platform democracy.” Ovadya argues that democratic processes can create independent “people’s mandates” for online spaces, a valuable outcome for impacted populations, governments that are constitutionally unable to act on speech, and platforms themselves.
Ovadya highlights the potential of a suite of democratic processes called “citizens’ assemblies” that typically involve creating a demographically representative “mini-public” that is compensated for a fixed time period to learn about an issue, deliberate together, and voice their conclusions. Ovadya points to successful examples of their use in conventional politics around the world including on abortion policy in Ireland and nuclear power policy in South Korea, and argues they could be successfully applied to difficult questions of platform governance.
Another approach to democratic processes can be found in Twitter’s Community Notes initiative (Twitter, 2022). Community Notes allows contributors to suggest a note adding context on any Tweet. If a note is rated as helpful by enough people from different perspectives, the note is displayed alongside the Tweet. Twitter’s pilot of the feature found that: (1) a majority of people found the notes helpful, (2) people were 20%–40% less likely to agree with a misleading Tweet after reading a note about it, and (3) most notes were rated highly for accuracy by professional reviewers. Elon Musk has endorsed the Community Notes initiative and has even found his Tweets subject to the notes (Leffer, 2022).
Democratic processes can be a win–win for communities and platforms—platforms get to shed some of the responsibility for making tough decisions, while the community has more of a say in the governance of the platform. This contributes to a sense of common ownership, while improving governance and taking pressure off the platform. In addition, it is an approach that’s been successfully tested by one of the most prominent institutions on the internet: Wikipedia. Wikipedia makes it clear that these processes are workable at scale and successful in practice, providing an inspiring model for many platforms to build on.
Technical Federation
Technical federation encompasses different approaches which rely on technological solutions to decentralize control over online spaces.
One way of thinking about technical federation is that it aims to make online spaces function more like email (Rajendra-Nicolucci & Zuckerman, 2020), with protocols facilitating choice and independence. Technical federation spans a spectrum, from applications with very different experiences interoperating (imagine YouTube and Twitter being able to speak to each other) to middleware which enables users to choose the algorithm of their choice (imagine that instead of using Twitter’s timeline algorithm, you could choose from a suite of algorithms provided by third parties, like the New York Times, or Fox News). Interest in these technologies has risen in recent years, with projects built around decentralized technology like Mastodon (2023), Matrix (2023), and Bluesky (2023), regulations which require interoperability such as the Digital Markets Act (Lomas, 2022), proposals that emphasize “middleware” such as Gobo (Lane, 2022) and Fukuyama et al. (n.d.), and the techno-cultural movement “Web3” (Roose, 2022).
Technical federation bakes federation into the technology of an online space, providing strong assurances about the decentralization of control. However, technical federation faces a number of challenges. First, technical federation can result in complex and confusing user experiences. Second, there are open questions about the financial sustainability of federation protocols, many of which have been built as open-source projects by volunteers. Third, without regulatory mandates or binding commitments to technical federation by major online spaces, technical federation likely faces major barriers to mass adoption. Finally, federated networks may face the same trust and safety problems as centralized networks with fewer resources to apply to those challenges and more complex coordination problems.
Experimentation
A flourishing culture of experimentation that encourages trying out different configurations of affordances, norms, policies, and governance structures can lead to more choice, agency, and best practices in the digital public sphere.
Supporting experimentation with online spaces would mean more people who find existing spaces unsuitable could build their own spaces. Casey Fiesler (2021) writes about Archive of Our Own (AO3), a “fan-created, fan-run, nonprofit, noncommercial archive for transformative fanworks” that serves more than 5 million people. AO3 was created in response to poor treatment of fanfic authors by existing platforms, including unexpected shutdowns and major feature and policy changes. A group of fans called for a movement to “own the servers” and AO3 was born. If other communities could similarly experiment with their own spaces, it would relieve the pressure on existing spaces to be everything for everyone, and support a sense of agency in the digital public sphere.
However, it is difficult to customize, control, and run the software needed to host an independent online space. Existing commercial solutions are expensive and limit the ability to control and customize the space’s software and data. Open-source software often requires technical expertise to set up, manage, and customize. This means that the group of people able to experiment with online spaces is limited to those able to pay for technical experimentation and those with the knowledge to experiment themselves. To truly enable a flowering of online spaces, we need systems that enable people with little technical expertise and money to spin up their own custom and controllable spaces.
Reddit is a good example of a culture of experimentation within a major platform. Subreddits have significant control over their spaces, including the ability to set their own community standards. That means subreddits can experiment with their spaces, sometimes in partnership with researchers, to improve them. For example, Nate Matias (2019) works with subreddits to experiment with ways to make their spaces more resistant to mis/disinformation and more welcoming to new users. The findings from individual subreddits’ experiments inform shared best practices that improve the platform as a whole. Reddit’s culture of experimentation also means that spaces with very different goals, norms, policies, and processes can co-exist. For example, r/changemyview which has detailed guidelines for participating that emphasize civility and open-mindedness and is dedicated to being (Reddit, 2022a) “[a] place to post an opinion you accept may be flawed, in an effort to understand other perspectives on the issue,” can co-exist with r/wallstreetbets, a rambunctious space, self-described as (Reddit, 2022b) “Like 4chan found a Bloomberg Terminal,” dedicated to discussions about stock markets and which fueled parts of the “meme stock” craze.
Our own work is on small social networks for civic purposes. We have built a social network—Smalltown—that communities can use to run carefully moderated conversations about local civic issues, increasing the number of people that are able to participate. We lovingly refer to it as the world’s most boring social network, because the conversations are respectful, civil, and about hot topics like parallel versus back-in-angled parking. We are also in the early stages of building a music discovery network—Freq—that is trying to create a space outside of the orbit of Spotify that puts control back into the hands of music fans. We think that online spaces built with a specific purpose and particular values can help us move beyond the shadow of major platforms. These independent spaces offer different ways for people to gather online that do not operate on the same logic as major platforms, contributing to a more diverse and fertile digital public sphere.
Public alternatives offer many of the same benefits. The tradition of public media shows that public alternatives can offer a way to shape an ecosystem and correct market failures. A promising early example is Public Spaces (2022), a coalition of public broadcasters and cultural institutions in the Netherlands who are trying to lessen their institutions’ dependence on surveillant software and hoping to build tools that are more consistent with their values. Developers closely linked to Public Spaces are launching a new social media platform called PubHubs (2022), designed for use by small geographic communities, “such as your family, sports club, school class, museum, local library, neighborhood, or municipality.” The platform takes advantage of a novel “attributed based” identity system, IRMA (IRMA Technical Documentation, 2022), which allows people to remain anonymous while presenting evidence that they are member of a particular municipality or have children in a particular school.
Professional Norms
Often professions which have a great deal of power develop public-regarding professional norms as a way to engender trust from society. The canonical examples are doctors and lawyers, but journalists, civil engineers, and police officers have also developed public-regarding professional norms (Gardner et al., 2001). Professionals, particularly engineers, who work on online spaces should similarly adopt a set of public-regarding professional norms. As Jack Balkin (2020) puts it: Social media companies have . . . become key institutions for fostering a healthy public sphere. They can’t just serve economic incentives . . . By analogy, think about journalism. It also serves a crucial role in the public sphere . . . If the professional norms of journalism are weakened or destroyed and the practice of journalism becomes solely market driven, journalism will make the public sphere worse, not better. It will choose stories and treatments that increase polarization, tribalism, and social distrust, and it will generate or help spread propaganda and conspiracy theories.
There are some encouraging signs of norms-building in nascent labor organizing (Tech Workers Coalition, n.d.), growing efforts to instill a sense of ethics in young software engineers (Karoff, 2019), and inspiring examples of public-regarding responsibility in action (Mac & Kang, 2021). However, there is a need for a more concerted effort at developing the organizations, education, and standards required for widespread adoption of public-regarding professional norms. The Trust & Safety Professionals Association is a promising example of what this could look like in practice. It serves (Trust & Safety Professional Association [TSPA], 2022) “the global community of professionals who develop and enforce principles and policies that define acceptable behavior and content online.” It provides its members with career development, networking, training, and knowledge-sharing, and works to improve society’s understanding of the field of trust and safety (TSPA, 2022).
While the emergence of a strong sense of professional identity around trust and safety is unquestionably a positive development, it surfaces the core tension we have explored here: is governance of a community of online users the responsibility of those users, or of professionals devoted to the safety of online spaces? It is our hope that the professional norms that emerge have due respect for the will of individuals and groups of users, and a willingness to allow those users to self-govern when possible.
Limitations
We do not believe participatory processes are a panacea for the digital public sphere. There are many challenges that participatory processes do not address. Our analysis and proposal is focused on addressing the crisis of legitimacy facing platforms.
We are aware that the effectiveness of participatory processes may vary across countries. For example, if a country tightly regulates who can build social media and what appears on it, experimentation and democratic processes are much less likely to be successful, let alone possible. Each of the types of participatory processes we identified—transparency, decentralization, and professional norms—depends on the political environment. However, particularly for large platforms used in many different countries, participatory processes may be viewed favorably even by countries that are hostile to democracy. A common complaint is that large platforms fail to take into account local laws and culture—countries may welcome dedicated processes for citizens and representatives to voice their opinions as well as solutions which decentralize control. One could imagine a country which has much stricter nudity laws welcoming the opportunity to use publicly available APIs to implement their own filters on content. Future work could explore these questions in more detail.
In addition, we acknowledge that participatory processes can obscure certain points of view and reflect existing inequalities. Implementors should take care to design their processes inclusively, so they do not end up simply reflecting the opinions of those with the most time or money to spare. Future work could explore what inclusive design looks like in practice.
Conclusion
The importance of online platforms will only increase as the digital public sphere becomes more central to the contemporary public sphere. However, platforms are facing a crisis of legitimacy, resulting from their attempts to balance the competing priorities of individual rights and public health using a model of online governance rooted in top-down bureaucracies that we term the “customer service” model. Earlier, pre-Web forms of online governance that emphasize community participation offer a promising model for overcoming this crisis, by reconciling platforms’ competing priorities in a way that leaves most stakeholders feeling that the decisions made by a platform are legitimate, even if they disagree with a particular outcome. These models may offer the additional benefit of preparing users for broader forms of civic participation.
For individuals looking for practical steps they can take to support community governance online, we suggest:
Volunteering as a moderator or administrator of an online space
Contributing to community governance initiatives, such as Community Notes
Pressuring platforms to adopt participatory processes
Experimenting with new forms of community online, such as the Fediverse
Contacting policymakers about legislation supporting transparency and decentralization online
Footnotes
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Ford Foundation John D. and Catherine T. MacArthur Foundation.
