Abstract
This commentary discusses an important disconnect between different literatures dealing with the power of algorithms – a disconnect that has important implications for the narratives about the role of algorithms in today’s societies that the humanities and social science construct. Theoretical work has regularly depicted algorithmic systems in the hands of large tech companies as powerful devices for effectively influencing behaviors through persuasive targeted information offers in online environments. However, this account goes against existing empirical evidence on the effectiveness of algorithm-based targeting. The paper highlights the different stories about the power of algorithms that these literatures tell and discusses why addressing this gap matters. In a nutshell, while the idea of people being steered by powerful algorithmic systems is arguably an intriguing aspect of digital-era capitalism, it risks distracting from more mundane, but also more relevant aspects of algorithms operating in online environments and how they can sustain power relations.
Introduction
When former Google CEO Eric Schmidt publicly stated that Google ‘can more or less know what you’re thinking about’ (Saint, 2010) and when the same companies’ Director of Engineering Ray Kurzweil (Khomami, 2014) claimed that Google would know people better than they knew themselves, this may have seemed strangely upfront. Why would they so openly state something that may be perceived as creepy by many who use the company’s services? After all, these statements were uncoerced and perfectly avoidable. They make perfect sense, however, as statements directed at those actors which are their actual – and paying – clients (Cohen, 2012, 2014): marketers that want to distribute advertisements to those about whom google allegedly knows so much about. For them, the idea that it is possible to not only reach huge audiences but to also know whom to best reach with which advertisements and product offers must clearly be enticing. Targeting people with the right advertisement at the right time based on a very fine-grained segmentation of consumers holds the promise of more persuasive messages.
A similar promise has also led actors in the political realm to rely on targeted online advertisements in their campaigns. Recent elections have taken the customization of online content in political campaigns to a new level as the Trump campaign starting in 2019 has been reported using several hundred 1000 distinct ads (Wong, 2020). The idea behind this targeting is the same as in commercial advertising: to reach citizens with content that is best attuned to their dispositions – which also includes not trying to reach certain citizens at all and instead saving the effort of addressing those voters who are very certain to not support a party or candidate (Barocas, 2012; Shorey and Howard, 2016). And similar to commercial advertising, businesses try to lure politicians with the promise of highly effective data-driven campaigns – a narrative that is then often perpetuated by media who garner attention with stories about potent personalized advertisements (Baldwin-Philippi, 2017).
The idea that fine-grained data on individuals together with algorithmic filters in online environments heavily boost the possibilities of distributing persuasive messages has not just taken hold among businesses and politicians. Academia too has partly adopted the narrative that large tech companies dispose of the means to intervene into and purposefully steer people’s individual behaviors on a massive scale (e.g. Gorton, 2016; Lanzing, 2019; Shorey and Howard, 2016; Zuboff, 2019). Scholarly work along these lines has been very valuable in elaborating on how these technological means can amount to a subtle but nonetheless effective form of steering that hardly compares with previous possibilities of regulating behavior.
However, there is also a discernible disconnect from existing empirical research on the effectiveness of algorithmic filters and targeting. It is important for the social sciences and the humanities to confront this gap to get their stories about digital capitalism and algorithmic power right. Otherwise, they may unwittingly reiterate a baseless narrative of potent algorithms that is mainly in the interest of certain commercial actors. This research note therefore briefly highlights the discrepancy mentioned above and discusses its implications. In a nutshell, while the idea of people being steered by a powerful and encompassing apparatus for persuasive behavioral influence is arguably the most sensational aspect of online platforms’ business models, it hides and distract from the much more mundane face of digital capitalism.
The narrative of potent algorithms
A burgeoning literature on how the adoption of algorithmic systems is used to manage risks and regulate behavior has highlighted how algorithms can intervene into social relations, sustain power relations and impact on people’s legitimate interests and personal rights. While algorithms may appear like objective solutions to given problems, they are never neutral and necessarily incorporate certain assumptions and values (Felzmann et al., 2019; Levy et al., 2021; Yeung, 2018). They may therefore operate in ways that entail undesirable biases from the point of view of those affected, and they can lead to forms of unfair discrimination, for example, based on gender (Barocas and Selbst, 2016; Lepri et al., 2018; Pasquale, 2015). As a result, they can do considerable harm in areas in which incisive decisions are made, such as in recidivism risk assessments, classification of social welfare fraud, or recruiting decisions. The harm caused by algorithmic biases in highly sensitive decisions is one major risk that is associated with the use of algorithmic systems.
Another potential harm that follows from their use is that they can have adverse effects on people’s personal autonomy (see also Mittelstadt et al., 2016). There are two facets to this possible harm to autonomy through affecting how people think and behave. First, algorithms used in platforms that operate as powerful intermediaries in the public sphere are akin to institutions that structure interactions and intervene into society by shaping what issues rise high on the agenda and thus what discourse revolves around (Just and Latzer, 2017). To the extent that algorithmic filters in online environments are expressly guided by values and policies that support certain positions rather than others, they can directly go against the interests of certain users without them knowing (Tene and Polonetsky, 2017). Shaping what people think about on a large scale clearly makes algorithms potent forces in society.
However, there is also a second way in which algorithms’ effects on people’s autonomy are discussed that is more direct and more focused on the individual than the large-scale, structural impacts of algorithms. Specifically, algorithms may more directly influence individual thought and behavior by targeting online content that is personalized and designed to purposefully induce certain behaviors, including through matching content to people’s dispositions and personality (e.g. Lanzing, 2019; Yeung, 2017; Zuiderveen Borgesius et al., 2018).
It is this latter way in which algorithms are potentially a major means of influence in society that the following discussion is concerned with. It will therefore focus on algorithm-based content provision in online environments as a business activity that potentially affects a large number of people. The collection and processing of personal data in online environments by large tech companies has come to be a defining feature of globally influential business models in the early 21st century. Beyond the sheer scope of the practices designed to create value from data, it is the unprecedented possibilities for personalizing information offers that has attracted much attention. For instance, Rouvroy (2013) has described the massive collection of personal data and algorithm-based profiling of individuals as a form of data behaviorism. It is based on detecting patterns in behavioral information about many individuals and on using these registered patterns to anticipate future decisions. Data behaviorism is therefore marked by an ability to predict and pre-empt the behavior of individuals, much as in the quote by Ray Kurzweil cited at the outset of this paper.
The notion of data behaviorism implies that algorithm-based steering could be a very potent form of influencing people in the interest of third parties. Indeed, some contributions even make much stronger and more explicit claims about the societal impact of algorithms. For instance, Zuboff’s (2019) seminal analysis of what she calls surveillance capitalism points to the emergence of prediction products, that is, behavioral predictions based on fine-grained profiling data that are sold to advertisers and marketers intent on steering masses of individuals in a certain direction. In this account, the use of algorithms to predict and modify behavior threatens to undermine personal autonomy on a large scale. Similarly, Couldry and Mejias (2019) have compellingly elaborated on how business models that are based on large-scale data extraction and value creation from data mirror well-known older forms of colonial appropriation. They see people’s personal lives and their social relations commodified through businesses turning their data into assets which are then processed to influence people’s behaviors. While the authors adopt a broad view that also critically discusses the creation of asymmetries and dependencies with data-based business models, their account also depicts algorithms as a very effective instrument for shaping behavior in online environments. Writing about the broader social effects of these practices, they warn of a ‘dispossessed life’ (Couldry and Mejias, 2019: 157) due to data relations increasingly interfering with people’s space of autonomy and decision-making (Couldry and Mejias, 2019: 161).
The wording used in some contributions thus suggests that algorithmic systems already today purposefully influence individual thoughts and behaviors of a multitude of people. Like rats in a maze, they become the object of behavioral interventions in large-scale experiments. Being subjected to constant surveillance, individuals are seen as being caught up in ‘incessant feedback loops between human and machine’ (Burrell and Fourcade, 2021: 228). Equally dramatic diagnoses can be found with regard to the consequences of algorithms in the public sphere (e.g. Franz, 2013; Gorton, 2016; Pariser, 2011). A prevalent narrative in scholarly work thus depicts algorithmic systems that are at work in online environments as powerful entities that lead people to make certain choices rather than others.
On the underwhelming evidence on algorithm-based behavioral influence
When looking at available evidence on how effectively algorithmic filters and targeting influence people’s behaviors, the picture looks very different, though. Regarding online political microtargeting, evidence from several experimental studies does indicate that targeting with personality-congruent content makes messages more effective (Dobber et al., 2021; Krotzek, 2019; Lavigne, 2021; Zarouali et al., 2020). While studies like these use rigorous research designs, there remains a problem of external validity: We do not know how strong effects are in people’s natural environments in which they are exposed to many different influences, including from their social network and larger information environment (Zuiderveen Borgesius et al., 2016), nor is it clear how lasting any registered effects are. Furthermore, there is evidence qualifying the effectiveness of algorithmic filters in online environments. Existing empirical findings cast serious doubt on the idea of powerful algorithmic filter bubbles that enclose people in homogenous content (for an overview, see Barberá, 2020; Bruns, 2019; Courtois et al., 2018; Dubois and Blank, 2018; Haim et al., 2018; Krafft et al., 2018; Zuiderveen Borgesius et al., 2016). It has also been shown that political microtargeting may often lead to reactance as people recognize that they are targeted and consciously reject the content (see also Burge et al., 2020; Kruikemeier et al., 2016).
Skepticism about the effectiveness of algorithm-based targeting is equally warranted with commercial ads. As with earlier forms of advertising, ascertaining its effectiveness remains a major challenge in the digital era. Whether people engage in a given behavior, such as buying a product after seeing an ad, is not generally known to advertisers and intermediaries. What is instead usually registered are clicks. But clicks are not predictive of conversion into sales, and people who decide to buy a product rarely click on ads (Aral, 2020: 152). Even if the actual outcome, a buying decision, were known, it would be hard to tell whether a personalized advertisement was responsible for it. A person may well have been targeted for the same reason that she was already very likely to buy the product anyway. The advertisement was then not so much the cause, but instead itself an outcome of the actual cause behind the buying decision. As Aral (2020: 148) notes, it is a dirty secret of online platforms that ‘[d]igital ads don’t work nearly as well as they’re advertised to’. Yet, as noted with regard to the two quotes cited at the outset of this paper: What counts is to create the impression that online platforms are highly effective at anticipating and influencing behaviors.
For marketers it is hard to corroborate whether such claims are true. As Hwang (2020) writes, they do not have access to important performance indicators for their ads that would allow them to judge their effectiveness. In light of a decreasing quality of online user attention and widespread fraudulent practices, such as click farms that manufacture seemingly good ad performance, Hwang (2020) argues that the online ad market is highly overvalued and finds itself at the verge of collapse, similar to the subprime crisis. Various empirical studies indeed suggest that targeted online advertisements have rather limited effectiveness. A study by Marotta et al. (2019) indicates that targeting individuals based on available cookies increases a publisher’s revenue by 4% – which clearly deviates from the idea of algorithmic targeting being far superior to non-targeted advertising and multiplying its effectiveness. In a similar vein, Winter et al. (2021) show only very limited effects of trait-based personalized ads, suggesting that consumers develop favorable attitudes toward the presented ads whether they are targeted or not.
Effects are also strongly conditioned by consumers’ dispositions as well as contextual factors, such as the website visited and transparency provisions (e.g. Bleier and Eisenbeiss, 2015; Dogruel, 2019; Evans et al., 2019; Kim et al., 2019). Although people may be led to follow advertisements if they are made aware that they have been targeted due to certain characteristics that they find favorable, this can also backfire if the provided explanation conflicts with people’s self-understanding (Summers et al., 2016). Similarly, targeted advertisements can lead to reactance, resulting in a negative rather than a positive effect on buying behavior (e.g. Farman et al., 2020; Fitzsimons and Lehmann, 2004; Ham, 2017; Li et al., 2021; Malheiros et al., 2012; van Doorn and Hoekstra, 2013).
All in all, existing evidence is hard to reconcile with the idea that people’s behaviors are incessantly and effectively nudged or shaped by powerful algorithms. This narrative, although it does point to potentially problematic practices, seems exaggerated and thereby risks distracting from those aspects that are more relevant when studying algorithmic systems and the larger context and practices in which they are embedded.
Toward a synthesis of the two views
Algorithmic forms of regulating behavior, through anticipating decisions and personalizing content on a mass scale, are arguably one of the flashiest parts of digital-era capitalism. A potential problem of depicting algorithmic regulation in online environment as a potent force that shapes how people think and behave is that – intended or not – it may reiterate a narrative about how effective the tech sector’s solutions are. Tech companies have cultivated this image with clever PR, branding themselves as disruptive innovators. An aura of almighty tech and especially AI has been carefully crafted and sustained by highlighting breakthroughs such as IBM’s Watson beating the best human player at Jeopardy or Google’s AlphaGo. If AI can do these things, the reasoning goes, it must be really powerful also in other domains. However, the performance of AI or algorithmic systems is often rather unremarkable (Broussard, 2018), and the greatest threat of AI does not lie in any superhuman capabilities, but in the fact the many applications are designed and work rather badly and yet are implemented in the real world (Crawford and Calo, 2016). In some cases, AI is even faked by humans working to make an AI seem to be operating. Crawford (2021: 65) describes this as ‘Potemkin AI – little more than facades, designed to demonstrate to investors and credulous media what an automated system would look like while actually relying on human labor in the background’. Perhaps, users of algorithm-based services, both consumers and markets, may have to confront the truth that it has been remarkably easy to satisfy both with half-baked solutions.
Expressly acknowledging that algorithms which operate in online environments are not that potent after all is no minor detail but changes the story one may tell about the power of algorithms in today’s societies. The story is then not one of information infrastructures with highly effective behavioral nudges or persuasive content offers that steer people via targeting and through shaping their decision environments. Rather, the story is one of an emerging information infrastructure that erodes privacy on a large scale for marginally effective behavioral influences of targeted online information offers – but with the systemic result of shifting market power to online platforms. There are two important aspects to this structural change.
First, marginal ad effectiveness gains for marketers come with benefits, but also with a price for consumers: an information infrastructure and business models that depend on large-scale surveillance. The issue is not that people are effectively steered by algorithms under these conditions and become the passive and defenseless victims of cybernetic control over individual actions, but instead the presence and scope of the surveillance. People are harvested for data on a massive scale that allows companies to sell audience attention to marketers. The behavioral impact through persuasion may be minimal, yet the systemic impacts on privacy and control over one’s personal information are very large.
Second, the efforts to build these surveillance infrastructures have effectively drawn away advertisers from traditional channels. Indeed, the most remarkable change in the advertisement market is not the growth of the pie, but how it has been redistributed: There has been a huge shift from traditional media (mainly television and newspapers) to online media since the mid-2000s, which has attained a share of more than 50% by 2022, while it was still only a fraction just 10 years before (Wood, 2020). What we see is thus the rise of powerful intermediaries displacing other intermediaries through creating online environments that generate a lasting attachment of users who are surveilled to commercialize their attention and data. And creating these ties and achieving the resulting shifts in market power seems to have been possible without notable persuasive effects through algorithmically personalized information by third parties.
The narrative that these online platforms have superior knowledge about consumers and can therefore realize more effective ads through online targeting, has arguably played no small role in this huge shift (Hwang, 2020). Another important reason can be seen in the fact that online platforms and services are where large audiences are in the digital era. Even if online ads do not work well, online platforms are and must be very effective at one thing: Capturing attention. They do so based on network effects, market concentration, and persuasive design that builds consumer habits and lasting engagement (Eyal and Hoover, 2014).
The real persuasive effect thus does not so much lie on the level of algorithmically targeted content and its persuasive impact on the individual, but on the systematic level of achieving sustained user engagement through ongoing social feedback and comparison. That the mechanisms of online environments are geared toward the overarching goals of fostering engagement and capturing attention became particularly visible when in late 2021, the content filtering algorithms on Facebook and Instagram briefly came under public scrutiny (Metz, 2021): As whistleblower Frances Haugen explained, the engagement-based sorting of algorithms operating on these platforms means that they often prioritize provocative misinformation, hostile and divisive content, and content that evokes and perpetuates harmful body images among teens.
It also follows from the above discussion that the notion of powerful algorithms in online environments is not wrong. However, a differentiated account about the power of algorithms surrounding users in this sphere needs to recognize how algorithms can both be rather ineffective and still help sustain practices that heavily intervene into society and social relations on a structural level. Existing evidence suggests that they are largely ineffective when it comes to influencing individual behaviors, yet they form an important element of larger information architectures that engender engagement. Bearing these distinctions in mind is important to guide the study where and how exactly algorithms can exert power and sustain power relations in ways that make them an important force in society and culture.
