A scanner darkly: augmented reality face filters as algorithmic images

1 Introduction

Starting from the mid-teens of this century, a technology for producing digital images has become widespread: Augmented Reality Filters or ARFs. ARFs make it possible to manipulate a video at the very moment it is shot; such manipulations can consist of transforming the appearance of objects and people, and introducing elements that are not physically present. In realistic ARFs, the application of the filter is entirely unnoticeable. ARFs are present in various areas of the digital world: dedicated platforms, such as those of numerous cosmetic companies; stand-alone apps like Facetune, Faceapp etc.; video walls and virtual mirrors in some stores; and live broadcast platforms, such as Zoom or Twitch. ARFs have also spread widely, primarily within major visual social media used through smartphones, such as Snapchat, Instagram and TikTok. Although the fields of application of ARFs are manifold, filters for real-time manipulation of the user’s face in motion are sometimes referred to as ‘ARFace’ (Unity, 2023). They have been particularly successful and have gained considerable visibility.

In this article, I examine ARFaces from two perspectives. In section 2, I summarize their history and explore the four issues that have most attracted the attention of scholars and observers: (i) beautification filters and the risk of ‘Snapchat dysmorphia’; (ii) the new affordances for marketing and advertising offered by branded ARFaces; (iii) the artistic uses of ARFaces, not without theoretical and political cues and provocations; and (iv) the implications of using ARFaces for specific surveillance processes related to face recognition. In section 3, I broaden my scope and interpret ARFaces as typical examples of ‘algorithmic images’. I argue that algorithmic images such as ARFaces radically modify the logic and equilibrium of 19th- and 20th-century technical images by replacing a logic of deferred appearance of reality with a logic of interactive data visualization. In my conclusions, I suggest that the traditional tools of visual studies run the risk of not fully grasping the implications of algorithmic images and I propose initiating the project of a political economy of light as an interpretative tool to understand their specificities.

2 Arfaces As Identity And Identification Tools

In September 2015, Snapchat, a social platform facing several competition problems with Instagram, launched a new feature called Lenses, in which users were allowed to add dynamic effects to their video selfies. The innovation came from having acquired a small Ukrainian startup, Looksery, which had invented this effect and introduced them to the app market the previous year. Although there were only seven Lenses to start with, they immediately proved to be a great success: the one that depicts users vomiting a rainbow was particularly trendy. Given this hit, in May 2017, Instagram introduced a similar feature called Augmented Reality Filters. In this case, the software came from a Belarusian startup, Masquerade, acquired in 2016. The effects, including fake glasses, animal muzzles, strange hats, etc., grew significantly and the competition between the two platforms increased correspondingly.

Snapchat took the next move: in December 2017, it introduced its Lens Studio AR developer tool. This free desktop software allowed users to create original filters and upload them to Snapchat. In addition to end-users of social media, the software was aimed at creative agencies and intended to promote the design of branded ARFaces, thus establishing a new market for advertisers. Instagram perceived the opportunity and launched the analogue Spark AR Studio between October 2018 and August 2019. Finally, in 2019, the Chinese platform TikTok entered the game by introducing its Effects and (from 2022) its ARF creation software, TikTok Effect House. The outcome of these solutions was the massive success of the new technology: worldwide revenues of augmented reality software for social media increased from US$1.8 billion (2017) to around US$3 billion (2023); furthermore, they are set to grow to US$5 billion by 2027 (Statista, 2023).

ARFs currently make it possible to introduce various dynamic visual transformations of the scene being filmed and displayed; however, from the outset, a prominent role was reserved for the manipulation of the human face, especially the face of the user. As mentioned previously, this use is often referred to as ‘ARFaces’, which also received particular attention from researchers and observers. Four broad issues have polarized the discussion.

The first issue involves ARFaces dedicated to beautification (Mihăilă and Braniște, 2021; Miller and McIntyre, 2022). Their peculiarity is to modify the moving image of a face in real time and in an entirely unnoticeable way (Conwill et al., 2023); thus, these filters allow their users to shape images of their faces at will, a practice that may have a substantial impact on their self-perception (Isakowitsch, 2023), especially at the levels of body-image and body-schemata (Tremblay et al., 2021). However, many commentators have pointed out that beautification ARFaces take up and introduce some risky or at least ambiguous aspects peculiar to ‘photoshopped’ selfies. On the one hand, ARFaces push their users (especially young women) to immediately and uncritically embody a set of highly standardized and culturally connoted parameters of aesthetic agreeableness: thinness, small nose, large eyes, full lips, high cheekbones, racial and skin colour standards (Elias and Gill, 2017; Gill, 2021). A particularly remarkable symptom of this trend is an increase in cases of plastic surgery (especially nose and eye retouching) aimed at conforming the patient’s face to an ‘ideal’ face resulting from the application of ARFaces, which some call ‘Instagram Face’ (Ryan-Mosley, 2022). On the other hand (and as a consequence), these practices could create in users a potentially dangerous misalignment between real, self-perceived and desired images of their bodies. British plastic surgeon Dr Tijion Esho proposed labelling this disorder with the successful term ‘Snapchat Dysmorphia’ (Ramphul and Mejias, 2018). Building on this, a good deal of empirical research conducted in recent years in various parts of the world has shown a correlation between ARFace use and some typical Body Dysmorphic Disorders (BDDs) such as low self-esteem, depression, self-objectification and nutritional behaviour disorders (De Valle et al., 2021; Ioannidis et al., 2021; Laughter et al., 2023; Rowland, 2022; Thompson and Harriger, 2023). In the meantime, beautification ARFaces have become increasingly technologically advanced and highly realistic, as witnessed by the successful ‘Bold Glamour’ effect introduced by TikTok in February 2023 (Pescott, 2023).

A different trend of studies has asserted that beautification filters constitute only a tiny part of ARFaces and that the actual motivations for their use are manifold: entertainment, coolness, curiosity, social interaction, silliness, having fun, creativity, brand fandom and so on (Dodoo and Youn, 2021; Ibáñez-Sánchez et al., 2022; Javornik et al., 2022). By emphasizing the playful aspects of ARFs and ARFaces, these research trends call attention to a second issue linked to branded ARFaces, which some companies commission from designers and artists for promotional purposes. In contrast to beautification ARFaces, these filters introduce evident manipulations into the image: for example, they immerse their user in a glamorous atmosphere (Gucci or Dior) or into the mood of a television series (Netflix), or they allow the user to do a try-on of certain products, typically facial make-ups (for instance, in 2018, the cosmetics giant L’Oréal acquired the Canadian company Modiface, which specializes in ARFaces); or, in other cases, they allow users to disguise themselves with posthumous prosthetics that often coincide with graphic reworkings of the company’s logo (Eugeni, 2022). Many AR marketing studies focus on these phenomena: according to them, people perceive branded ARFaces as more original, creative, fun, interactive and informative than unbranded ones. Furthermore, they imply an increase in product purchase intentions and a positive attitude toward brands. In this sense, their use enhances the self-brand connection (i.e. the process in which consumers incorporate brands into their self-concepts); the result would be an ‘augmented/extended self’, charged with renegotiating the gap between the current self and the ideal and desired one with the help of the products involved in the virtual try-on. Ultimately, therefore, branded ARFaces also offer their users a tool to shape their self-images thanks to a live, fluid and interactive representation of their faces (Ambika et al., 2023; Hawker and Carah, 2021; Javornik et al., 2021; Kumar and Agarval, 2023; Yim and Park, 2019).

A third issue concerning ARFaces is linked to visual arts practices and markets. In this field, many observers have started using the term ARtFace (Harrington, 2022). Even though the area of ARtFaces is somewhat fluid and constantly evolving, many specific trends and several prominent artists have emerged. In particular, ARtFace makers like to play on the manipulation of the image of the user’s face, and thus on the undermining of the process of identification between subject and face. This can take place in various ways: through a material alteration of the face’s skin texture turned into a shiny, plasticized surface (Johanna Jaskowska: https://johwska.com/); through the proliferation of three-dimensional make-ups, close to cyborg and posthuman aesthetics (Ines Alpha: https://inesalpha.com/) or to the biological morphology of tentacle and appendages (Omega C: https://www.instagram.com/omega.c/); and, again employing the appearance on the face of other parts of one’s body (S()fia Braga: https://sofiabraga.com/), a multiplication of faces (Cibelle Cavalli Bastos: https://www.instagram.com/aevtarperform/), the progressive and uncontrollable crumbling and flaking of their parts (Aaron Jablonski: https://www.exitsimulation.com/), or even their perforation (Mark Wakefield’s Hole In The Head: https://realityaugmented.co.uk/hole-in-the-head).

Not surprisingly, some observers have seen in these artistic operations an explicit critique of the ‘dysmorphic’ abuses of ARFaces (Behrmann, 2019) and highlighted the political nature of many of them (Malaspina et al., 2022). Indeed, artistic interventions bring to light and problematize the role of ARFaces as tools for self-identity construction and maintenance (Biggio, 2021; Leone, 2021; Villa, 2020) – a role that, as we have seen, is crucial both in beautification filters and in branded ones. From this perspective, ARFaces stand at the crossroads of a dual polarity. Firstly, the construction of personal identity that they activate blends the means and dynamics of self-portrait construction (Belting, 2017; Giannachi, 2023; Ibrahim, 2018; Maes, 2020; Mirzoeff, 2015; Mitcheson, 2021; Rettberg, 2014; Tinel-Temple et al., 2019; Zilio, 2020) with those of ‘individuating’ interaction with technical tools (Koukouti and Malafouris, 2021; Malafouris and Koukouti, 2020). Secondly, this identity practice mixes individual and private aspects with public, social and political ones: the identity processes conveyed by the ARFaces as personal and intimate ‘technologies of the self’ (Foucault, 1988) are affected by collective and shared aesthetic values embedded in their functioning. From this dual intersection is derived the complex nature of ARFaces as identity tools rather than ‘machines of faciality’ (Deleuze and Guattari, 1987: 167–191). In this sense, ARFaces can be defined as ‘machines of facialization’, that is, mechanisms aimed at exploring in more or less gamified form extensions, versions and limits of an ‘extreme self’ (Basar et al., 2021).

The fourth issue regarding ARFaces is linked to a different set of considerations: the role of ARFaces within the contemporary ‘culture of surveillance’ (Lyon, 2018). ARFaces are based on Face Recognition Technology (FRT): to cover the face with a digital mask, it is necessary to recognize its presence in the image (face detection), follow its movements in space (face tracking) and take into account its expressive transformations at any given moment (face monitoring). To do this, the algorithms construct and constantly update a three-dimensional digital representation of the face, using a ‘feature extraction’ process and producing a mesh based on the conjunction of a number (from 100 to about 500) of face landmark points. However, this face mesh is also a ‘faceprint’, that is a unique and permanent identifier for each individual; faceprints can be stored and used for face verification (in the case of 1:1 matching, as when a phone is unlocked by showing the face to the camera) or face identification (as when criminals are identified by comparing the image of their face with those in a public or private database of faceprints, see Andrejevic and Selwyn, 2022). In other words, ARFaces are part of soft or platform biometrics (Crampton, 2020), the most recent expression of a long history of quantifying the human body for surveillance purposes (Van der Ploeg, 2012). Although the connection between ARFaces and surveillance is neither explicated nor investigated, several incidents have demonstrated the use of faceprints derived from them for commercial, security or, in some cases, political surveillance purposes (Eugeni, 2023).

In conclusion, the literature on ARFaces and related phenomena highlights two aspects. On the one hand, they are tools for constructing the identity of their users through the activation of processes of individuation, de-individuation and re-individuation of the self. More precisely, they intersect and combine the dynamic construction of the image of one’s face and the interaction with a series of technical objects, the representation of one’s intimacy and the incorporation of social cues. On the other hand, ARFaces are at least potentially tools for the identification and thus surveillance of their users. Identity construction and surveillance or self-surveillance thus appear inextricably linked: this is exactly what Bob Arctor, the main character of Dick (1977) (whose novel, A Scanner Darkly, lends its title to this article), experiences.

3 Arfaces As Algorithmic Images

In this section, I argue that ARFaces can be taken as exemplary cases of a new type of image that has been defined as computational (Anderson, 2017), post-representational (Purgar, 2019) or algorithmic (Eugeni, forthcoming). Thus, I recover and update the idea of an ‘algorithmic turn’ (Uricchio, 2011) in visual studies. To understand the novelty represented by algorithmic images, it is helpful to consider how they modify specific fundamental characteristics of earlier images and, in particular, what Vilélm Flusser (2011) calls ‘technical images’ and what Friedrich Kittler (2009) connects to ‘optical media’: that is, photographic and cinematographic, electronic and even digital images.

Technological images are based on a logic of spatial and/or temporal distinction and continuity between the moment of image formation on the one hand (input) and that of its display and viewing on the other (output). The process of transfer from input to output can be imagined as a linear chain: at one end, a pattern of photons is fixed on a two-dimensional surface (a film, an electronic or digital sensor) while, at the other end, the pattern of photons is returned to visibility, either through a frontal projection onto a reflective screen, or through the activation of phosphor pigments on the screen of a cathode ray tube, or the regulated switching on of pixels on an electronic screen. Between these two extremes, there is the passage of the image through space and/or time that may consist of a physical transit of a material object (the film) or a transmission of signals (the other two cases). In any case, it is aimed at ensuring the highest possible fidelity (and thus the lowest possible decay, noise or interference) between the image constituted in the input phase and the one observed in the output phase. To be accomplished, this process requires a series of unified, dedicated and recognizable technologies, practices and institutions to which the technical image owes its specificity: these have historically been identified with the media apparatuses. Thus, the uses of the same technologies in other sectors (e.g. in war and social security, scientific research, and so on) remained less obvious and less socially and culturally exhibited. To sum up, technological images are based on three principles: the distinction between inputs and outputs, the continuity in the transmission of visual information between them, and the relative separation of these transmission processes from other areas of social, political and economic action. I argue that algorithmic images such as ARFaces modify all three basic assumptions of technological images.

It is easy to show that the algorithmic image undermines the first principle of technological images. As we have seen with ARFaces, the moments of input and output coincide since the image is produced and transformed as the operator performs the gestures and operations necessary for such production and transformation. In other words, algorithmic images are produced moment by moment by the same subject who observes them, or at any rate thanks to a substantial contribution on his part – and they bear a constant trace of this mainly motor activity of production. Beyond the example of ARFaces, think of virtual reality images that change and adapt according to the user’s movements and gestures. Many authors have conceptualized these phenomena in terms of a tradition of ‘immersiveness’ (for instance, Grau, 2003). Although this aspect is in many cases unquestionable, our analysis of ARFaces leads us to emphasize a complementary aspect: algorithmic images are reflexive. On the one hand, they offer users a form of technologically-assisted mirroring of their appearance and their acting – not coincidentally, Pinotti (2024) places the myth of Narcissus at the origin of virtual reality; on the other hand, they allow users to reflexively experience the embodied, relational and enactive dynamics of their experience within and of the world (Diodato, 2021; Hansen, 2004).

Secondly, the algorithmic image compromises the central principle of the process of constitution of technical images: the continuity between input and output. Once digitized and translated into a data structure (or data set) through computer vision processes, the original photon pattern becomes available for a series of algorithmic treatments before and irrespective of its eventual final visualization. At first sight, this second transformation seems to conflict with the first. In reality, this contrast is part of the paradoxical nature of algorithmic images: their input and output coincide despite the fact that they are separated by computational processing. We can explain this paradox by resorting to the particular temporal nature of algorithmic images, which are produced by processors at a speed greater than the threshold of human perception, and thus appear to be constructed live despite the complexity of the computational processes underlying them.

The computational processes involved in contemporary images are numerous and engage different types of algorithms (Somaini, 2023). If we limit ourselves to what is brought into play by ARFaces, we can highlight three main types of operations. Primarily, we find the extraction of specific information from the original data set: for example, as we have seen, the face mesh of the subject is turned into a faceprint which is used, through comparison processes, to obtain an identification of the subject. Then, the extraction processes are complemented by integration processes: the lens-based, optical source data of the user’s face are blended in real time with the different ‘masks’ designed by the ARFace creators. Finally, based on the datacubes obtained in the previous phases, the algorithms activate actuators, which can be pragmatic (the sliding doors that open at the airport because the detecting machine has recognized our face: we can call them ‘effectors’) or sensitive (in this case we speak of ‘interfaces’): in the latter case, we have a visualization of, for example, ARFaces on the screen of our smartphone. But beware: it is always a partial visualization; for instance, the face mesh produced by the dispositive and actively functioning in all these processes is never shown to us. In other terms, algorithmic images participate in a condition of ‘invisuality’ (MacKenzie and Munster, 2019; Parikka, 2023): they are not constructed to become (in any case and completely) visible.

What I have observed so far opens up the way to overcoming the third characteristic of technical images: their ability to define a social sphere of belonging and functioning such as that of the ‘media’. Indeed, since the centre of gravity of the process of its constitution becomes the automated processing of data, the algorithmic image becomes part of a galaxy of social processes that use data to carry out analyses and surveys, enact controls, predict certain developments and behaviours, and so on. It is therefore not surprising that many technological solutions born in the media sphere are used in other areas of social life and that, conversely, media algorithmic images adopt tools that originated in even distant spheres (from astronomical or heart observations to body and health screenings, to cite a few). The main consequence of this new condition of the algorithmic image is obvious: its de-specialization places it entirely within the processes of automated data processing typical of that particular economic and political regime that has been defined as ‘platform capitalism’ (Srnicek, 2017) or ‘surveillance capitalism’ (Zuboff, 2019). In other words, algorithmic images fully reveal their ‘operational’ nature (Farocki, 2004; Parikka, 2023), since the scope of their application and effectiveness widens from the circumscribed and reassuring sphere of the media to the complex and ambiguous space of society as a whole. ARFaces, by proving to be instruments not only of actual identity construction but also of identification, both commercially and politically, have offered an eloquent example.

4 Conclusion: Towards A New Political Economy Of Light

In the first part of this article, I presented augmented reality facial filters (ARFaces) used in visual social media and explored both their identity valences (related to beautification, branded ARFaces and artistic productions) and their identification uses (related to face recognition procedures they imply and sometimes exploit for surveillance purposes). In the second part of the article, I broadened the horizon of my considerations and argued that ARFaces constitute a particularly evident and widespread example of a new type of ‘algorithmic’ image; the algorithmic image modifies the logic of technical images proper to previous media as it shifts the centre of gravity of the processes of the visual constitution from the remote transfer of information to the automated extraction and processing of data.

The consequences of this radical transformation have yet to be fully focused and analysed by visual studies: for example, the case of ARFaces has shown how identity processes undergo certain transformations that simultaneously and inextricably involve the spheres of cultural forms, aesthetic and technological trends, physical and mental health, marketing strategies, visual arts and surveillance practices. From this perspective, I argue that algorithmic images require visual studies to revise and update their theoretical and analytical tools, to adapt them to their new status. While I do not have the ambition to exhaust such a topic here, I would nevertheless like to quickly indicate some possible directions of research in this regard (Eugeni, forthcoming).

In my opinion, algorithmic images should be analysed against the background of a complex and comprehensive political economy of both physical and symbolic resources: raw materials, technological objects, etc. but also pictures and visual objects, attention, time, reputation and identity, as well as the various forms of labour and energy. This approach should be able to reconstruct how these different resources are produced or extracted, exchanged (and hence valued or devalued), transformed, discarded or recycled within a particular social space. It is reasonable to think that many of these management operations are entrusted to ‘dispositives’, which tend to automate the processes of resource administration and treatment also through subjects’ involvement (Agamben, 2009). Furthermore, we should pay attention to the diachronic evolutions of political–economic regimes, particularly for the advent of new types of resources (e.g. a new energy source, a new technological device that requires new raw materials to be constructed, and so on).

With this background in mind, it is possible to re-read what I have said in this article to reach a double conclusion. First, algorithmic images appear not only as objects (pictures) but as dispositives in charge of managing specific sets of resources. To stay with the case of ARFaces, we have seen how the user’s investment of time, attention, labour and energy leads to producing a visual object representing a reputational and identity investment. But we have also seen how the same dispositive implies an extractive process on the part of the social platform through the face recognition algorithm, and that this process preludes possible economic returns on the market of buying and selling data for commercial or political purposes. Therefore, the scholar’s critical work should consist of deconstructing the dispositives of algorithmic images from time to time, to understand which automated functions they bring into play.

The second conclusion is a more general one. Suppose we observe in the light of the background traced above the evolutions of the economic–political regime of images. In that case, we realize that ARFaces and, more generally, algorithmic images represent the symptom and instrument of a profound and irreversible turning point. In the case of technical images, the use of that specific form of energy that is light (both in its natural and artificial forms) was put at the service of the production and display of images: the political economy of light constituted the essential infrastructure for the political economy of the visual. The advent of the new data economy, which loomed at the beginning of the 1960s but exploded in the second half of the 2000s, brought about a radical change: light (and, in general, the energy range of the electromagnetic spectrum) now constitutes the medium for the production and extraction of data. Algorithmic imaging stems from this transformation, which enslaves the political economy of light to that of information and makes the visual a marginal component of the broader data management process. This is the political economy regime in which visual studies researchers must exercise their analytical and critical skills today.