Abstract
Animation has become ubiquitous within digital visual culture and fundamental to knowledge production. As such, its status as potentially reliable imagery should be clarified. This article examines how animation’s indexicality (both as trace and deixis) changes in mixed realities where the physical and the virtual converge, and how this contributes to the research of animation as documentary and/or non-fiction imagery. In digital culture, animation is used widely to depict both physical and virtual events, and actions. As a result, animation is no longer an interpretive visual language. Instead, animation in virtual culture acts as real-time visualization of computer-mediated actions, their capture and documentation. Now that animation includes both captured and generated imagery, not only do its definitions change but its link to the realities depicted and the documentary value of animated representations requires rethinking. This article begins with definitions of animation and their relation to the perception of animation’s validity as documentary imagery; thereafter it examines indexicality and the strength of indexical visualizations, introducing a continuum of strong and weak indices to theorize the hybrid and complex forms of indexicality in animation, ranging from graphic user interfaces (GUI) to data visualization. The article concludes by examining four indexical connections in relation to physical and virtual reality, offering a theoretical framework with which to conceptualize animation’s indexing abilities in today’s mixed realities.
Keywords
‘Is it working?’ ‘Not really. I don’t know what to do.’
The first scene in the animated Swedish film Slaves (David Aronowitsch and Hanna Heilborn, 2008) begins with a dialogue between two interviewers about how to fix the malfunctioning microphone (see Figure 1). Slaves is an animated documentary based on audio interviews about child slavery in the civil war of Southern Sudan. While the interviewers discuss the faulty equipment, the animated imagery portrays silent children and the adults accompanying them waiting for the technical difficulties to be resolved. This highlights a gap whereby viewers see something that appears reasonable but for which no proof is forthcoming. The interviewers offer only aural evidence of the ‘real’ which contributes to the scene’s credibility, although it actually creates believability rather than fact (the interviewers could actually have been in an empty room, or have used actors). 1 Audio interviews are increasingly common in animated documentaries since they ground the depiction in familiar documentary conventions and enable the soundtrack to act as a link to the embodied existence and physical presence of unseen protagonists. 2

Slaves (2008), by David Aronowitsch and Hanna Heilborn. © Story AB. Reproduced with permission.
This first scene introduces many underlying assumptions about animation’s informative capabilities, contested evidentiary status and subsequent often-used ‘warranting devices’, which assert that ‘what we are watching is (to some degree) true’ (Ward, 2008: 198). These ‘anchors to realities’ and/or familiar stylistic documentary conventions increase a sense of truth value and help steer the viewer into a documentary ‘mode of spectatorship’ (see Odin, 1995: 213). 3 If the audio acts as a warranting device, more fluid visual depictions can be sanctioned since they are not the only – or even the main – criteria for authentication; presence is signified by aural, not visual, elements. However, without the soundtrack, what is animation’s link (i.e. indexical status) to the world it claims to portray when used in documentary?
The discourse surrounding indexicality, which relates to questions of evidence and is grounded in photography, has been highlighted by theorists such as Rosalind Krauss (1985), James Elkins (2007) and Mary Ann Doane (2007). Indexicality has been used in theorizations of documentary, photography, live-action and animation before: Honess Roe (2013: 36) explains that although indexicality ‘has become a shorthand way of claiming the validity of documentary film’ both indexicality and resemblance can be questioned. Since animation is based on generated rather than captured images, however we understand the nature of documentary’s relationship with reality, this relationship is surely severed when animation is used instead of live-action film as the means of visual representation . . . [since] what we see in an animated image did not exist in front of the camera in that form. (p. 37)
She concludes that an epistemological blurring exists between icon (i.e. resemblance between sign and referent) and index (i.e. causal link between sign and referent), whereby ‘we do perhaps still take the iconic as evidence of witnessable events’ (p. 142). In other words, viewers may believe animated images because they can visually resemble the referents they portray (to a certain degree and depending on style). Why is it significant to clarify animation’s status as potentially reliable in documentary and non-fiction imagery? Animation has become ‘the dominant contemporary media’ and is ‘increasingly fundamental to processes of knowledge production’, claim Leslie and McKim (2017: 207). Since the connection between sign and referent may also influence the extent of a sign’s perceived veracity, understanding the links between visualization and referents is important for the proliferation of animation in non-fiction contexts. Thus, it is important to consider the links between image and referent, and the degree to which the representation should be believed. Although animation’s indexicality has been theorized in varied ways, 4 digital culture creates new indexical connections and transforms relations between icon and index, all of which is relevant to animation’s role as believable imagery in non-fiction.
How does animation’s indexicality change in mixed realities where the physical and the virtual converge? And how does this contribute to the research of animation as documentary and/or non-fiction? In digital culture, animation is used widely to depict both physical and virtual events and actions. As a result, animation is no longer just an interpretive visual language depicting the non-physical that cannot be photographed, as in Waltz with Bashir (Ari Folman, 2008) for example, where memories and trauma were portrayed. Instead, animation in virtual culture acts as real-time visualization of computer-mediated actions, their capture and documentation. Now that animation includes both captured and generated imagery, not only do its definitions change but its link to the realities depicted and the documentary value of animated representations require rethinking.
In what follows, four indexical connections are examined in relation to physical and virtual reality, thus offering a theoretical framework with which to conceptualize animation’s indexing abilities in today’s mixed realities. A continuum of strong and weak indices is introduced to map out and theorize the hybrid and complex forms of indexicality in animation, ranging from graphic user interfaces (GUI) to data visualization. This continuum is important because it is grounded in degrees of what is generated vs what is captured within the image, which influences animation’s perceived credibility. The continuum also clarifies the complexity and wider issues regarding the decipherability of the source of images and their consequent truth status. This study begins with definitions of animation and their relation to the perception of animation’s validity as documentary imagery; thereafter, it examines indexicality and the strength of indexical visualizations, and it concludes with animation in mixed realities.
Definitions and opposing worlds
‘Animation’ derives from the Latin animatio which has two key meanings: one refers to movement and the other to bestowing life (see Wells, 2011). ‘To animate’ means to endow with life, but also to represent as if alive, emphasizing the illusion of life rather than life itself (Levitt, 2018: 3). Cholodenko (quoted in Levitt, 2014: 123) emphasizes this illusion of life through the placement of successive images that trick the retina by creating the effect of perfect continuity of movement. 5 Alternatively, Levitt (2014: 118) claims that animation encompasses ‘a wide array of cultural productions from cartoons per se to modes of simulation used across aesthetic and scientific practices’, where the production of life deconstructs ontology and transforms our conception of life (including artificial life), ethics and biopolitics in relation to contemporary media. By analysing animation through its indexical traits in today’s mixed realities, we will see that animation is no longer grounded in an idea of illusion of life, but rather a capture of technologically mediated presence and actions, shedding new light on what animation is and how it should be viewed in a documentary context.
Nichols (2001: xi), for example, differentiates between documentary and fiction: documentary addresses ‘the world in which we live rather than a world imagined by the filmmaker’ (emphases in original), whereas Buchan (2006: vii) defines animated ‘worlds’ as those ‘realms of cinematic experience that are accessible to the spectator only through the techniques available in animation filmmaking’. The infinite possibilities of visual styles and content brought to life through movement, and thus different from what is perceivable in the physical world, create a gap between the two so-called ‘worlds’, the animated one on-screen and that of the viewer. Wells’ (2011: 22–24) definition of animation differentiates between the animated world and that of the viewer, and questions whether both can exist in the same time and space, or whether animation is relegated to another dimension. This gap plays an important role in differentiating animated imagery from the physical world of the viewer, and must therefore be addressed in any discussion of animation’s validity as a documentary language. Mixed realities and new digital techniques change past assumptions. Animation’s contemporary forms of indexicality shed new light on the use of animation in documentary as the connection between the two ‘worlds’. The credibility associated with animated imagery changes as a result.
The complex connection between documentary images and the ‘real’ they claim to depict leads to contradictory ideas: on one hand, numerous debates in media and animation circles – including various scholars’ definitions of animation in relation to photography – have established that in digital culture the binary between animation and photography no longer holds (see Cholodenko, 2014: 99; Manovich, 2001: 295, 2013: 294; Wells, 2007: 12). On the other hand, the comparison between animation and live-action film persists (see, for example, Frank, 2016; Gunning, 2014; Honess Roe, 2013). Despite photography’s long history of manipulation, it is nonetheless still the predominant aesthetic in documentary contexts and a warranting device in many animated documentaries that include photographs or live-action segments. 6 This is attributable to: (a) photography’s mimetic visual style that resembles the referent portrayed, and (b) analogue photography’s indexicality as a basis for its evidentiary status. By comparison, non-naturalistic animation looks different from the physical world and is often generated rather than captured imagery, therefore it is assumed to be separate from the physical world. Animation can be perceived as not real or less real when used in documentaries.
Indexicality and indexical visualizations
Although indexicality raises many questions in relation to digital culture, it is often considered the basis upon which photography has been historically privileged as a documentary language. Barthes (1993: 76) famously explained that, unlike referents of other representational systems, such as painting, the photographic referent was necessarily placed before the lens, thus confirming that the object had indeed been in the physical space before the camera. Although the shift from analogue to digital visual culture generates an ontological anxiety about the loss of the real through the loss of indexicality (Osborne, 2013: 128), the issue of indexicality continues to reverberate in documentary theory (see Doane, 2007: 129–130; Elsaesser, 2005: 90–92; Hadjioannou, 2008: 123–124). This referential quality, or ‘indexing’, is highly valued as a validation that links the referent to its sign and results in a certain truth-value that is expected of the genre. As curator and art critic Enwezor (2010: 10) has suggested, the documentary is expected to embody a direct correlation to its physical referent as an evidentiary act. That said, as technologies and forms of representation change, mediations of realities change as well, and convention and trust outweigh privileged access to any truth claim. Dai Vaughan (1999: 84–85) claims that ‘what makes a film a “documentary” is the way we look at it; and the history of documentary has been the succession of strategies by which filmmakers have tried to make viewers look at films this way.’ The active role of the viewer as arbiter of what constitutes a documentary thus becomes paramount, shifting the focus to the viewer’s persuasion and experience of the work. What, then, establishes an image’s truth-value?
The term ‘index’ appears in Charles Sanders Peirce’s semiotic trichotomy of signs, which comprises the icon, the symbol and the index. An icon is a sign that shares qualities, referred to as a resemblance or likeness, with its referent (Short, 2007: 215–229). An example of an icon is a painted portrait although an icon’s likeness is not necessarily mimetic or even visually similar, opening the discussion to ever-wider possibilities and interpretations. A symbol is based upon arbitrary conventions that define the sign as referring to the object it denotes, such as the word ‘dog’, for example, which is an arbitrarily chosen sound that refers to a four-legged animal of a specific species (pp. 220–221).
The index occupies a more complex position, having a dual definition as both trace and deixis. The index functions as a trace or imprint of its object when the object acts as the cause of the sign, such as a footprint or bullet hole, implying a material connection between sign and object (Doane, 2007: 136). Although the index as trace received privileged status in moving-image theory, Gunning (2007: 30) has argued that it is only one genre of index, and not necessarily the most crucial or definitive, distracting attention from the index as deixis. Peirce’s discussion of the index includes a large range of signs and indications, including ‘anything which focuses attention’ (Buchler, 1955: 108). Chandler (2009: 37) explains the index as: . . . a mode in which the signifier is not arbitrary but is directly connected in some way (physically or causally) to the signified – this link can be observed or inferred: e.g. ‘natural signs’ (smoke, thunder, footprints, echoes, non-synthetic odours and flavours), medical symptoms (pain, a rash, pulse-rate), measuring instruments (weathercock, thermometer, clock, spirit-level), ‘signals’ (a knock on a door, a phone ringing), pointers (a pointing ‘index’ finger, a directional signpost), recordings (a photograph, a film, video or television shot, an audio-recorded voice), personal ‘trademarks’ (handwriting, catchphrase) and indexical words (‘that’, ‘this’, ‘here’, ‘there’).
Gestures such as a pointing finger or words such as ‘I’ or ‘there’, as listed by Chandler, are examples of the deixis, signs that are context-based and classificatory (based on interpretation) (Nunberg, 1993: 8). In other words, the deixis can demonstrate, illustrate and indicate but does not embody a trace to the referent and, unlike the icon, does not have to be based on resemblance. The interpretation of the deixis depends on context and our reaction to it, and this corresponds to the notion that the documentary is based on how viewers see it and the meaning they give it as persuasive and grounded in truth, or not.
It may be argued that any sign indicates a referent, so what is the importance of the deixis when discussing animation and virtual culture? Firstly, the deixis indicates and points, which infers something physical that can be pointed to, as will be discussed below. In an era of virtualization, where dematerialization is a defining characteristic, it is interesting to see what roles the deixis plays within aspects of reality that are not material. As a deixis of the physical, as is often the case in data visualization, animation is similar to a so-called ‘empty’ signifier that receives meaning through context, such as the word ‘I’. Secondly, a continuum of indexical visualizations ranging from trace to deixis demonstrates the contemporary challenge in the use of animation in non-fiction: it may not be clear where to place the animation since the way it was made is not always obvious or accessible to viewers. Even if viewers know that an index has a direct link to something real, if that ‘real’ cannot be recognized, the indexicality has little meaning (McMullan, 2011: 8–9). If a sign is a trace but it is not clear of what, its interpretation is based on context, making it more similar to a deixis. If the source is never clear, then animated imagery may always potentially lean towards the weaker index on the continuum, shaping believability in any case. Thirdly, understanding animation’s complex forms of indexicality contributes to its theorization in documentary contexts.
Definitions of documentary are and always have been ambiguous. Renov (1993: 5) states that the ‘documentary desire’ is a desire to know, whereas Rosen says the documentary’s prime purposes are instruction or recording (p. 66). This is similar to the index, if and only if, both definitions of Peirce’s index are accounted for, since one (the trace) acts as record and the other (the deixis) points to, instructs or informs. As documentary definitions become more fluid due to ever broader exploration and experimentation, we must ensure that expanding a definition does not divest it of its meaning. This has been called the ‘post-documentary’ moment (see Corner, 2002). However, viewing documentaries as part of an epistemological exploration of realities shows that definitions based on recording and/or instructing still hold. Therefore, rather than abandoning or destabilizing the notion of the index as a basis for the representation’s epistemological stake and documentary value, accepting the index’s dual definitions as trace and deixis maintains its relevance in today’s evolving digital cultures and technologies of representation. If the multiple meanings of the index supposedly lead to its collapse, the notion of a document collapses too, thus eliminating any base for documentary. It is therefore important to address rather than dismiss the complexity and multiplicity of the concept of indexicality and its repercussions.
Offenhuber and Telhan (2016: 289) explore indexical visualizations that represent an object or its impact on its environment so that it becomes legible in contemporary digital culture. Although ‘indexical visualizations’ may initially seem to be an oxymoron since an index has a causal link to its source whereby visualization is always mediated, they explain that the term only represents a specific aspect of the object, rather than the whole; the display is linked to the phenomenon and thus its appearance is somewhat autonomous rather than fully within the designer’s control. The appearance of indexical representations varies depending on context and medium, but the main criterion is the causal link between an object/phenomenon and its display (linking it to the index as trace). This link may be immediate and direct, like the pointing of a weather vane. Alternatively, there may be a longer, mediated and semantic distance that translates into an arbitrary visual language, like that of a mercury thermometer (p. 290). Offenhuber and Telhan differentiate between strong and weak indices: a representation with a short causal distance and minimal symbolic mediation would be strongly indexical, whereas one with a long chain of causal events and a highly arbitrary representation would be weak. Analogue technologies may be more strongly indexical since all digital representation is mediated in a way that is highly abstract and symbolic due to its mathematical reliance on code. Although the direct link to the original object/phenomenon may seem to disappear in digital culture, different forms of indexicality exist in digital visualization techniques, including animation.
Since animation refers to many production techniques and infinite styles, it can be trace and/or deixis, as well as icon and symbol. A smile emoji, for example, may be seen as an icon resembling a smiling face, but also as a symbol reducing any human expression down to a stick-figure. The same emoji can be a deictic index indicating any person using it, as well as a range of nuanced emotions loosely linked to happiness, amusement, or pleasure. Alternatively, as in animation techniques based on facial recognition, a smile emoji can also be a visualization based on the user’s actual facial expression, thus acting as trace. 7 Theorizing the visual sign depends on the technologies used and, as technologies evolve, new questions arise.
These ideas are useful theoretical references for the discussion of animation and its varied visualizations as an index. Although they do not refer to the deixis, both kinds of indexicality – trace and diexis – are relevant since the ‘strong’ causal index is the trace, whereas the ‘weak’ mode of visualization embodies an ambiguous causal link between sign and referent, and requires interpretation and context for meaning, like the deixis. Within the digital domain, the link between physical object and its visual representation is often lost or obscured since data based on physical referents is translated into code then visually represented in user interfaces. The result is a reduced understanding of the causal link between on-screen representation and physical referent. This is why a continuum between strong (i.e. trace) and weak (i.e. deixis) indices is introduced here and is used to examine the range of animation’s indexicality, from obviously direct signs based on physical causation to those that are more ambiguous. The more the imagery is captured, the clearer the causal links will be, making it a strong index; if the imagery is generated, connection to the real is less transparent and therefore the index is deemed weaker. The continuum highlights the inability to always decipher the connection between referent and representation, which makes the index’s perceived strength fluid. Additionally, the multiple simultaneous roles an index may take also emphasize its changing perception/definition and strength (as trace or deixis) and underscore the convergence of today’s physical and virtual realities, and thus the complexity and richness of the imagery that indicates/denotes them.
Animation’s indexicality in mixed realities
The following section analyses animation in relation to the physical (both as trace and deixis) and the virtual (both as trace and deixis).
Animation and the physical
While the centrality of the physical has been questioned in relation to contemporary technologized and virtual cultures (Shields, 2003: 2), documentary works still appear to rely heavily on the physically indexical nature of representation to achieve trustworthiness. This dichotomy ensures that the issue of materiality and conceptualizations of the index must be reconsidered. This section focuses on the role of the physical as a link between virtual and physical worlds, and as a basis for truth claims.
Animation as the indexical trace of the physical
Although animation may seem to break, or at least challenge, the link to the physical required by the index as trace, analysing different animation techniques reveals various connections between image and physical referent. Techniques that rely on photography share the dual signatory traits of icon and index, like the photography on which they are based. 8 So, for example, stop-motion, whereby objects are photographed and then placed sequentially to create the illusion of movement, is based on physical objects and their photographic depiction; in rotoscoping, live-action footage is traced over frame by frame (manually or by computer). Even though the photographic imagery may be disguised, and therefore not visually mimetic, the animated imagery is not entirely divorced from the (assumed) indexicality of the photographic footage from which it is sourced.
Digital techniques that enable real-time visualization, however, completely alter animation’s indexicality as trace. Whereas in the past, animation may have been defined by absence, since the imagery is generated rather than captured (see Greenberg, 2011), this is no longer necessarily the case. Animation in virtual culture relies on human–computer interaction dictating the visualized action, as well as on the presence of players who control their avatars and interact with others. GPS, for example, visualizes the users’ movements in their physical environment translated in real-time into minimalistic animated visualization. Any real-time animation used in such interactive platforms acts as an indexical trace of the physical, translating the players’ commands and, increasingly, their physical actions, into the animated visuals of a technological system, i.e. the graphic user interface (GUI). Such technology links animated screen-based virtual worlds with the physical one, contradicting past definitions of animation that claim these worlds are inherently separate. In this case, the real-time visualization of the GPS overshadows the arbitrary or non-photorealistic/visual design of the animation.
Indeed, Offenhuber and Telhan (2016: 298) explain that ‘as long as the representation remains recognizable as a causal effect of the indicated phenomenon, the display becomes a persuasive demonstration of evidence.’ Even if animation does not look real, it may still be perceived as evidence of an action. This means that real-time visualization is more important to believability and the perception of a representation as a strong index than the image’s iconicity, or resemblance to physical appearance.
Similarly, real-time animation techniques used as GUI in interactive contexts like games capture the player’s physical movements and translate them into animated visual form, maintaining a trace of the physical referent, which is then combined with different visual options ranging from an iconic avatar resembling the player, to a symbolic denotation of the user with a dot, spaceship, humanoid, or cursor. Chow (2013: 2, 4) sees the digital environment as ‘animated phenomena’ because ‘it is “endowed with life” rather than just “movement”. . . since these computer-generated visuals also include our bodily experiences through dynamic and responsive visual and audio output.’
Similarly, motion capture (MoCap) animation techniques record the actions of human actors on whose movements 2D or 3D computer animation is based. MoCap captures live movement as computer data, which is then digitally modified and finessed into a finished product in postproduction using animation. Deemed as a strong index due to its immediate visualization, MoCap can be used as real-time visualization; it can also be used in cinematic (rather than interactive real-time contexts) as an example of a weaker animated index, characterized by a more distanced link, though still grounded in the physical. The final MoCap imagery is both captured and generated, blurring the boundaries between photography (as recorded imagery) and animation (as generated imagery). 9 MoCap thus retains the performance of the recorded actor, but eliminates the camera since it produces spatial coordinates over time rather than video footage. Such images act as a trace but can also vary between the iconic (resembling the referent) and symbolic (arbitrarily designed). In this example, more of the procedure is hidden from the viewer and thus the design process takes on a larger role; this introduces a longer semantic distance between cause and effect, i.e. physical movement and animated representation, making it a ‘weaker’ index.
Animation as deictic index of the physical
Animated deictic indices of the physical are varied and multiple. Here the focus is on animation techniques that create ‘empty signifiers’, like deictic indices in language, where the more abstract the animation, the more its interpretation depends on context. Unlike the mimetic appearance often expected of the trace’s iconicity (due to its centrality in photographic theory), the power of the index as deixis is denotative. The deixis draws attention to a particular object, not by representing it visually, but by designating it and pointing ‘there’ (see Hartshorne and Weiss, 1932: 24; Short, 2007: 230). How, then, can animation act as a deictic index to the physical world, creating a link between the two in a referential rather than visually mimetic manner?
Some of these techniques, MoCap and animation as GUI, like the cursor, can also be theorized differently, as a deixis of the physical. Woodcock (2016: 1) explains, for example, that MoCap holds movement information as numeric code with no formal visual qualities, and therefore has a ‘unique capacity to store motion data as potential movement: movement itself that has a life “elsewhere” and at other times, and independent of the performer’s time spent in the live mocap recording session’. MoCap can thus act as a trace of physical movement, but also as a deixic index that remains ‘empty’, until it is given meaning through the visualization it receives. There are other ‘empty’ signifiers that act as a deixis of the physical, which can only be interpreted by context. This leads to data visualization.
Koblin’s data-based artwork Flight Patterns (2009) is a deictic index that visualizes the air traffic routes over North America during a 24-hour period in animated colour and form, but its meaning relies on context (see Figure 2). Each flight is represented by a single line, which could arguably be interpreted as an icon, resembling a flight moving between two points. The line-mapping motion from starting point to destination partially resembles a flight, but it is nonetheless a very abstract signifier that lacks the visual features that usually denote flights, and can therefore be deemed symbolic rather than iconic. The line is mainly recognizable due to context and may not be understood without the work’s title, at the very least. As the work progresses, the many lines depicting flights form the recognizable shape of the United States. The appearance of a map acts as an icon since it shares a likeness, in this case in form, to an identifiable referent, which anchors what is on display and contextualizes the information. The further a sign is removed from recognizability, the more it is reliant upon context and narrative, which makes it both symbolic (arbitrary visualization) and more of a deictic index. The rich indexing potential of this work lies in the animation’s ability to signify more than is directly visualized. This work also illustrates the vast amounts of people and cargo flying each day, thus pointing to the myriad financial, geographic, cultural and environmental effects of such extensive air traffic. The context and interpretation are key in deictic indices which point and make one aware.

Flight Patterns (2009), by Aaron Koblin. Reproduced with permission.
Animation and the digitally virtual
As technology shapes culture, contemporary virtual realities – ubiquitous screens and the centrality of online platforms – require new visualization methods. Once virtual screen worlds become interactive, the on-screen world becomes an in-screen world, in which the user–viewer plays an active role. Animation has become central as a virtual aesthetic because it uses dynamic moving imagery that can respond in real-time to user input. Since our contemporary mixed reality includes the virtual as well as the physical, new theorizations of documentary that transcend the capacity of photography are required. 10 As computerized environments and screen-based virtualizations flourish, a certain dematerialization occurs. What kinds of indexicality can images hold when the reality they denote is not material?
Animation as deictic index of the digitally virtual
While animation has been widely used in documentaries to portray subjective accounts of events and explore personal interpretations of realities (Honess Roe, 2013), the non-photographable, non-physical realities surrounding us are of the digital–virtual kind. As far back as 2001, Manovich (2001: 45–48) listed transcoding as a central characteristic of new media, emphasizing the distance between the numeric coding upon which computation ontology rests and the visual layer (interface or graphics) presented to the user.
The relation between indexicality and ontology is tricky, to say the least. Doane (2007: 135) explains that indices are ‘limited to the assurance of an existence; they provide no insight into the nature of their objects; they. . . simply indicate that something is “there”.’ The notion of the deictic index as a form of pointing exposes assumptions about the physicality of what can be pointed to. Therefore, in a virtualized culture in which non-material realities are given visual form, a new form of indication is conceived through changing uses of animation that point to the existence of such realities (though not physically), and which consequently influence conceptions of these realities. The deictic index of the virtual points to an ‘existence’ thereby allowing the user to take an active role within the visually inaccessible realms it denotes. These issues are central to a reconsideration of what constitutes the ‘real’ and its believable signification or proof in documentary works. They also contribute to the research of animated documentaries and the development of ‘virtual documentaries’ about today’s mixed realities, in which animation plays a central role (see Ehrlich, forthcoming, 2021).
Whereas the index, as conventionally used in visual cultural theory, relies on a material connection to its referent, the digital is grounded in mathematics and code. As culture becomes increasingly computerized and digitized, dematerialization becomes increasingly significant. How do these developments redefine animation’s indexing status, and non-fiction imagery’s truth claims more generally? German media philosopher, Friedrich Kittler (1999: 1), suggested decades ago that the digitization of information and rise in computer communication would reduce human-oriented interfaces. McKim (2017: 294), however, suggests that: Rather than claim that representational or sensory media is becoming less relevant in a computational age, perhaps we should acknowledge instead that our contemporary media is increasingly constituted by mediations of media itself. Or, in other words, there is an increasing preponderance of media that translates digital information into a humanly perceptible form. Seen from this perspective, the relevance of digital animation, comprised of everything from data visualizations to computer generated images, has actually expanded in our post-phenomenological or post-experiential computational environments.
Animation in digital worlds as GUI portrays unplanned activity based on user input. When used in digitally virtual worlds, such as online game environments, animation is the direct visualization of code. It provides the visual interface of the virtual world that all users see, 11 and the façade through which they experience the platform. Animation is thus a deictic index that points to and thereby visualizes the underlying code, indicating its existence. Therefore, on one hand, the GUI is a trace of the physical actions of users as it is dictated by their actions and input (see section about animation as the indexical trace of the physical) but, on the other hand, it is also a deictic index pointing to the realm of code.
Animation as the indexical trace of the virtual
Chandler’s (2009) explanations of the index describe it as directly connected to the referent, whether physically or causally, in an observed or inferred way. Although an indexical trace based on physicality would be impossible in a realm based on code, the trace is also seen as a causal link, which is indeed possible. The trace of the virtual manifests itself in the capture of virtual footage whereby animation plays varied and significant new roles, influencing its documentary nature. As a causal link, animation can act as a trace of the virtual when it is a cinematic (and often edited) recording or capture of virtual settings. Capture not based on the referent cannot exist; its representation is a consequence of what appears before the lens, making it a trace of the virtual.
Gayeton’s (2007) HBO documentary film Molotov Alva and His Search for the Creator, for example, portrays the adventures of an avatar in the popular online world called Second Life. Many of the scenes in the film were recorded, like cinéma verité, with Gayeton pointing a high-definition camera at his own computer monitor because that was the only way he could record animation from his computer screen, though this is no longer the case as the documentation of games and virtual worlds has evolved significantly (Lowood, 2011: 13). The term machinima, a contraction of ‘machine’ and ‘cinema’, refers to the convergence of filmmaking, animation production and game-development technology that creates original content using game engines and recorded gameplay. Defined as ‘the art of making animated films within a real-time, virtual 3D environment’ (Lowood, 2011: 1), machinima is therefore a new form of animation technique (the real-time interactive animation characteristic of digital games), and of animated filmmaking based on recorded game content. Artist and game designer, Eddo Stern (2011: 45), discusses non-fiction machinima as the result of the live nature of online gaming, combined with technological capacities that allow the capture of virtual events, somewhat like live-action documentary, cinéma verité and surveillance video.
In this virtualized context, the gap between referent and representation differs from previous theorizations. Honess Roe (2011: 215) has proposed three key functions whereby the absence of filmed material is solved through the use of animation, style based on the content at hand and the creator’s preferences: mimetic substitution, which offers knowledge of what could potentially have been seen directly (and photographed); non-mimetic substitution, in which animation adds meaning through its chosen visual forms; and the interpretive function, which is used to engage with concepts and states of mind that are difficult or impossible to represent through live-action footage. Animation used to portray personal experiences and physical events that could have been photographed is an interpretive visual language because it depicts events that look different from their animated portrayal, or events with no visible form. In machinima recordings of virtual occurrences, however, the animated depiction is a direct capture of events as they appear on screen. The gap between animated representation and referent narrows because animation’s mediation and constructedness become less prominent, making it a potentially strong index. In the documentation of virtual realities using machinima, animation does not function as an interpretation, or a substitute for photography, but rather as a mimetic visual portrayal of events, directly capturing animated referents, like live-action films that capture the referents before the camera. The animated documentary, Another Planet (Yatziv, 2017), depicts six computer-generated animated simulations of the Auschwitz–Birkenau concentration camp, directly capturing the appearance of the reconstructions as they appear to users rather than using animation as an interpretive visual language. The film’s credits even state that it was ‘filmed’ in specific virtual locations, emphasizing the blurred boundaries between animation and photography.
In the past, non-physical realities, such as belief systems, were interpreted and represented differently in diverse media, whereas in today’s networked world the visualization of digital non-physical realities has a uniform appearance visible to all users. Machinima acts as footage of the virtual, presenting viewers with what occurred in the virtual environment as seen by its inhabitants (see Ehrlich, forthcoming, 2021).
Conclusions
In today’s mixed realities, animation has different forms of indexicality, both as deixis and as trace, in relation to physical and virtual realities. This changes previous definitions of animation and is thus a crucial factor in documentary theory.
Firstly, past definitions include those of Greenberg (2011), for example, that emphasize absence and animation as generated imagery, or Wells (2011: 24), that express animation’s separateness from the lived physical world of the viewer. Digital technologies, however, demonstrate that animation includes both generated and captured images. This strengthens ideas about animation’s similarity and convergence with live-action cinema. Animation may seem more acceptable as documentary imagery rather than ostensibly opposing more traditional documentary conventions based on photographic technologies. Moreover, examining animation’s indexicality challenges the opposition between on-screen animated worlds and the viewer’s physical surroundings, showing them instead as connected. Thus, in documentary, animation can no longer be deemed separate from the physical world, which also weakens the idea introduced by Nichols (2001: xi) about the differentiation between ‘a world’ vs ‘the world’ (emphases in the original).
Secondly, examining animation as both trace and deixis highlights how animation embodies the contemporary convergence of physical and virtual realities. According to Hansen (2006: 2), it is motor activity, rather than representational mimesis, that enables movement between virtual and physical worlds. Animation does this when it acts as a trace of the physical, as with GPS, which introduces a sense of the multiplicity of tangential worlds within which users/viewers are active today, easing the transition into the ‘other’ space of the animated world. Animated and material worlds thus converge through these new techniques and inventive forms of referentiality. The centrality of animation in virtualized culture, as a sign that indicates the physical movements and input of users, cements its central representational role in the arena of human–computer interactions. As a trace of the virtual, based on its causal dependence on it, animation enables those absent from or unfamiliar with the virtual platform to be mediated witnesses and learn about it through the captured footage of virtual events. This makes animation similar to live-action footage, though of virtual events.
As a deixis, animation also has multiple roles and significance: when used as a deixis to indicate physical events, animation as data visualization of such events also grows exponentially. The proliferating use of animation to portray the physical reduces the alleged gap between the animated on-screen world and the viewer’s physical reality because viewers become accustomed to receiving factual information about the physical world through animated imagery. The blurring of boundaries between animation and non-animated worlds contributes to animation’s reception as credible imagery, despite visualizations that do not resemble physical referents. Animation as a deixis of the virtual points to the existence of code and visualizes it as the GUI whereby users act within the virtual. The deixis is thus an important form of indexicality because: (a) it visualizes code and thus makes the virtual accessible; and (b) by deictically indexing physical referents, animation shows that so-called ‘empty’ signs have become essential in mixed realities. Animated GUI can denote anyone using the technology, as in the case of a cursor; it is a key characteristic of contemporary digital culture in which we are accustomed to arbitrary visual signs representing our presence and actions in virtual realms. This break with visual resemblance/iconicity influences the imagery viewers accept as believable, thus habituating viewers to animated depictions of themselves and their world, and thereby linking the discussion to documentary discourse in which conventions depend on what viewers deem acceptable. The deixis also emphasizes the dual role of the document, as articulated by Rosen (1993), who claims that the document both proves and teaches, expanding ideas about documentary and its relation to expectation and conventions, and highlighting the importance of considering the often under-researched deixis, rather than only emphasizing the index as trace.
Thirdly, the continuum demonstrates that animated signs can act as both trace and deixis, sometimes simultaneously (like an avatar as trace of the player’s actions, as deixis of someone in the physical world and of the game’s virtual code) (Ehrlich, 2013: 263). The continuum emphasizes the possible overlap between a trace and deixis when representing mixed realities, questioning the perceived fluctuating strength of certain indices and influencing the possible theorization of animation as a document in documentary and non-fiction contexts. However, a continuum of indexical visualizations with its innate fluidity as a spectrum also shows how the perceived strength of an index may waver. We cannot always know from watching the animation which techniques were involved in its making. Even if viewers become accustomed to varied visualizations, it is difficult or impossible to know how much of the imagery was captured vs generated; the extent to which the final image was constructed, designed or manipulated would change the so-called strength of the sign. This emphasizes the semantic gap that always exists in visualization as mediation and re-introduces the complexity of what is deemed believable imagery. Arguably, the same can be said of today’s photographic and live-action imagery, which is frequently edited and doctored. So, should animation and live action be theorized differently in relation to believability in non-fiction? This inability to decipher should encourage reflective viewing, but does it? Or does it merely blur images’ connection to the real and evoke a confusing dilemma – believe everything or nothing at all? More research into criteria of imagery and its believability would thus be useful as techniques and assumptions change (see Ehrlich, 2019).
Fourthly, engaging with the thorny issue of animation’s indexicality sheds light both on animation’s iconicity and why animation becomes essential to documentary in technologically augmented realities. When used to portray the physical world, animation introduces a new approach to documentary imagery. Interestingly, animation as an indexical trace of the physical relies on the physical but does not necessarily resemble the referent. Although Honess Roe (2009: 142) has proposed an epistemological blurring of icon and index whereby ‘we do perhaps still take the iconic as evidence of witnessable events’, I argue that digital animation techniques are indexical in varied ways and thus the techniques used introduce new causal connections between visual display and referent portrayed. The iconicity of the movement remains but not the referent’s physical appearance.
Maintaining the importance of the index as a basis for documents and thus documentaries, engenders a post-photographic mentality, not in the sense that digital-production methods and a growing awareness of potential image manipulation have established digital photography as a new medium. Rather, that the logic of the photographic based on indexical trace is maintained, though not the photographic aesthetics that rely on resemblance. Honess Roe (2009: 142) seems to propose a ‘pre-photographic’ logic whereby what is similar in appearance to the referent is considered sufficient, an observation of animated documentary that echoes modes of visualization that predate photography, such as painting. Here, the emergence of a post-photographic logic is emphasized since the elements that made photography credible – specifically, its analogue relation as trace of the physical – are maintained, whilst the aesthetics of photorealism are modified. This explains the significance of animation as physical trace in a documentary theory based on indexicality. It also allows for visual changes that embrace the symbolic in an increasingly virtualized culture less reliant on the appearance of the physical, since presence now also refers to in-screen actions symbolized by myriad, stylized referents. Accustomed to seeing ourselves visualized in unlimited ways, other characteristics such as interactivity and real-time visualization take precedence over visual resemblance in contributing to the believability of signs. Animation is no longer problematic as a documentary language just because it looks different from the physical world or seems separate from it in its modes of production.
Finally, the merging of physical and virtual realities leads to an augmented view and experience of reality, which exceed the merely physical. 12 Enhanced reality enabled by technology has become ubiquitous due to personal portable screens and wi-fi. These new experiences of contemporary reality must be translated into suitable new documentary aesthetics. The impact of advanced technology on daily life makes transcending the limitations of photography in documentary an inevitable necessity. In her research into the subjective additions that animation makes to documentary, Honess Roe (2009: 89–90) concludes by discussing the desire to transcend the limitations of photographic media in documentary (emphasis added). Although I clearly agree, it is important to emphasize that, by analysing contemporary digital culture and mixed realities, what becomes evident is that such an expectation is not limited to documentaries; it permeates the actuality of life in the early 21st century and therefore becomes an objective necessity rather than a subjective ‘desire’. For documentaries to remain relevant depictions of contemporary mixed realities they must include animation, a central visual language depicting user input in virtual realms. This changes the way animation has been approached in documentary theory and, hopefully, opens up new questions in the future.
Footnotes
Acknowledgements
I would like to thank the reviewers of this article for their insights, and the animators and artists who generously gave me permission to use their work in this publication: Aaron Koblin, David Aronowitsch and Hanna Heilborn.
Funding
The author received no financial support for the research, authorship and publication of this article, and there is no conflict of interest.
