The Multilocal Self: Performance Capture,Remote Surgery,and Persistent Materiality

Abstract

Over the last two decades, the technologies of performance capture and robotic surgery have increased in both use and visibility. While these technologies might initially seem quite dissimilar, they each produce a human–machine assemblage that enacts itself across different scales. Each technology ‘captures’ a performance, translates that performance into digital information, and recodes that performance into another body. This article argues that both performance capture and remote surgery penetrate the materiality of the body and reconstitute that materiality elsewhere, as a human’s bodily movements are captured, transmitted, translated, and finally recoded into that of another body, be it an analogue or digital form of embodiment. The shift in scale produced by each technology – in terms of movement, perception, experience, and sensation – demonstrates the extent to which these technologies of telepresence foster a multilocal experience of the body, the dispersion of authorial control across the human–machine assemblage, and a reinforcement of embodied experience despite an embrace of cultural fantasies of the disembodiment of information. This article takes an explicitly phenomenological position, examining the connective tissue that binds actor and avatar, surgeon and robot. The ligaments that connect human and nonhuman both separate and draw the entities close together, and this article explores the resultant shifts in scale, perception, and experience engendered by performance capture and robotic surgery.

Keywords

digital animation digital avatar embodiment film studies performance capture phenomenology materiality robotic surgery telepresence

Magician and surgeon compare to painter and cameraman. The painter maintains in his work a natural distance from reality, the cameraman penetrates deeply into its web. There is a tremendous difference between the pictures they obtain. That of the painter is a total one, that of the cameraman consists of multiple fragments which are assembled under a new law. Thus, for contemporary man the representation of reality by the film is incomparably more significant than that of the painter, since it offers, precisely because of the thoroughgoing permeation of reality with mechanical equipment, an aspect of reality which is free of all equipment. (Walter Benjamin, 1968[1936]: 233–234)

Introduction

During an exploratory search for images of performance capture – the process by which an actor’s movement and performance are recorded, translated and then recoded into a digital avatar – I entered a series of terms into a Google image search, which produced results that have lead directly to the subject of this article. In my search for content related to performance capture, I encountered a productive juxtaposition of images: that of performance capture technology and the technology of remote robotic surgery, which allows a surgeon to perform a procedure on a patient in a different location to that of the surgeon (see Figure 1). The results churned out by the Google algorithm led me to question what the relationship between these seemingly unrelated technologies might be and whether analysing them in conjunction might yield fruitful conclusions. In what ways are the technology and experience of performance capture similar to the technology and experience of remote robotic surgery? How does each technological assemblage, through its dispersion of the human throughout a network, reconfigure the experience of movement, time, and space? How does each alter the scale at which the ‘performer’ – as well as the ‘viewer/patient’ – senses and perceives the world, enacting a sensual, proprioceptive experience of technology? In what ways does each think through – and visualize – the relationship between the analogue and the digital, crystallizing and animating contemporary fantasies of disembodiment and informationalism?

Figure 1.

Google, unfortunately, was unable to answer these questions satisfactorily, but it did provide a necessary nudge. Just as this article is a product of a complex network of human–machine assemblages (myself and the Google image search), so too are performance capture and robotic surgery embodiments of contemporary technological and cultural assemblages. While robotic surgery might at first appear far removed from traditional studies of media, it is, in fact, quite indicative of contemporary media practices, and it demands the inquisitive and interdisciplinary approaches that media and screen studies can provide. Robotic surgery employs many of the strategies of mainstream media (in particular, blockbuster film and video game) production, namely an emphasis on sensory immersion through visual, sonic and haptic interfaces. Its technological apparatus thus mimics many of the goals of contemporary media trends such as IMAX, 3D computer imaging, Dolby Atmos, haptic interfaces, and motion control systems (e.g. Microsoft’s Kinect). More broadly, and as I will discuss throughout this article, both performance capture and robotic surgery provide an illustration of our relationship to contemporary media technologies, as well as how media technologies can rework not only our experience of media but also the ways in which we perceive and interact with the world around us.¹

Performance capture has featured prominently in mainstream special effects cinema and video games, including the films The Polar Express (Robert Zemeckis, 2004), King Kong (Peter Jackson, 2005), Beowulf (Robert Zemeckis, 2007), Avatar (James Cameron, 2009), Rise of the Planet of the Apes (Rupert Wyatt, 2011), the Lord of the Rings (Peter Jackson, 2001–2003) and Hobbit (Peter Jackson, 2012–2014) trilogies, and the video games Heavy Rain (David Cage, 2010) and LA Noire (Brendan McNamara, 2011), to name some of the most prominent examples. While performance capture and remote surgery might be employed for different reasons in divergent contexts – that of the media industries and the medical–military industries, respectively – they each produce a human–machine assemblage that enacts itself across different scales. Each technology ‘captures’ a performance, translates that performance into digital information, recodes that performance into another body – be it robotic or digital synthespian – which can then be experienced in the analogue body of the patient or viewer. Each ‘performer’ is, in many ways, acting as a puppeteer, manipulating an extension of his or her body through space and time.² The shift in scale produced by each technology – in terms of movement, perception, experience and sensation – demonstrates the extent to which these technologies of telepresence foster a multilocal experience of the body, the dispersion of authorial control across the human–machine assemblage, and a reinforcement of embodied experience despite an embrace of cultural fantasies of the disembodiment of information.³

Over the last 15 years, ‘vactors’ (virtual actors), synthespians, cyberstars, and performance capture have received growing attention in film, media and screen scholarship. Examples of work in the area include contextualization of performance capture through discussions of the history of animation, rotoscoping and embodiment (Bouldin, 2004; Gunning, 2006); analyses of performance capture’s technological history, apparatus and spectator engagement (North, 2005, 2008); explorations of realism and special effects (Allison, 2011; Prince, 1996, 2012); discussions of the production culture of animation and digital visual effects (Freedman, 2012; Telotte, 2010); a psychoanalytic investigation of digital actors (Creed, 2000); and interrogations of the hybrid/monstrous ontology of performance capture and the uncanniness of the image (Aldred, 2011; Bode, 2006; Brown, 2009; Monnet, 2004). While almost all of these works touch on issues of phenomenology and the body, none takes the embodied relationship between performer and digital visualization as its primary concern. In this article, I take an explicitly phenomenological position, examining the connective tissue that binds actor and avatar, surgeon and robot. The ligaments that connect human and nonhuman both separate and draw the entities close together, and the resultant shifts in scale, perception, and experience will be explored in the remainder of this article.

As indicated by this article’s epigraph, Walter Benjamin, in his seminal ‘Work of Art’ essay (1968[1936]), has briefly, but provocatively, drawn parallels between the work of the camera and that of the surgeon, highlighting the ways in which humans and nonhumans collaborate in the construction of perception and experience. Following Benjamin’s analogy, this article provides a phenomenological reading of this passage and argues that both performance capture and remote surgery penetrate the materiality of the body and reconstitute that materiality elsewhere, as a human’s bodily movements are captured, transmitted, translated, and finally recoded into that of another body, be it an analogue or digital form of embodiment. This ‘elsewhere’ is of a fundamentally different scale than the originary body, having undergone a translation from an analogue body into a digital presence and then back into an analogue sensation.⁴ The multilocal body thus operates on a multitude of scales: the translation of the actor/surgeon’s movement into smaller or larger actions, the increase or decrease of the speed of movement, the time and place of perception, and the distance from which the multilocal body is experienced by both practitioner and viewer. Both remote surgery and performance capture offer a meditation on the complexity of subjectivity and embodiment within a network while at the same time providing a snapshot of the tension between embodiment and disembodiment in contemporary digital culture.

Performance capture and the multilocal body

In the performance capture process, actors must navigate both analogue and digital terrains to achieve a haptic verisimilitude in the digital rendering of their movements and presence. They don a combination of body markers and facial cameras, which translate their performance into a digital environment to be animated. The movements of the actor’s body are captured and uploaded into a computer where an animator adds various textures and ‘skins’ to bring the digital character to life. Frequently, this digital creation is coupled with the voice of the actor, adding a phenomenological persistence of the human voice to an otherwise digital character. The 2011 video game LA Noire, for example, utilized facial performance capture as a fundamental component of its gameplay (see Figure 2). In the game, players are tasked with examining the facial movements and expressions of the game characters in order to assess whether a character is lying or telling the truth. While the game was ultimately unsuccessful in fully realizing its ambitions – after all, it’s very difficult to determine if someone is lying in real life – its technology was successful in capturing the detailed movements (if not emotions) of the actors who played the characters.

Figure 2.

Screenshot of analogue and digital faces in LA Noire—Tech Trailer (2010) by IGN. Available at: http://www.youtube.com/watch?v=aL9wsEFohTw

A production video for the game demonstrates the technology utilized by the game’s designers and programmers (IGN, 2010). The process consists of two parts. In the first part, actors wear a motion capture suit, which records their movement and eyelines. In the second part of the process, dozens of cameras surround and record the actor’s face, and this information is fed into a computer, which processes the data from the cameras and compiles the information into a 3D rendering of the actor’s face. Because the system utilizes such a high amount of visual data, it produces a remarkable fidelity to the subtle facial movements of the actors. As one designer puts it – no doubt interested in promoting his product by incorporating an industrial rhetoric of photorealism – ‘The level of detail is just so high that they’re just like a real person in real life’ (IGN, 2010). An actor’s facial performance is thus captured by a cocoon of cameras (the producers refer to this technology as ‘MotionScan’), translated into a digital avatar, and then transplanted onto a digital body, and this process actualizes Benjamin’s ‘multiple fragments’. The body of the actor is fragmented by the multi-camera setup and then, as Benjamin (1968[1936]: 234) writes, ‘assembled under a new law’. These fragments produce a new whole, which is then attached to the digital body, which is itself the product of performance capture. The resulting interactive image translates and extends the movements and material ‘weight’ and presence of the actor into a different space, allowing for the multilocality – in both time and space – of the actor’s body.⁵

Writing about Beowulf, media scholar William Brown (2009: 162) describes the fundamental shifts in scale that a performance must pass through in its transformation from human body to digital avatar, noting the potentially innumerable alterations that can be made to the digital image. In addition to the spatial transformations of scale discussed by Brown, the performance must also pass through ontological and temporal dimensions of scale. The movements of the actor must be captured and coded by a combination of data point and video recording, translated into digital video information, modified by animators, composited into the live action segments of the shot (if any are featured in the film), and finally translated back into analogue consumption through the viewer. In terms of its temporal component, the performance, like all recorded performances, can be manipulated both in terms of speed (e.g. slow motion) and, necessarily, in terms of consumption: the performance is consumed at a time subsequent to its recording.⁶

This mutation of the scale and ontology of the actor’s body functions as a translation and manipulation, altering the materiality of the body as the process of performance capture extends it into its digital avatar. Gabriela Galati (2011), relating the materiality of new media technologies to the bilocality of religious relics, explores the relationship between the lived body and the projected body. In the history of religion, Galati notes, the cult of relics conceptualized each part of the deceased holy body as equally possessing the spirit of the whole:

The physical contact with the ‘sacred’ was considered of great importance, and each part of the holy person was considered to have exactly the same value, the same sacred characteristics as the ensemble; all the remains were said to have power derived from the saint. (p. 130)

Popular understandings of new media technologies – e.g. avatar identities in programs like Second Life and World of Warcraft, social network profiles, and Skype – conceive of presence in a similar manner. Just as Galati’s theory of bilocation allows for the presence of one individual in two or more places, so too do new media technologies facilitate the extension of the self into multiple locations. Galati, however, complicates this understanding, and she resists the simple conflation of analogue and digital presence, arguing instead that, while digital technologies do indeed extend the self, the multilocal presences are of a differing ontological status. Galati is also careful not to create a dichotomy between analogue embodiment and digital disembodiment, positing that the extension of presence through new media technologies reworks ideas of the body rather than dismantles embodiment.⁷

Though the analogue and digital body might in many ways inhabit different ontological realms, it is the connective tissue binding them together that proves most illuminating in exploring the relationship between actor and avatar. While the question of the ontology of the digital image/being is vitally important – and has been a topic of much concern in contemporary film, media and screen studies (see, for example, Cubitt 2004; Galloway, 2006; Manovich, 2001, 2006; Murray, 2008; Rodowick, 2007) – I want to emphasize the connection between bodies as they travel across and through networks, the ways in which embodiment undergoes a shift in spatial, temporal and experiential scale. In other words, rather than delineating the ontological differences in analogue and digital being, I am more interested in exploring the phenomenological ‘umbilical cord’ that connects the body of actor and avatar, surgeon and robot. Roland Barthes (1981: 6), discussing his experience of photography, argues not only that viewers share a carnal link with the photographic image, but also that the photographic image shares a phenomenological connection with that which it depicts: ‘In short, the referent adheres.’ And while processes of creating analogue photographs differ ontologically from those of creating digital avatars, there still exists an affective and phenomenological connection between body and image.⁸

Delineating analogue and digital bodies along purely ontological trajectories rehearses many of the philosophical pitfalls of classical theorizations of a fundamental mind–body split. One of the biggest challenges of analysing digital imagery and embodiment stems from a reliance on Cartesian theories of the mind, which conceive of subjectivity as housed within the prosthetic of the body, rather than as a process emerging out of an embodied interaction with the world. Those who take a Deleuzian approach to understanding digitality, such as Brian Massumi (2002), Anna Munster (2006) and Jussi Parikka (2010), offer a way around the limitations of Cartesian thinking. The approach of these writers highlights the role of technology in refiguring the body, rather than standing apart from it. Subjectivity is viewed as dispersed throughout a network, the result of the modulations of sensation emerging from an interaction with the affects, intensities, and foldings of an environment and its human and nonhuman inhabitants. Thus, rather than conceiving of digitality as a technological apparatus distinct from the body, the digital is instead viewed as an assemblage that realizes itself through the interaction of human and (nonhuman) technology.

Embodiment persists within such a schema, though we must take care to remember that embodiment here is not a static phenomenon but rather an ongoing and perpetually emerging process, and thus subject to modulations depending on our point of contact with the body (i.e. the actor or the avatar, the surgeon or the robot). The intensive flows of the analogue body interact and combine with the flows of the digital body, creating new and divergent forms of embodiment. Bodies, within such a framework, become more than simply meat upon which the ‘software’ of consciousness can run.⁹ Embodiment becomes crucial to subjectivity, and this extends beyond the analogue world into the digital. Bob Rehak (2003: 123), writing about digital avatars, reinforces this perspective on embodiment, arguing that we ‘take the body with us in the form of codes and assumptions about what does and does not constitute a legitimate interface with reality – virtual or otherwise’. The relationship between human and avatar is not one of either simple projection or straightforward extension of presence. It is, instead, a complex and ever-changing reconfiguration of human and nonhuman elements.

The camera and the surgeon

A promotional image from Intuitive Surgical functions as a pictorial summation of this article’s argument (see Figure 3). In the image, a robotic ‘hand’ is superimposed over the gloved hand(s) of a surgeon, encouraging a comparison of movement and utility between the human and nonhuman beings. The image also uses a repetition of the hands in order to convey a sense of motion and to draw parallels between the physiognomy – fingers and wrist – of the surgeon and the robot. Occupying the foreground of the image, the robotic hand is given subtle priority over the surgeon’s hand, but at the same time, the size and scale of the human hand dwarfs the robotic implement, implying a sense of command over the tool. My dual reading of the image highlights the ambivalent nature of authorship not only in regards to robotic surgery but also in regards to performance capture. Neither surgeon nor robot is in control; rather, control emerges from the human–nonhuman assemblage. The image also calls attention to issues of scale, accentuating the discrepancy in size between the two figures as well as the smaller, more precise movements of the robotic hand.

Figure 3.

This image, however, also traffics in transhuman fantasies of disembodiment and the easy exchange of information between mediums. Elided here is the intervening mechanical and computing apparatus that connects the surgeon and the robot. Depicted instead is a seamless interaction between bodies, neglecting the processes of translation and transformation that accompany any change in medium. The technologies of both remote surgery and performance capture can also tell us something about the role of animation in translating analogue human sensations and movements into a digital environment. In each case, a human’s bodily movements are captured and translated into a digital computing environment. In performance capture, an animator’s computing algorithms transform an analogue body into a digitally animated avatar. In remote surgery, the role of animation is less visible, but the surgeon’s movements nevertheless ‘animate’ the robotic tools.

I conceive of both the actor–avatar and the surgeon–robot assemblages as signal examples of the intertwining of analogue and digital embodiment, and it is out of these assemblages that a hybrid form of embodiment emerges. The body, in this scenario, is not simply a matter of analogue embodiment and digital disembodiment. Rather, as is also the case in the relationship of actor to avatar, the relationship between surgeon and robot produces a new kind of embodiment, one that consists of the human body of the surgeon, the algorithms that translate this body into a specific kind of code, and the robotic apparatus that interprets and realizes this embodiment in its mechanical form. In the case of robotic surgery, the computer algorithms that link the analogue and digital bodies together function as a central process that unites each body under a new regime of hybrid embodiment. The algorithm interacts with the surgeon (both in terms of vision and haptics), produces a magnified and manipulable image of the body, and acts on the patient’s body. The hybrid body resulting from such a reconfiguration exemplifies what Mark Hansen (2006) calls a ‘body-in-code’ and what Beth Coleman (2011) terms ‘X-Reality’. For Hansen, all reality is experienced as a mixed reality – an imbrication of digital and analogue experience – and the body-in-code is the kind of being that inhabits this reality. Describing the body-in-code, Hansen (2006: 20) writes:

By this I do not mean a purely informational body or a digital disembodiment of the everyday body. I mean a body submitted to and constituted by an unavoidable and empowering technical deterritorialization – a body whose embodiment is realized, and can only be realized, in conjunction with technics. (emphases in original)

Coleman (2011: 20), emphasizing the everyday nature of our interactions with digital visual technologies, similarly argues that X-Reality is ‘an engagement of networked media integrated into daily life, perceived as part of a continuum of actual events’.

Both performance capture and robotic surgery produce multiple body-in-code ‘segments’ as well as a larger human–nonhuman assemblage that can itself be considered a body-in-code. Examples of the body-in-code segments include surgeons interacting with their consoles, patients on a table with an endoscope in their bodies, actors wearing facial markers and animators manipulating the captured performance. The larger body-in-code assemblages would include the entire robotic surgical system (including broadband infrastructure and attending nurses) and the whole system of performance capture (including the apparatuses of capture and exhibition). Embodiment thus extends past the boundaries of the actual bodies of the robot and surgeon, actor and avatar, and emerges from the cooperation and interaction of a particular technological assemblage. As these examples make clear, traditional concepts like ‘mind’ and ‘body’ make little sense once subjectivity and embodiment extend past the boundaries of the molar body.

What these bodies-in-code share is a notable fragmentation of experience, and in her essay on cybersurgery and (dis)embodiment, Julie Doyle (2007) uses Benjamin’s thoughts on the camera and the surgeon to offer a critique of the ways in which cybersurgery and its processes of visualization function to fragment the body, presenting it as a collection of small pieces rather than a holistic unity. For Doyle, ‘digital imaging and (computer manipulated) surgical technologies penetrate and mediate the body, presenting an image of the fragmented body to the surgeon through which the material practice of surgery is remotely performed’ (np), and these surgical ‘technologies fragment the body and prioritise its image over an embodied reality’ (np). Just as a film is the product of piecing together image fragments, so too is cybersurgery when it breaks down and reconstitutes fragments of the body, obscuring a perception of the unified totality of the interaction of those pieces.

Two of the major researchers and developers of robotic surgery are Intuitive Surgical, which produces the da Vinci and ZEUS Surgical Systems, and SRI (Stanford Research Institute) International, which manufactures the M7 Surgical Robot. Each developer produces a similar product, which shares much with an interactive arcade gaming rig: a robotic apparatus that utilizes haptic feedback interfaces, high definition stereoscopic video, and immersive sound technology in order to allow surgeons to extend their bodies into multiple locations.¹⁰ These remote surgical technologies function as a mechanical prosthetic for surgeons, allowing them to translate, refine and transmit bodily movements over great distances. The technology functions to improve and refine the motions of the surgeon, facilitating, for example, operations on a beating heart, which obviates the need for the heart–lung machines used in cardiopulmonary bypass surgeries. The robot syncs the surgeon’s movements with the motion of the beating heart, creating a human–machine assemblage that utilizes the skillset of both surgeon and robot. As media scholars Timothy Lenoir and Sha Xin Wei (2002) argue, robot-assisted surgery, through its dispersion of control and subjectivity, deconstructs the notion of the ‘surgeon–author’, replacing the surgeon-as-expert with the surgeon-as-data manager. At issue here are questions of control: is it the robot or the surgeon who performs the surgery? This question of control may soon become more pressing as the Becnel Law Firm, LLC and Riley & Jackson, Attorneys at Law have established a website (http://badrobotsurgery.com/) to bring lawsuits on behalf of patients who have suffered from ‘bad robot surgery’. While the justice system will need to sort out liability and fault – the hospitals, doctor, software designer, hardware engineer, IT supervisor, etc. – from my purview, the question of robot–surgeon control is misguided. Control emerges not from either/or but rather from the effects of this human–nonhuman technological assemblage.

A closer examination of the operating room procedures of robotically mediated surgeries reinforces this view that cybersurgery, through its technologies of vision, serves to fragment and reassemble the body. In the da Vinci surgical system (produced by Intuitive Surgical), the arrangement of the operating room consists of three primary equipment carts: the patient cart, the vision cart and the surgeon’s console. The surgeon’s console is the master of this system, and it is connected to and controls both the patient cart and the vision cart. The patient cart is stationary and is positioned next to the operating table on which the patient undergoing the procedure lies. This cart contains all of the cameras, tools, and robotic implements that will be acting on the actual body of the patient, and it is overseen by a surgical assistant, standing near the patient and patient cart, who watches to confirm that the surgery is proceeding as planned. The robotic implements on the patient cart enter the patient’s body through small (1–2cm) incisions (Intuitive Surgical, 2013b). The vision cart, which is essentially a high-tech TV stand, is positioned near the patient cart, and it provides a 2D view of what the surgeon sees in his or her console. The images on the vision cart are available to the entire operating room team, allowing the surgeon’s view to be shared with the entire surgical staff. This double vision – between the surgeon’s console and the vision cart – is the first method by which the body of the patient becomes fragmented.

The surgeon’s view from his or her console is the second way in which the robotic surgical apparatus fragments the body. Rather than providing a holistic view of the body, the da Vinci system focuses the surgeon’s attention on a particular site of the patient’s body, providing an enhanced, but limited, view of the small area. A small camera captures the relevant area, and according to promotional materials from Intuitive Surgical, ‘the vision system is equipped with a high-definition, 3D endoscope (flexible tube with a camera and light at the tip) and image processing equipment that provides true-to-life images of the patient’s anatomy’ (Intuitive Surgical, 2013d). This endoscope is viewed through the surgeon’s console, which, through the use of stereoscopic lenses, provides the surgeon with a high-resolution, 3D image of the surgical site. This endoscope also incorporates a digital zoom functionality, which produces up to 10x magnification (Intuitive Surgical, 2013b). Additionally, both the surgeon’s console and the vision cart incorporate a head-up display (HUD), which overlays information about the tool being used, the magnification level, and the screen size onto the image of the patient’s body. In this way, the computer mediates the surgeon’s access to the patient’s body – both in terms of visibility and information overlay – and it reassembles the body as a robot–computer–human hybrid.

A final example of fragmentation and reassembly is worth noting. As a training and observation tool, the da Vinci system also allows for the use of dual surgeons’ consoles (Intuitive Surgical, 2013c). Within this arrangement, two surgeons on two different consoles can see the same image. If this setup is being used as a training tool, control over the system can be transferred easily between student and teacher. If this setup is being used for collaborative surgery, the surgeons can operate in concert, each controlling a different aspect of the robotic implement. Within this scenario, access to the patient’s body – as well as control over the robotic surgical system – is further fragmented, with control dispersed among two surgeons and the robot.

Whereas Doyle’s critique highlights this fragmentation while making an important intervention in our understanding of the processes of visualizing the patient’s body, my own interests lie more with the relationship between the surgeon’s body and the body of the robotic apparatus (as well as the relationship between actor and avatar). Doyle does, however, share my concern with the embodied experience of technology, a concern also articulated by Benjamin. Much of the rhetoric surrounding both robotic surgery and performance capture technologies emphasizes the immersive nature of the technological apparatus, the extent to which the technology ‘disappears’ during use, presenting the human–machine assemblage as a seamless merger of flesh, hardware, and digital processing. As Benjamin (1968[1936]: 234) asserts, the technology is understood as working through bodies, offering a ‘permeation of reality with mechanical equipment’. It is important to note, however, that this permeation of reality only produces the appearance of being ‘free of all equipment’. Because of the extent to which humans and technological apparatuses become intertwined with each other, merging their bodies and reworking their mutual techniques of perception, this equipment is frequently positioned as a non-actant, a passive conduit through which human experience flows.

Within the rhetoric of the industry, humans maintain complete agency, and the technologies are subservient to the desires of their human users. As Intuitive Surgical (2013a) describes it, ‘the immersive quality of the 3D vision provides a virtual extension of the surgeon’s hands and eyes into the patient’s body.’ Here we can see the way in which the technology, through its immersive visual and haptic systems, is understood as providing a seamless extension of the surgeon’s body, through the computer and robot, into the patient’s body. Describing his team’s M7 robotic surgical apparatus, Thomas Low, director of the medical systems and telerobotics program at SRI International, reinforces this idealization of a technological prosthesis:

The combination of the audio field, the 3D visuals, the force feedback, and everything aligned properly … quickly you become so immersed in the environment that, as I said before, the technology disappears, and you’re just doing the job. But you may be 10,000 miles away. (Scoble, 2011, emphasis added)¹¹

Overlooked within this understanding of the relationship between technology and user are the intervening processes of image rendering and kinesthetic mediation. Also ignored is the complexity of the human–nonhuman assemblage, as well as the processes of translation through which perception must pass on its way to becoming sensation and experience. Moreover, and as has been implicit in my discussion of robotic surgery thus far, this conceptualization of technology as extending one body into another functions as a means for the surgeon to see and be at a distance. The surgical technology allows for perception at a distance: the extension of the senses, through technological mediation, into a proxy body (that of the cameras and robotic arms). While this kind of rhetoric is quite appealing from a technological fetishist point of view, it overlooks both the processes of mediation at play and the different modes of embodiment engendered through this mediation and translation.

Mimesis and the body’s scalar travels

The resultant bodies-in-code of both the robot and the digital avatar share a persistent link with their parent bodies, and their linkage – and reflection of each other – embodies both what Benjamin (1978) called humankind’s ‘mimetic faculty’ and what philosopher Maurice Merleau-Ponty (1962) understood as the reversibility of subject/object relationships. As Jennifer Barker (2009: 19) interprets it:

For Merleau-Ponty this double sensation provoked by one hand touching the other is the archetype for the subject/object relations in the world: irreducible one to the other, but embedded in a constantly mutual experience, constituted by the same ‘stuff’.

In performance capture, actors’ bodies both touch and are touched by their digital avatars in a recursive feedback loop. Through the process of viewing their digital avatars in real time, actors can adjust their movements based on the screen image, and the screen image itself adjusts to the actors’ movements. Neither is reducible to the other, and neither actor nor avatar ‘loses itself’ in the image of the other. Rather, the subjectivity and presence of both actor and avatar emerge in their interaction, partly due to the fact that the digital performance is invested with human emotions while at the same time, the human performance is invested with the technological apparatus of the capture equipment. In other words, the ‘actor’ and the ‘avatar’, as categories of being, emerge only out of their interaction as an assemblage, and their relationship constitutes their unique subjectivity and embodied presence.

Actor Andy Serkis, renowned for his pioneering performance capture work in The Lord of the Rings (Peter Jackson, 2001–2003) and Hobbit (Peter Jackson, 2012–2014) trilogies, King Kong (Peter Jackson, 2005) and Rise of the Planet of the Apes (Rupert Wyatt, 2011), echoes this reading of the relationship between actor and avatar. While it is important to keep in mind that Serkis and the relevant production studios have vested personal, professional, and economic interests in promoting Serkis’s acting skills – most recently James Franco (2012) campaigned for an Oscar nomination recognizing Serkis’s work in Apes – this kind of rhetoric, as demonstrated by Yacov Freedman (2012), is useful in examining how performance capture is understood and defined by practitioners in the industry. Serkis, in a behind the scenes production video for Apes, describes the performance capture process thusly: ‘The basic usage of performance capture is, to see it on the screen, you will see the apes, but they are apes which are infused with the heart and soul of an actor’s performance’ (movieclipsTRAILERS, 2011). Serkis describes his process in many of the same terms as Method actors – he uses his emotional background in order to give life to his characters – and this, as Dan North (2008: 162) indicates, is perhaps constitutive of the captured performance. Invested in each other, the human actor and digital avatar share a similarity of movement and presence, though they are separated by the translation of code over distance and, ultimately, through time.

This temporal dimension is where performance capture and remote surgery most notably diverge from each other. The temporal gap between the production and consumption of the performance capture avatar displaces the haptic and material connection between the human actor and the digital synthespian over a wide scale of time, from the almost simultaneous experience of an actor watching his or her digital ‘puppet’ perform in real time to a consumption that could potentially take place decades or centuries after the actor’s death. In this way, the consumption of performance capture shares much with Barthes’ ‘umbilical cord’ previously discussed. The production and consumption of remote surgery, conversely, tends to take place simultaneously, though it is not difficult to imagine a future scenario in which routine operations could be completed by an autonomous, preprogrammed robot.¹²

Regardless of these spatial gaps and shifts in temporal scale, the human body in a very real and material way phenomenologically persists within its digital avatar, and the digital avatar persists within the human actor. This, perhaps, is symptomatic of our contemporary experience of human and technological assemblages, what WJT Mitchell (2005: 321) describes as a logic of ‘biocybernetic reproduction’, which he articulates as a 21st-century update of Benjamin’s ‘mechanical reproduction’:

In the age of biocybernetic reproduction, two new figures have appeared on the scene. The cameraman is replaced by the designer of virtual spaces and electronic architectures, and the surgeon adopts the new techniques of remote, virtual surgery. The surgeon operates at an unnatural distance from the patient’s body, performing his gestures in a remote location – another room, perhaps even another country. He moves his hands inside data gloves like a shaman, making passes over a virtual body and removing a virtual tumor with sleight of hand. He is able to rehearse his movements on a virtual body many times before the actual operation takes place. The digital miniaturization of his movements allows him to cut deeper and finer than any operation conceivable in traditional manual surgery.

In biocybernetic reproduction, Mitchell is diagnosing a powerful fantasy of genetic and cybernetic technologies that appear almost as magic. Because of the radical shifts in scale and the ‘invisibility’ of the complex processes of translation that connect the surgeon and robot, actor and avatar, the technological apparatus seems to disappear, leaving only an unmediated mimetic behavior that seemingly allows a person to extend his or her presence into multiple locations.

While I find this rhetoric of technological disappearance interesting as a symptom of cultural fantasies of informational disembodiment, it does not adequately unpack the complexity of the phenomenological and material interaction of human and nonhuman machine. In other words, rather than viewing remote surgery as a utopian form of transhuman transfer of consciousness between mediums, I view the human–machine assemblage in Hansen and Coleman’s terms, as a material experience of a mixed reality. This view is articulated, with variations, by Marshall McLuhan (1994[1964]), who understands media as an extension of the senses; Merleau-Ponty (1962: 165–166), who, through his example of the blind man’s stick, understands technology as incorporated into one’s bodily schema; and Benjamin, who, as previously discussed, understands the relationship between ‘surgical penetration’, filmic mediation, and perception as a ‘thoroughgoing permeation of reality with mechanical equipment’. Within each framework, technology becomes incorporated into the experience of daily life, and the extension of perception and subjectivity of an actant through a network appears as a naturalized form of embodiment.

Both remote surgery and performance capture technologies rely on the extension of the human subject, through technology and digital translation, into a robotic or digital avatar. At the same time, these avatars provide feedback to their human counterparts, creating a recursive loop of perception and experience. Each technology fits into a much larger context of the experience of media as ‘perception at a distance’. While outside the scope of the current article, the links between performance capture, remote surgery, and media in general, which allows information to travel beyond the limitations of human speed and movement, would add much to the discussion of the links between embodiment and information. This concept of perception at a distance has been discussed, with different valences to be sure, in McKenzie Wark’s (2012) conceptualization of ‘telesthesia’, Lisa Parks’s (2005) theorization of ‘remote sensing’ and the televisual, and Timothy Lenoir’s (2002) discussion of ‘medialization’, the process by which the medical body is transformed into the digital body. What each framework shares is an emphasis on the embodied and intersensorial experience of telepresence, and each accounts not only for the ways in which the human subject is extended across time and space but also for the ways in which this extension itself alters the nature of perception and experience.

As the Google image search with which I opened this article attests, Benjamin’s links between the camera and the surgeon speak to a fragmentation of experience facilitated by the ‘new law’ of human–machine assemblages within digital visual technologies. In this article, I have tried to maintain this emphasis on embodiment and materiality, exploring the phenomenological implications of ‘technologies of presence’ and the ways in which they actualize a persistent cultural fantasy that imagines a fundamental translatability of flesh and information and the dispersion of body and mind throughout an informational network. Our experience of contemporary visual and material culture is one marked by a powerful mediation of embodiment and subjectivity, from the experience of social networks to multiplatform media consumption to ubiquitous image capture and exhibition technologies. The experience of the self in such a media environment is one of multilocality, wherein our material and virtual presence becomes dispersed throughout a network, subject to the modulations and translations of visual and computing technologies. While it is very easy to slide into transhumanist fantasies of informational disembodiment, we must always be mindful of the ways in which embodiment, perception, and sensation persist within – and are altered by – the forces and channels through which the self is extended through these networks of communication and vision.

The multilocal self, rather than simply extending the body and subjectivity into new digital and material terrains, both produces new beings and alters the originary self. As the examples of performance capture and robotic surgery demonstrate, the relationship between ‘subject’ and ‘object’, actor and avatar, and surgeon and robot is one of reciprocation, and it is vital to understand not only how digital technologies of vision produce new hybrid beings but also how these technologies reconfigure both our sense of self and our ideologies of control. The multilocal, techno-embodied self is constitutive of the experience of digital, networked environments, and this condition defines the ways in which we inhabit and experience the world. The various modes of perception and experiences of scale facilitated by these technologies rework our embodied perception, allowing us to travel through these digital spaces while also generating new actants and modes of existence. The materiality of perception is fundamental to a sense of self, and the position of digital actors and robot surgeons within the world not only calls attention to the complexity of the relationships between actants in material–semiotic networks but also reinforces the extent to which humans and nonhumans become imbricated in such networks, each completing the other in a reciprocal, mutually constitutive relationship.

Footnotes

Funding

This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.

Notes

Author biography

Drew Ayers is an Assistant Academic Specialist in the Media and Screen Studies program at Northeastern University. His work has appeared in Film Criticism, Configurations, Scope, and In Media Res, and his essay, ‘Bleeding synthetic blood: Flesh and simulated space in 300’, will appear in a forthcoming anthology on special effects. Dr Ayers is currently at work on a manuscript that explores the ‘vernacular posthumanism’ of contemporary film, media, and visual culture.

References

Aldred

(2011) From synthespian to avatar: Re-framing the digital human in Final Fantasy and The Polar Express. Mediascape. Available at: http://www.tft.ucla.edu/mediascape/Winter2011_Avatar.pdf (accessed 8 December 2013).

Allison

(2011) More than a man in a monkey suit: Andy Serkis, motion capture, and digital realism. Quarterly Review of Film and Video 28(4): 325–341.

Barker

(2009) The Tactile Eye. Berkeley: University of California Press.

Barthes

(1981) Camera Lucida: Reflections on Photography. New York: Hill and Wang.

Benjamin

(1968[1936]) The work of art in the age of mechanical reproduction. In: Arendt

(ed.) Illuminations. New York: Schocken Books, 217–251.

Benjamin

(1978) On the mimetic faculty. In: Demetz

(ed.) Reflections. New York: Schocken Books, 333–336.

Bode

(2006) From shadow citizens to Teflon stars: Reception of the transfiguring effects of new moving image technologies. animation: an interdisciplinary journal 1(2): 173–189.

Bouldin

(2004) Cadaver of the real: Animation, rotoscoping and the politics of the body. Animation Journal 12: 7–31.

Braidotti

(2013) The Posthuman. Malden, MA: Polity Press.

10.

Brown

(2009) Beowulf: The digital monster movie. animation: an interdisciplinary journal 4(2): 153–168.

11.

Coleman

(2011) Hello Avatar: Rise of the Networked Generation. Cambridge, MA: MIT Press.

12.

Creed

(2000) The cyberstar: Digital pleasures and the end of the unconscious. Screen 41(1): 79–86.

13.

Cubitt

(2004) The Cinema Effect. Cambridge, MA: MIT Press.

14.

Doyle

(2007) Cybersurgery and surgical (dis)embodiment: Technology, science, art and the body. Transformations 15.

15.

Franco

(2012) Oscar Exclusive: James Franco on why Andy Serkis deserves credit from actors. Awardsline. Available at: http://www.deadline.com/2012/01/oscar-exclusive-james-franco-on-why-andy-serkis-deserves-credit-from-actors/ (accessed 23 August 2013).

16.

Freedman

(2012) Is it real … or is it motion capture? The battle to redefine animation in the age of digital performance. The Velvet Light Trap 69: 38–49.

17.

Galati

(2011) Presence, telepresence, images and the self. Technoetic Arts: A Journal of Speculative Research 9(2–3): 129–134.

18.

Galloway

(2006) Gaming: Essays on Algorithmic Culture. Minneapolis: University of Minnesota Press.

19.

Gunning

(2004) What’s the point of an index? Or, faking photographs. NORDICOM Review 25(1–2): 39–49.

20.

Gunning

(2006) Gollum and golem: Special effects and the technology of artificial bodies. In: Mathijs

Pomerance

(eds) From Hobbits to Hollywood: Essays on Peter Jackson’s Lord of the Rings. Amsterdam: Rodopi, 319–350.

21.

Hansen

MBN

(2006) Bodies in Code: Interfaces with Digital Media. New York: Routledge.

22.

Hayles

(1999) How We Became Posthuman: Virtual Bodies in Cybernetics, Literature, and Informatics. Chicago: University of Chicago Press.

23.

IGN (2010) LA Noire–Tech Trailer. Available at: http://www.youtube.com/watch?v=aL9wsEFohTw (accessed 23 August 2013).

24.

Intuitive Surgical (2013a) 3D HD Vision System. Available at: http://www.intuitivesurgical.com/products/davinci_surgical_system/features_benefits/3Dvision.html (accessed 23 August 2013).

25.

Intuitive Surgical (2013b) Core Technology of the da Vinci Surgical System. Available at: http://www.intuitivesurgical.com/products/davinci_surgical_system/features_benefits/ (accessed 23 August 2013).

26.

Intuitive Surgical (2013c) da Vinci Si Optional Dual Console. Available at: http://www.intuitivesurgical.com/products/davinci_surgical_system/davinci_surgical_system_si/dualconsole.html (accessed 23 August 2013).

27.

Intuitive Surgical (2013d) The da Vinci Surgical System. Available at: http://www.intuitivesurgical.com/products/davinci_surgical_system/ (accessed 23 August 2013).

28.

Kim

(2001) Phenomenology of digital-being. Human Studies 24(1–2): 87–111.

29.

Lenoir

(2002) The virtual surgeon: Operating on the data in an age of medialization. In: Thurtle

(ed.) Semiotic Flesh: Information and the Human Body. Seattle: University of Washington Press, 28–51.

30.

Lenoir

Wei

(2002) Authorship and surgery: The shifting ontology of the virtual surgeon. In: Henderson

Clarke

(eds) From Energy to Information: Representation in Science, Art, and Literature. Stanford, CA: Stanford University Press, 283–308.

31.

Manovich

(2001) The Language of New Media. Cambridge, MA: MIT Press.

32.

Manovich

(2006) Image future. animation: an interdisciplinary journal 1(1): 25–44.

33.

Massumi

(2002) Parables for the Virtual: Movement, Affect, Sensation. Durham, NC: Duke University Press.

34.

McLuhan

(1994[1964]) The Medium is the Message. Understanding Media: The Extensions of Man. Cambridge, MA: MIT Press, 7–21.

35.

Merleau-Ponty

(1962) Phenomenology of Perception. New York: Routledge.

36.

Mitchell

WJT

(2005) What Do Pictures Want? The Lives and Loves of Images. Chicago: University of Chicago Press.

37.

Monnet

(2004) A-Life and the uncanny in ‘Final Fantasy: The Spirits Within’. Science Fiction Studies 21(1): 97–121.

38.

movieclipsTRAILERS (2011) Rise of the Planet of the Apes (2011) HD Weta Featurette Making of Behind the Scenes. Available at: http://www.youtube.com/watch?v=XM9Pvfq1KhE (accessed 23 August 2013).

39.

Munster

(2006) Materializing New Media: Embodiment in Information Aesthetics. Hanover, NH: Dartmouth College Press (University Press of New England).

40.

Murray

(2008) Digital Baroque: New Media Art and Cinematic Folds. Minneapolis: University of Minnesota Press.

41.

North

(2005) Virtual actors, spectacle and special effects: Kung fu meets ‘all that CGI bullshit’. In: Gillis

(ed.) The Matrix Trilogy: Cyberpunk Reloaded. London: Wallflower Press, 48–61.

42.

North

(2008) Performing Illusions: Cinema, Special Effects and the Virtual Actor. London: Wallflower Press.

43.

Parikka

(2010) Insect Media: An Archaeology of Animals and Technology. Minneapolis: University of Minnesota Press.

44.

Parks

(2005) Cultures in Orbit: Satellites and The Televisual. Durham, NC: Duke University Press.

45.

Prince

(1996) True lies: Perceptual realism, digital images, and film theory. Film Quarterly 49(3): 27–37.

46.

Prince

(2012) Digital Visual Effects in Cinema: The Seduction of Reality. New Brunswick, NJ: Rutgers University Press.

47.

Raengo

(2013) On the Sleeve of the Visual: Race as Face Value. Hanover, NH: Dartmouth College Press (University Press of New England).

48.

Rehak

(2003) Playing at being: Psychoanalysis and the avatar. In: Wolf

MJP

Perron

(eds) The Video Game Theory Reader. New York: Routledge, 103–127.

49.

Rodowick

(2007) The Virtual Life of Film. Cambridge, MA: Harvard University Press.

50.

Scoble

(2011) Robots at SRI that help with remote surgery, part II (part of our SRI International Tour). Available at: http://www.youtube.com/watch?v=_ghwOLCA35o (accessed 23 August 2013).

51.

Telotte

(2010) Animating Space: From Mickey to Wall-E. Lexington: University Press of Kentucky.

52.

Wark

(2012) Telesthesia: Communication, Culture, and Class. Malden, MA: Polity Press.