Bodily Intra-actions with Biometric Devices

Abstract

We investigated the interface between biomedia and humans by inviting participants to interact with biometric devices that measured and visualized their body data. At first, they struggled with the alienating and disembodying nature of the devices and the constrained, reductionist representation of data. Through their bodily interactions with these devices, however, participants reframed the data and inserted their bodies into the process of data collection. Drawing on the ideas of Bergson, Grosz, Merleau-Ponty and Bachelard, we argue that by working with their subjectivity in a mediated process of becoming, participants ‘filled in the intervals’ of the visual representations of their bodies to interpret them in ways that marked the duration and meaning of their selves. We conclude that even when presented with artificial representations, individuals convert the representation of the data into narratives inspired by their embodied experience, and the ‘virtual’ pasts of their own lives.

Keywords

Bachelard Bergson digital embodiment

Digital representations are frequently accused of flattening or stratifying human experience, portraying the human self as fundamentally computable (Cheney-Lippold, 2011; Galloway, 2004; Stafford, 2009; Rose, 2007). While recognizing the relevance of such arguments, in this article we reconsider the apparently unidirectional relationship between machine and body such critiques may imply. Instead, we view this relationship as an unfolding narrative marked by ongoing, mutually constitutive intra-actions between machines and the humans who use them. Our inquiry focuses on biometric devices used to measure human biological characteristics such as electrocardiograms (ECGs) and electroencephalograms (EEGs), where the potential for a digitized alienation of the body seems particularly clear. These machines, which visualize body data such as heart rate and brainwave frequency, are increasingly used in inexpensive consumer-grade applications to monitor athletic performance or health issues. They are an apt site for probing the contemporary human–machine interface because they indulge our desire to probe our internal biomechanics, even while their computational logics fracture and disembody the human subject.

Our team initiated the Biomapping research project to query our relationships with such machines that intimately probe, process, and visualize our bodily matter. Although we found considerable evidence in support of the alienating impact of digitized bodily representations, we also discovered that, when allowed to play and ‘tarry’ with these technologies, users created dynamic, reflective relationships with the machines that can be characterized as productive, affective, and intra-active (Barad, 2006). These findings indicate the importance of considering the porous boundary between the experimental apparatus and humans (Blackman, 2012, 2014; Despret, 2008) and the narrative space that is opened up when users are allowed to interrogate digital representations.

An initial stage of our research involved filming and interviewing people as they wore and interacted with both an ECG and an EEG sequentially, observing them as they negotiated the automated renderings of their biodata. On balance, there are few technological experiences with more potential for creating a sense of disembodied alienation than seeing one’s physical self portrayed two-dimensionally as data via algorithmic code. As equipment that purports to measure and visually convey bodily activity, biometric tools are uniquely situated to influence users’ images, relationships to our bodies, and our understandings of ourselves as subjects necessarily improved by technologies. The ‘scientific’ data that is pulled and processed by the device’s ‘black box’ of computer algorithms and then translated into graphic visuals provide users with an image of something that is at once intensely personal and yet distanced: the functioning of our unseen, but felt body systems.

We found that initially, almost all participants in our study noted the alienating nature of the visual portrayal of their body data captured by these biometric machines and rendered on computer screens. However, our experiment allowed participants to play with and challenge the equipment measuring their data, resulting in some unexpected outcomes. Our user-testers dealt with constrained visual representations by employing other ways of making sense of their bodies relationally and poetically; they ‘tarried’ with memory, experience, and pleasurable forms of embodiment to create more complex narratives in the virtual spaces between the linear points of their digital biodata representations. This indicates that, although the materialization of data from the body can have a disembodying effect, it appears that data can be ‘re-embodied’ through playful experimentation in ways that supplement the meaning of digital visualizations.

Our analysis probes the commentary users provided in this re-narrativization in order to understand how they engaged with visuals that reify potentially disembodying computational logics. Our theoretical approach uses Bergson’s understanding of perception, the phenomenological approaches of Bachelard and Merleau-Ponty, and feminist theories of embodiment to address the role of the body, affect, and intuition in comprehending information. We suggest that body data may indeed be ‘artificially’ represented by biometric devices to reflect the computational logic of the code that captures and produces it. Instructively, our participants responded to this artificiality by ‘filling in’ the intervals between the discrete algorithmic moments of data capture by drawing on affective experiences, including memories and powerful embodied ‘intra-actions’ with the machines, turning data representations into unfolding stories. As Grosz (2005: 106) advises, ‘what duration, memory, consciousness bring to the world is the possibility of an unfolding – a narrative – a hesitation. Not everything is presented all at once.’

We describe this resistive and lively practice in Barad’s (2006) terms as ‘intra-action’, or an entanglement between the machines and our user-testers. This entanglement indicates a distribution of experience across what is clearly a porous boundary between human and non-human actors (Blackman, 2014: 371) that may be related to the very nature of our experiment. As Suchman (2007) notes, for users to engage in active, embodied involvement with machines in ongoing, contingent, and co-productive practices, we must understand the rules that structure machines. She calls for ‘situated actions’ – both planned and creative – that allow users to reflect on proscriptive (machinic) instructions, to reject responding to plans and context, and instead to act in reflexive response to things generated. For Despret (2008), when experiments allow for the more equal distribution of control between apparatus and participants that such practices encourage, a constant process of attunement in the experimental process becomes evident. In this ongoing process, there is a mutual construction of meaning between machine and participant, where interpretations of the data emerge over time.

These findings are relevant to studies that consider digital technology as constitutive of human experience (Ihde, 1990; Munster, 2006; van Dijck, 2007; van Doorn, 2011; Verbeek, 2011). In particular, they provide important insights into Featherstone’s (2010) and Hansen’s (2003, 2004) explorations of the relationship between digital media, flows of affect, and the body. Specifically, they affirm Featherstone’s (2010: 211) emphasis on digital representations as processes of engagement that draw on affective involvement with the human body to shape meaning. Frequently, our user-testers’ bodily experimentation with the machinery allowed them a way into interpretations of their body data that clearly relied on being ‘affected’ by the experience. This, in turn, supports the idea that material objects are actors that must be addressed if we are to enhance our ethical understandings of ourselves as human (Bennett, 2010; Latour, 2004).

The Biomapping Project

Biomapping was a four-year, interdisciplinary research creation project involving a team of artists, social scientists, and creative coders. Our goal was to create experiments whereby participants aesthetically represented or responded to their own biodata, allowing them to consider computational framings of subjectivity reified by common body-measuring technologies. This project critically addressed familiar practices of reducing the self to computable data, a biological reduction that causes consumers to accept science unproblematically and reduces our understandings of self, subjectivity, and cognition to a neoliberal sameness (Deleuze, 1992), encouraging the brain to work on remote control (Stafford, 2009). Biomapping sought to expose how biometric tools and visualizations enforce such cultural practices and addressed contentious conceptions of the self as fundamentally computable (Cheney-Lippold, 2011; Galloway, 2004; Rose, 2007). We invited users to understand these tools by exploiting them in transgressive practices – to capture their own data, to process it in 2D, 3D, or 4D form, and finally to ‘print’ it as a visual, an artifact or even as a poem. In short, Biomapping endeavoured to expose contemporary discourses that frame humans as computable, and to expose their ontological groundings and epistemological products by inviting users to engage with all levels of the computational regime – from data collection, to processing and printing.

As an initial stage of our research, we wanted to understand what our user-testers thought about augmenting their bodies with consumer-grade biometric devices and how they understood the visuals that represented and interpreted their body data. To that end, we held a series of user-testing trials to examine how people interacted with biometric technologies. In these trials, participants were invited to employ an EEG and an ECG sequentially, and to comment upon and interact with the visualizations of their body data.¹ As they experimented with the technology, we observed their interactions with the devices, questioned participants on their interpretations of the data, and documented their affective and physical reactions. As we expected, many participants demonstrated frustration, alienation, and even irritation with the machines as they struggled to interpret and to control the ‘scientific’ digital representations portraying their bodily functions. We also noticed, however, an interest in, and determined effort to understand, the technology in its own terms, and also to read the digitized visual representation of their bodies in ways that relied on intuitive memory, affect, and even poetic recourse.

Our users’ simultaneous interest in comprehending the machines’ logic, coupled with their continual input of other context (bodily experience, memory, affect, metaphor) to re-contextualize their data, suggest a complex and productive querying of subjectivity. Their comments suggested a willingness to both comprehend the machine’s logic and augment it with experience. These findings challenge suggestions that we imagine our human–machine experience in a dualized material/digital manner, but rather suggest an embodied, complex ontology of the self.

Proscribed Practices with Biometric Devices

Biometric devices capture and demonstrate data from modular body systems (e.g. brain waves or heart rhythm); they render invisible the flow of data to other bodily systems, suggesting that the subject resides in a disembodied non-complex system (Gardner and Wray, 2013). Biometric technologies codify modular systems as producing sensible biodata, in effect ‘demystifying’ the body by converting it visually into a set of digital moments or points on a line. In this sense, biometric devices are a hybrid version of what Thacker (2003: 58) refers to as ‘biomedia’ – the informatic re-contextualization of biological components and processes.

Biomedia rest on an assumption of equivalency between body ‘codes’ and computer ‘codes’, and thus go beyond the more familiar notion of the cyborg or ‘posthuman’, where technology plays a more prosthetic role. Rather than simply extending or improving the body, biomedia reconfigure the body as computable data, a sleight of hand that appears to make the technology ‘disappear’ (Thacker, 2003: 53–4). Terranova (2004: 31) argues that such strategies of ‘representing’ data falsely suggest a resemblance between representation and the state it describes. Information reduces complex processes into linear, mathematical, macro stories or averages that suggest a direct relationship between the macro state and complex micro flows of smaller, disorganized, and fluid dynamics. Aggregated data representations fail to reveal the varied micro flows, dynamic shifts, and abundance of reality in informational spaces such as the body.

The algorithmic codes used to visualize body data in our EEG, for example, shaped and controlled the representation of the human body by taking raw data drawn from the brain’s electrical pulses and transforming it into a series of discrete temporal events. These events were then represented in longer durations of data flow to tell coherent stories about complex operations like cognition. The machinic assumption was that frequencies simply repeat, presumably because we assume that the systems they come from are coherent and isolated. In this sense, constrained visual representations of body data are the consummate example of Deleuze’s (1992) notion of the ‘dividual’ as an endlessly subdivided, then algorithmically aggregated, identity. In Terranova’s (2004) terms, this focus on macro data obscures the information it is possible to derive from more complex micro data.

Crucially, biotechnological apparatuses and practices tangibly employ users in reducing complex, interactive data, and mind/body interaction into coherent, streamlined epistemologies, evacuating not only complexity but also the affective phenomena that reside in and alongside the mind/body system. When mainstream science discourse normalizes modularity and reductionism as sensible sorting methods (Agamben, 2004; Galloway, 2004; Rose, 2007), we necessarily agree to lose much affective complexity that resides in and alongside code; instead, users are pressed to affirm human data processed in concise, coherent modules. Because consumers simply observe data pulled and interpreted incognito by these devices, the machines reify an information-creation epistemology that absences other bodily data. This practice makes normal gross, over-determined, and imprecise interpretation practices that Dumit (2003) refers to as ‘good enough’ science, as an everyday consumer practice with these tools.

Compliant user reading of this ‘data’ is always conditioned by our cultural reliance on experts to interpret medical data. Most of us do not have the training to interpret biometric data ourselves, and we might fear we lack the ability to deeply understand its meaning. As such, we commonly rely on experts to interpret the probable meaning of biometric data collected in medical environments. Crucially, consumer-grade monitors omit the expert. Most consumer-grade biometric devices offer little explanation of how data is captured and provide instead standard medical (recognizable) graphic representations of the data and simple interpretations that are meaningful to consumers. ECG monitors might tell you whether your exercise is within a fat burning rate, while EEG monitors tell you that you are ‘relaxed’, ‘meditating’ or ‘thinking hard’. Consumer biometric devices also produce graphic outputs that coincide with two-dimensional, time-based graphs that consumers recognize from medical media they have witnessed in popular media or medical environments.

These aesthetic representations translate modular body data according to constrained logics and suggest that the bodily action being measured can be understood not through the interface, but through the computed output emitted from the interface. The devices, that is, constrain users from identifying with their bodily experience of data capture or from employing multiple senses to interpret the (aestheticized) data. In this respect, they transform the body into something that can be measured, manipulated, and improved, rather than interrogated.

Users’ experience with these devices is also conditioned by stories told to us by science media, science policy, and health recommendations, where biometric data is represented as ‘the solution’ – as always already a good, productive, and useful thing to remedy flawed bodies and behaviours. In turn, biodata capture devices are fairly uncontroversial and, as such, they are becoming increasingly popular applications to facilitate one’s personal athletic, movement, and health monitoring. Statistics suggest that 10 percent of Americans own an activity tracker (Endeavour Partners, 2014); considering these high rates of buy-in, it appears likely that users tend to embrace the devices’ normative promotions of personal data surveillance as an effective solution to health and wellness problems.

When looked at more closely, the devices are not constructed in ways that allow users to engage freely with them, but instead they tend to proscribe behaviours. Users understand them as reasonable data collection tools because they are supported by broad transhumanist framings that reduce the body to computable data and suggest that corporeal limitations are transcended by technological augmentation. In Hayles’ (2011) terms, transhumanist discourse decontextualizes the human and oversimplifies our relationship with technology.

Although our experiment provided some evidence in support of these critiques, our users were not altogether compliant with these devices. While they demonstrated a desire to understand the computational device and its macro computing work, our users readily rendered the data more complex, inputting their own experience and memory when reading their data. This move toward a more multifaceted contextualization of data visualizations verifies the need for a nuanced view of the machine–body relationship. Braidotti (2013), for example, views enhancement and extension technologies as tools that craft our contemporary posthuman present, which she sees as a now ‘permanent state’ of hybrid, nomadic, emancipated, multi-ethnic realities. This state is multilayered and contradictory, with elements both of ultra-modernity and ‘splinters of neo-primitivism’ (2013: 184). As such, it invites both nomadic liberations and acknowledgement of the constraints of computational logics of subjectivity. This complex, internally contradictory logic is, in our analysis, inscribed in the manners in which our users experienced the biometric devices – as linear logics of their bodies and, at the same time, as flawed stories requiring other context (such as memory) to ‘complexify’ the stories told by the machines’ visual outputs.

This supports Kember and Zylinska’s (2012) call for a more performative description of media/social interaction that avoids practices that separate the object (the body) from representation, and material objects from practices of becoming. In the context of our study, this means that, rather than simply focusing on how the practices of representation inherent in biometric tools affect those who use them, we must also query how the user and the machine co-constitute one another.

Barad (2006) argues that physical and physiological systems are not ‘coherent’, nor do they possess clear boundaries or follow consistent linear patterns or rules; their internal systems (e.g. atoms) are dynamic, not static. Systems are in fact intra-active and, as such, they materialize bodies (and other phenomena) – never through causal means but through chaotic entanglements of forces. Her idea of intra-action draws on Niels Bohr’s insight that the physical apparatus used to measure an object essentially determines what is observed or measured. If the apparatus ‘determines’ the object being measured, how can the object and apparatus be ontologically separate? The meaning of each ‘object’ depends on mutually constitutive, intra-active practices, in contrast to interaction, which relies on a ‘metaphysics of individualism’ that assumes prior existence of separately determinate entities (Barad, 2006: 128).

Intra-action is more suggestive of a performance that respects the role of ‘matter’ than does the term interaction. Intra-action suggests that meaning is not a property of individual bodies or technologies, but is a phenomenon, or ‘an ongoing performance of the world in its differential dance’, in which ‘part of the world becomes determinately bounded and propertied in its emergent intelligibility to another part of the world’ (Barad, 2006: 149). Material (e.g. biological, physiological) and social relations intra-inform to establish phenomena, including understandings of subjectivity, perception, and cognition. Such ‘entangled’ practices are productive, and Barad advises that entangling our own critical practices can produce improved systems.

Our experiment encouraged participants to enter into this performance by physically experimenting with and challenging the devices. By asking them to report how they felt about the visualization of their body movements and encouraging them to experiment with the impact their bodily activity was having on the visualizations emitted by the devices, we effectively gave them permission to undertake such an engagement. This allowed them to transgress the ontology of the static, coherent image and challenge its linear representation of their body data – that is, to experience the visualization of their body data as a process in which they could intervene both physically and psychically. To borrow a distinction originally made by Massumi (2002), the experience converted what was initially an experience of mirror-vision, or a digital image based on seeing oneself as others see one, to a process of movement-vision that draws on a variety of senses and sensibilities in addition to vision, including muscle memory and proprioception. In this process of movement-vision, the subject–object symmetry of mirror-vision is broken down, resulting in what Massumi called a body without an image (Featherstone, 2010: 208–9; Massumi, 2002: 59–61).

The implications of this move were twofold: first, the subject–object boundary between the user and the device was revealed as porous, making it clear that participants could enter into the data visualization process. They became knowing, interested, and embodied research subjects, in Despret’s (2008: 126) terms, ‘restating and inverting’ the relationship of control between themselves and the devices we as experimenters presented them with. The result was the realization that they could both affect and be affected by the apparatus they were experimenting with (Blackman, 2014: 371). In fact, the users actively engaged with this revelation of entanglement by attempting to alter and experiment with the visualizations on the screen.

Second, this acknowledgement of entanglement appeared to open up an intuitive understanding of participants’ relationships with the machine, which, in turn, impacted their interpretations of the data. Rather than relying on the notion that some invisible ‘expert’ must know better than them how to interpret the data, they engaged with the visualizations in ways that resonated with personal experience, sound, rhythm, and memory. In other words, they gradually became ‘affected by’ the experience, a finding that resonates with arguments from Featherstone (2010) and Hansen (2003) that the interpretation of digital media images relies on flows of affect. This idea of affect filling in the spaces of disjuncture between machine and body is also consonant with Munster’s (2006) description of a ‘baroque’ articulation between embodiment and technics. In the baroque, she argues, we find an experience based on differential relations that rely more on the logic of the fold than the classical notion of the binary. A baroque interaction can be perceived as an ‘event’ in which body and machine impinge upon one another to produce constantly unfolding and mutating outcomes (2006: 5). A ‘baroque’ tarrying or playing with the ongoing discordance and concordance of machine and human can work to embody users, enabling an acute querying of subjectivity and the body.

In our experiment, although the biometric visuals worked to gloss the digital and material, and to flatten the temporal/spatial experience of data, something about the intra-active experience created a baroque and productive space wherein users tested the constraints of data intervals. In other words, our users engaged in a messy entanglement with time- and space-constrained data, taking on and transgressing machinic readings and augmenting them with a range of other embodied and memory experiences.

The Experiment

Our user-testers demonstrated a sense that their relationship to these machines was both interactive (where bodies are separate from representation) and also intra-active, as they tarried with the meaning of the biodata as a process emanating from the complex human biological system mediated by the tool’s interface. Most participants, although initially frustrated by the read-outs of their body data, continued to physically intra-act with the machines, experimenting and attempting to alter the visualized data. During these repeated physical intra-actions with the machine and its output, many users spontaneously drew on past memories, associations with loved ones, geographical or spatial markers, and sounds to help them ‘interpret’ the data.

This ‘tarrying’ could more accurately be described as affective labour. The consistent communication between users’ bodies and the machines drew them into a more meaningful association with the process of pulling data and the alterity of the output. Users worked to coax the mathematized and constrained visualization of the data into complexity, intuitively mediating the present with memory or embodied understanding. Users did not feel in relation to the data – they felt through the data. Participants either knew intuitively or explicitly acknowledged that they, together with the machine, were co-constituting the images appearing on the screen. Users also exploited the machines’ data capture and representational practices, repeatedly attempting, over a period of about 20 minutes, to engage their bodies and/or senses to produce data that held meaning for them. While the experiment asked them to engage with the visuals, this ongoing mediating via the body and critical questioning illustrates a decided attempt to overcome the machines’ absencing of bodily experience and the restrained visuals that minimally interpret the data.

Our varying group of individuals experimented with two consumer-grade biometric tools, an EEG and an ECG. Users ranged in age from their early 20s to mid 60s, were almost equally divided between males and females, and varied considerably in terms of their comfort level with biometric and/or digital technologies. The laboratory atmosphere that these experiments took place in, although friendly, was not intimate, nor did it allow for private interventions with the technology; users were filmed, recorded, observed, and interviewed as they interacted with the devices. As well, participants were attached to the machines via USB cords that required that they remain attached to the computer terminals. As such, we contextualize the data with an understanding that users likely felt observed and felt obligated to engage with the technology to understand its visual output.

We helped users to hook up the EEG or ECG and then allowed them to observe their bodies outputting data, in visual form, in real time. The representations they viewed would be familiar to anyone who has seen such machines in medical or popular media situations: heartbeats and brain waves were portrayed as moving, angular lines in a mathematical, grid-like format. User-testers frequently referred to the awkwardness and discomfort of this contrived situation and the impermeability of the visual output from the devices, complaining, for example: ‘I don’t think I could ever learn how to interpret these things’ (user A1), or ‘I don’t know what this means, I don’t know what any of this means’ (user A7). Many users felt they lacked the ‘expert’ knowledge to interpret the data, as in user B1’s comment that there was a ‘sense of there being an unknown and … feeling excluded from a body of knowledge that I have no access to, and not being able to read something comprehensive or access the secret’. Another participant remarked on the artificiality of the EEG’s representation of her brain data, saying:

I have no idea what they [EEGs] do and I’m sure that scientists do. But the fact that we’re seeing it so linearly when we’ve been taught that you can map the brain as this sort of circuitry that has … no linearity at all and is probably more circular or cyclical – it makes it sort of hard …

User A3 talked about ‘trying not to have the researchers be a part of the input’. Several went further to argue that the data were inaccurate in the sense that the machines did not seem to respond in any systematic way to their body movements or, in the case of the EEG, to rapid changes in thoughts or emotions. In other words, there was no shortage of evidence that users perceived the technology and the context of the laboratory itself to be alien to their body experience.

Despite this alienation and scepticism relative to the experimental apparatus, users almost universally continued to engage with the technology, both physically and emotionally. When asked questions about how they read the output, they independently enacted body movements and contrived changes in their thought patterns or emotions, attempting to influence the visual read-out from the machine. They ‘played’ with the technology, trying to control the shapes or colours of the data visualized on the screen, or make either the ECG or EEG ‘flat line’. They rocked back and forth, raised their arms or legs, thought of their children, and sang songs, to provide a few examples, all the time producing a continuous commentary of their thoughts and interactions with the machine. Almost everyone mentioned a family member or pet, seeking to understand how such thoughts might impact the visual output of their biodata. Many became immersed in their experience with the technology, focusing intently on it and/or interacting with it physically in ways that appeared to block out the artificial atmosphere of the lab. This attempt to understand and play with the technology is evident in the following exchange: A2:

I know I can control it in some way, and some of it I cannot understand. I don’t know what’s happening.

Interviewer:

And if you can’t understand it? What is your impulse?

A2:

My impulse is that I will just wait. And control it. Just wait and find out.

Interviewer:

Experiment?

A2:

Yeah – I will find out.

In their attempts to ‘control’ the machines by using their bodily movements to change the representation of their body data on the machine, many participants seemed to be aware of the performative, mutual constitution of themselves and the technology in the process of mediation. As one user commented, just by seeing the data one might be affecting the next digital representation of it. This suggests that the user in such cases becomes the ‘apparatus’ (in Barad’s terms) and the technology is the phenomenon being observed, rather than the inverse relationship inferred by some critiques of digital technologies. The boundary between the subjective experience of the user and the ‘objective’ portrayal of the data becomes porous, revealing their mutual constitution. As user A5 put it, in a remark that also indicates the un-representable nature of body data:

It’s almost like looking at yourself in the mirror. If you watch it, you might affect it. The action is .… You have a feedback loop. So I’m not sure if you can actually see it in its natural state in yourself.

Many participants engaged bodily with the technology by eliciting metaphorical spatial references such as allusions to familiar landscapes, cityscapes, or buildings – a sort of cognitive mapping that referenced geographic and architectural markers rather than ‘scientific’ codes. In fact, 50 percent of participants made the comparison between the visualization of their data and a familiar landscape, while 30 percent described it as a cityscape. The visual graphics offered by the biometric devices were sufficiently vacant and uncontextualized that most users chose to describe them as a sort of architectural blueprint – perhaps as a system or scape. This evidence of discursive tarrying suggests a slow, durational meandering with the data across representations of the data. These were pleasurable, contemplative moments where they considered the meaning of the data visualized (or imaged) in relation to their experiments of watching it pulled and visualized in real time. This discursive process, we suggest, recognizes the potential for intra-activity in systems that realize our data, and the shortfalls or losses in the transmission of data from device to representation.

Users also sang songs and expressed the desire for the data to be represented acoustically as well as visually. Seventy percent of participants expressed a desire to hear the representation of their body data, as opposed to having them visualized.

In the words of user A5:

What do I want to see? Maybe, instead of a line to mark, a sound maybe?

The key role played by aesthetic and sensorial sensing in perception (as articulated by Benjamin, 1969 [1937]; Deleuze, 1995; Johnson, 2007) is accented here; we found that users took alternate routes into sound and music to interrogate both the functioning of the technology and the validity of the visual representations offered by the devices. Their acoustic experiments engaged them in a different discursive tarrying with sound as a more helpful rendition of the experience of machines’ pulling, processing, and representation of their data.

Acoustically spatialized recall may be understood as a form of ‘portable territorialization’ that Deleuze and Guattari (1987) term ‘the refrain’, the organization of a limited space through sound, landscape and other marks. This organization of data in space is a method of coping with chaos and is not necessarily territorial in a material sense. They use the example of a child trying to cope with the unknown forces of the dark:

A child in the dark, gripped with fear, comforts himself by singing under his breath. The song is like a rough sketch of a … calm and stable center in the heart of chaos. Perhaps the child skips as he sings, hastens or slows his pace. But the song itself is already a skip: it jumps from chaos to the beginnings of order in chaos and is in danger of breaking apart at any moment. (Deleuze and Guattari, 1987: 311)

Like birdsong, the child’s song is territorial in that it creates a melodic landscape or milieu that enables her to centre herself in the face of uncertainty.

In addition to singing, our users made frequent use of rhythm to engage with the technology. Often, the repetitive nature of the visualized data, combined with the rhythmic interactions of their body and the technology, drew users into what might be termed entranced states of engaged reverie. Their bodily linkage to the machine became a critical element in their tarrying with meaning making, as they worked it to forge a link between machine and image, tool and the experience of pulling data. Deleuze and Guattari argued that rhythm is another central aspect of the refrain, since rhythm joins milieus and communicates between them (Deleuze and Guattari, 1987: 313). As Blackman notes, Laing and Tarde also emphasized the use of rhythm as a form of non-conscious transmission of energy that contributes to relational dynamics between subjects (Blackman, 2012: 111).

Supplementing the Data

In all of these experiences – spatial imagining, sonic creation and even in these entranced moments of rhythmic reverie – users worked hard to insert their bodies into the process of making meaning, using sound, rhythm, and memory as methods of supplementing data visualizations. This observation is consonant with Hansen’s (2003) description of audience interactions with Digital Facial Images (DFIs) in digital media art installations. He describes audience interactions with DFIs that engage the viewer in conversation or encourage virtual physical interaction with a ‘virtual creature’, showing how such images encourage affective flows as a medium with which to interpret and understand both the digital image and ourselves as humans (2003: 212). He points to an installation by artist Kirsten Geigler where the viewer interacted with a relatively detached and unresponsive digital facial image. Although the viewer could converse with the DFI as if it could reply, the longer viewers interacted with the DFI, the more obvious its unresponsiveness appeared, generating a feeling of uneasiness. Citing Murray (2000), Hansen notes that this unpleasant experience is evidence of the ability of the digital image to stimulate the viewers’ own ‘virtualizations, fantasies, and memories in consort with the artwork’ (2003: 217). Hansen argues that, in contrast to most human–computer interfaces, DFIs generate an affective interaction by allowing us to query the gap between digital information and human sensory ratios.

In other words, the DFI used the embodied, affective response of the viewer as a sort of ‘collaboration’ or supplement to the information provided by the technology. Since there is no precise analogy between the digital information on the screen and embodied experience, ‘the bodily response to information – that is to say, affectivity – must step in to forge a supplementary one’ (Hansen, 2003: 225). Both Featherstone (2010: 211) and Hansen (2003: 208) argue that this ability of the image to trigger an embodied response indicates a transfer of affective power from the image to the body, creating what Munster (2006) refers to as a baroque space for the aesthetic processes of composition.

Our experiment revealed that this kind of embodied affective experience is not limited to DFI images. We would also qualify this argument to note that the interaction Hansen and Featherstone describe is in fact an ‘intra-action’, relying not only on a unidirectional transfer from the image or medium to the body but also, as both indicate, the flow of engaging with the embodied experience and memory of the viewer/user. What triggers this affective process may not be the nature of the digital image itself, but rather the actual process of rhythmic bodily movement and engagement with the technology, which in our study seemed to activate an almost unconscious state of reverie as participants physically engaged with the devices. Despret’s (2008) argument above regarding the impact of a more equal distribution of control between apparatus and participant is relevant here. The mutual construction of meaning Featherstone and Hansen describe is evidence of a constant process of attunement in the experimental process, where interpretations of the data emerge over time.

An interesting aspect of this attunement is what might be described as a process of ‘becoming unconscious’ analogous to Gertrude Stein’s early experiments with automatic writing. As Blackman argues, these early experiments showed the potential for more performative interpretations of psychological phenomena that occur as a process of intra-action between the experimental subject and the apparatus itself (Blackman, 2014). In our experiment, many participants lapsed into what might be termed a ‘daydreaming’ state, creating poetic musings that drew on metaphoric representations of their body data. After noting that the EEG was ‘mesmerizing’, user A5 created the following metaphor, a surprising way to describe the digital output of ‘zombie’ technology:

It’s like a wave hitting a beach in a way, just like the wave rush. That might actually be a restful way of localizing it. Wave sounds.

In a similar sort of daydreaming reverie, user B2 mused:

It feels like a tapestry of woven pieces coming together. It almost feels like it’s tactile in some way. It’s complicated and nuanced. It feels like it’s murky and it feels organic. It feels almost like a lake.… I could do this for a long time, actually. Never knowing.

This ‘user poetry’ came from a wide range of users with diverse familiarity with digital media, medical tools and with formal education. Their musings bring to mind Bachelard’s argument that daydreaming reverie can provide a ‘fleeting glimpse’ into the soul, an ‘instant’s freezing’ that distils the essence of being and memory. These poetic ‘instants’ bring a flash of déjà vu – an intuitive sense of remembering that unconsciously orients our bodies and our psyches (Bachelard, 1964). Bachelard reminds us how poetically we dwell, drawing attention to the poetic flash of déjà vu, or the intuitive sense of remembering that guides us on our daily journeys.

Although Bachelard developed his ideas in relation to the notion of the house, his work is nevertheless useful in understanding the links between poetic musing and meaning making in our experiment. Although he acknowledged that the unconscious primordial world may also be accessed through night dreams, Bachelard saw daydreams as a superior mode of communication with our localized past. He compared the poetic image to daydreaming, arguing that ‘To read poetry is essentially to daydream’ (Bachelard, 1964: 17). One would think, he notes, that a geometrical object such as a house, with its straight lines of plumbing and walls, would resist ‘metaphors that welcome the human body and soul’ (1964: 48). But, defying all rationality, the house translates into the human plane once it is recognized as a place to defend and contain intimacy, or as relevant here, subjective, embodied knowledges.

Like Bachelard’s house, digital technologies, algorithms, and binary code seem an unlikely impetus to poetic reverie. Also like a house, such technological objects are considered to be spatial in nature because they, quite literally, take up space. They appear to be finite and unmoving, unchanging over time. Their coherent, modular representations are reified as inviolable structures. Yet the unchanging nature of space comes into question once one returns to the role of memory that Bachelard insists on, despite his spatial focus. Even if the object appears to have a purely spatial and coherent presence as it sits before us, it pushes us back in time, asking us to access memories that help us to make sense of it. From this perspective, space is never fully stationary, but also encapsulates time. Participants used poetry to tarry in and out of the machinic reading of space and their bodily and memory experiences.

This process of crafting a comprehensible architectural or poetic conception of time encapsulated in space becomes more embodied when the ideas of another phenomenologist, Merleau-Ponty, are considered. Combined with the work of Bergson, Merleau-Ponty’s work helps us to understand users’ poetic notions of embodiment and becoming with the machine and their body data. In his early work on the phenomenology of perception, for example, Merleau-Ponty (1970) employed Heidegger’s concept of ‘being-in-the-world’ to describe the primordial openness of human subjects to the world through the intermediary of the body. Both challenged the Cartesian view of the world as ‘fixed and determinate’ or operating outside of human consciousness. In his later work, Merleau-Ponty urged that we view the body as a ‘total visible’ including parts that are ‘always behind, or after, or between the aspects we see of it’ (1968: 136). Our body commands the visible for us not by explaining it as a knowing subject, but by ‘concentrating the mystery of its scattered visibility’ (Merleau-Ponty, 1968: 136). We access the invisible, in-between aspects of the total visible through bodily, carnal, or fleshy experience. The invisible is not the opposite of the visible or sensible, but rather its lining and depth – the body and the vision ‘tangled up’. In Merleau-Ponty’s words:

the surface of the visible is doubled up over its whole extension within an invisible reserve; and is finally, in our flesh as in the flesh of things, the actual, empirical, ontic visible, by a sort of folding back, invagination, or padding, exhibits a visibility, a possibility that is not the shadow of the actual but its principle, that is not the proper contribution of a ‘thought’ but is its condition … (1968: 152)

One user’s poetic musings seemed to illustrate Merleau-Ponty’s suggestion that the invisible becomes visible through the body. While also referencing the intrusiveness of attempting to represent body mysteries, user B1 commented:

It feels like disclosure even though that is sort of irrational. But it feels that way. Rendering – making something that’s otherwise entirely private and unknown, visible.

Although Merleau-Ponty was critical of Bergson’s notion of duration, our users’ many references to spatial, sonic, and personal experiences indicate that the latter’s understanding of time and memory is also key to understanding this embodied process of perceiving the invisible. One way to describe this dynamic engagement is through Bergson’s understanding of perception. As noted above, the convoluted outputs of biometric devices provide a disjointed and disaggregated view of the body that falsely suggests human cognition as frozen moments of modular time-based data. Time is conceived of as a coherent moment of a constrained system that is then extrapolated to represent many moments of what is in fact a dynamic system. The full implications of this problematic are understood when contrasted with Bergson’s (1988: 186) notion of duration, where ‘we see ourselves acting’ and our ‘states melt into each other’. In keeping with this idea, the process by which we perceive things is in constant change. Bergson provides an example: even when we examine a motionless object from the same angle and in the same light, the vision of it in the present moment differs from the vision in the previous instant. The previous vision is older and we carry a memory of it into the vision of the present moment, which in turn affects the current vision, a process that continues over time. As our mental state ‘advances on the road of time, [it] is continually swelling, rolling upon itself, as a snowball on the snow’ (Bergson, 2007: 1). This snowballing state is in fact, duration: ‘the continuous progress of the past which gnaws into the future and which swells as it advances’ (2007: 3).

For Bergson, being does not merely mean passing from one separate state to another, as if time could be sliced up into discrete events or visions placed side by side. The process of perception involves ‘cutting’ or detaching discrete aspects of a potential representation that is always virtually there as part of the ‘totality of the images of the material world’ (Bergson, 1988: 30). These potential representations only emerge from this virtual totality to become ‘actual’ when we cut or detach some aspects of a thing to allow them to become a ‘picture’, supressing those aspects of a thing that do not interest us (1988: 28). Although the process of perception involves making cuts by suppressing aspects of objects that we are indifferent to, according to Bergson these cuts must be re-joined to form the image. This process occurs through memory, which ‘contracts into a single intuition many moments of duration’ (1988: 80). Interestingly, this process also draws the material object into ourselves. Whereas de jure we are perceiving matter as matter, de facto we are perceiving it within the context of our own embodied memories (1988: 80).

Like Munster’s emphasis on the baroque fold noted above, for Bergson, our visions are not a set of discontinuous, separate acts arranged side by side, but rather a ‘gentle slope’ comprising points that stand out against the continuity of everything we think and feel – our whole psychical existence. These visions, points, or discrete events exist only through the intervals linking them. Like beads on a necklace, they must be joined together by the thread of psychical life. Because we cannot see this thread we assume it does not exist, yet the psychical life unfolding beneath these ‘beads’ is the very stuff that time is made of (Bergson, 2007: 2).

We cannot understand the ‘thread’ or interval that links these discrete visions to one another through reason, but rather by a ‘fringe of vague intuition’ that surrounds intellectual representation (Bergson, 2007: 32). The intuitively perceived hesitation, or interval, between the discrete events of vision turns what may appear to be a series of points along a straight line into a curve. Using an example relevant to our analysis of the limitations of algorithimic code, Bergson presents the example of a mathematician to illustrate the impossibility of life without the interval:

When the mathematician calculates the future state of a system at the end of a time t, there is nothing to prevent him from supposing that the universe vanishes from this moment till that, and suddenly reappears. It is the t-th moment only that counts – and that will be a mere instant. What will flow on in the interval – that is to say, real time – does not count, and cannot enter into the calculation … he is always speaking of a given moment – a static moment, that is – and not the flowing time. In short, the world the mathematician deals with is a world that dies and is reborn at every instant – a world which Descartes was thinking of when he spoke of continued creation. (Bergson, 2007: 14–15, original emphasis)

Bergson’s interval helps us to realize the problematics of computational and mathematical freezings of matter into distinct knowable and calculable things. We can update his understanding of the interval, however, by emphasizing that it is an intra-active, chaotic space – a messy confluence of multiple time periods, spatial configurations, matter, and flows, including body, machine, visualization, memory, metaphor, and affect. Braidotti (2013) reminds us that nomadic thought is required to render an affective opening-out in such a space, in that the subject must be gratuitous (non-profitable), aimless (in the flow), and express transformative becoming. As Deleuze and Bergson insisted, remembering is a creative reworking. In this case, the subject observes data flowing in linear time while also experiencing data in a complex human–machine–affective network. Memory, in the posthuman nomadic mode, is ‘the active reinvention of a self that is joyfully discontinuous, as opposed to being mournfully consistent’ (Braidotti, 2013: 167). Our users delighted in the chaotic entanglement of the interval between algorithmic points, indicating that perhaps Bergson does not go far enough in pointing to the transformation from the linear to the curve involved in perception.

In this respect, perhaps his notion of the ‘zone of indetermination’ is a more appropriate way of conceiving this in-between space. Grosz (2005) interprets the experience of perception as a sort of cerebral delay in which the individual filters out or suspends some of the object’s ‘indiscriminate’ or real actions on the world, processing them in terms of the person’s virtual world of memory. This delay, or what Bergson calls a ‘zone of indetermination’ (in Grosz, 2005: 100), allows the object and the person to ‘enter unexpected connections, to make something new’ (2005: 100). For Deleuze, what fills up these zones of indetermination are ‘affections, body-memories (or habit-memory), and pure recollections (duration). Through their interventions, perception becomes “enlivened,” and capable of being linked to nascent actions’ (Grosz, 2005: 100). Viewed from this perspective, Grosz concludes, ‘The thing is what we make of the world rather than simply what we find in the world, the way we are able to manage and regulate it according to our needs and purposes’ (2005: 133).

Ostensibly, this cumulative notion of time differs markedly from the progression of the body over time represented by visualized biometric data. The isolated modular and coherently coded systems represented by biometric devices appear to freeze data in static spaces and linear moments, to imagine and render them out of intra-action. System-isolated data obfuscates what in lived experience is a dynamic mingling of material data and our lively experience and perception of it as embodied subjects. Yet users did not ‘fill in’ the interval between data points to reconstitute the data, but rather played out the tensions and contradictions between the machine-produced data and their varied experiences of it. The subject, as Braidotti (2013: 164–7) argues, becomes relational in a complex manner, connected to multiple others including, in Terranova’s terms, other actors in the network – the machine, data output, and subjective experience. Braidotti insists that we must familiarize ourselves with non-human actors to do the necessary (posthuman) work that will move us beyond our habits of anthropomorphism. Here, that means recognizing the work of the machine in so far as it is an actor in this complex network wherein we make meaning of our relations to machinic readings of our bodies, and wherein affect is produced.

Our experiment suggests that the process that our user-testers experienced as they bodily intra-acted with the ECG and EEG over time played out as a sort of moving metaphor for the process of subjective understanding and perception. Although the machine made ‘cuts’ to the virtual matter of the user-tester’s body that were revealed in the visual representations on the screen, this ‘virtual’ representation of the data became meaningfully reabsorbed through mashings of temporally different data and complex affective flows generated by the user-tester’s sensorial, consciously embodied intra-actions with the machines. Participants drew on metaphor and memory in a process of attunement that allowed them to ascribe meaning to the images appearing on the screen. In other words, the ‘cuts’ inherent to the biometric representations of data were linked by an intra-active process whereby participants were able to provide varied, complex readings of the data. As Hansen (2003: 217) puts it, there are two distinct virtualities at play here: the virtual representations of the digital technosphere, and Bergson’s sense of the virtual nature of perception itself.

Conclusion

Our experiment lends credence to the arguments of those who argue for the intra-active relationship between human bodies and digital technologies. There was a constant process of attunement (Despret, 2008) between our user-testers and the apparatus that made it possible for them to make personal meaning through their bodily engagements, even with biometric devices that many initially described as alienating and proscriptive. Through the use of spatial or auditory associations, or poetic lapses into a sort of daydreaming déjà-vu, our users related to the technology and made sense of it in their own terms, with reference to their own experiences, with their own metaphorical poetics. This process of ‘becoming unconscious’ is comparable to Gertrude Stein’s experiments in automatic writing, which similarly involved an entanglement between human (un)consciousness, matter, and the technical aspects of the experimental apparatus (Blackman, 2014: 371).

This is not to say that the affordances of these technologies do not influence users’ perceptions of their bodies, or that they do not distort the representation of bodies on a flattened screen. On the contrary, this seems to be the proscriptive intention and result of linear, causal, reductive epistemologies embedded in the biometric machines and algorithms. Instead, we are noting that when presented with artificial representations and the opportunity to ponder the codified apparatus and its computational, highly constrained outputs, individuals tend to consider the intervals in varied, embodied ways, translating the data representations into more complex narratives inspired by their embodied experience, and the ‘virtual’ pasts of their own lives. Our findings confirm Hansen’s (2003) and Featherstone’s (2010) arguments pointing to the importance of affective flows in the interpretation of digital representations. Crucially, slowed experience and musings invited users into these digital spaces via creative practices that allowed for these creative reworkings and affective experiences. Drawing on a range of experiences with media, time and space, from the past and in process, our participants made multiple meanings in the context of what was a highly alienating experience: the representation of their sensed bodily output in collapsed, fragmented, disembodied format. This process points to the importance of considering perception and interpretation of our human–machine operations as practices that always already include a complex intra-action of past and present memories, of time into space, of bodies with minds. It demonstrates that, despite the ordained epistemological structures and pressures of biometric machines, we humans will, if given time to pause, create complex narratives of human–machine interaction in the hesitations and absences inherent in digital code and digital process.

Footnotes

Note

References

Agamben

(2004) No to bio-political tattooing. Le Monde diplomatique 10 January.

Bachelard

(1964) The Poetics of Space. Boston, MA: Beacon Press.

Barad

(2006) Meeting the Universe Halfway: Quantum Physics and the Entanglement of Matter and Meaning. Durham, NC: Duke University Press.

Benjamin

(1969 [1937]) Art in the Age of Mechanical Reproduction. In: Illuminations, edited by Arendt

trans. Zohn

. New York: Schocken Books.

Bennett

(2010) Vibrant Matter: A Political Ecology of Things. Durham, NC: Duke University Press.

Bergson

(1988) Matter and Memory, trans. Paul

Palmer

. New York: Zone Books.

Bergson

(2007) Creative Evolution. New York: Palgrave Macmillan.

Blackman

(2012) Immaterial Bodies: Affect, Embodiment, Mediation. London: Sage.

Blackman

(2014) Affect and automaticity: Towards an analytics of experimentation. Subjectivity 7(4): 362–384.

10.

Braidotti

(2013) Posthumanism: Life Beyond Theory. Cambridge: Polity Press.

11.

Cheney-Lippold

(2011) A new algorithmic identity: Soft biopolitics and the modulation of control. Theory, Culture & Society 28(6): 164–181.

12.

Deleuze

(1992) Postscript on the Societies of Control. October 59(Winter): 3–7.

13.

Deleuze

(1995) Difference and Repetition, trans. Patton

. New York: Columbia University Press.

14.

Deleuze

Guattari

(1987) A Thousand Plateaus: Capitalism and Schizophrenia. Minneapolis, MN: University of Minnesota Press.

15.

Despret

(2008) The becomings of subjectivity in animal worlds. Subjectivity 23: 123–139.

16.

Dumit

(2003) Picturing Personhood: Brain Scans and Biomedical Identity. Princeton, NJ: Princeton University Press.

17.

Endeavour Partners (2014) Inside Wearables: How the Science of Human Behavior Change Offers the Secret to Long-Term Engagement. White Paper. Available at: http://endeavourpartners.net/assets/Endeavour-Partners-Wearables-White-Paper-20141.pdf (accessed August 2015).

18.

Featherstone

(2010) Body, image and affect in consumer culture. Body & Society 16(1): 193–221.

19.

Galloway

(2004) Protocol: How Control Exists after Decentralization. Cambridge, MA: MIT Press.

20.

Gardner

Wray

(2013) From lab to living room: Transhumanist imaginaries of consumer brain wave monitors. Ada: A Journal of Gender, New Media, and Technology 3. Available at: http://adanewmedia.org/2013/11/issue3-gardnerwray/ (accessed August 2015).

21.

Grosz

(2005) Time Travels: Feminism, Nature, Power. Durham, NC: Duke University Press.

22.

Hansen

(2003) Affect as medium, or the ‘digital-facial-image’. Journal of Visual Culture 2(2): 205–228.

23.

Hansen

(2004) New Philosophy for New Media. Cambridge, MA: Harvard University Press.

24.

Hayles

(2011) Wrestling with transhumanism. In: Hansell

Grassie

(eds) Transhumanism and Its Critics. Philadelphia, PA: Metanexus Institute, 215–226.

25.

Ihde

(1990) Technology and the Lifeworld. Bloomington, IN: Indiana University Press.

26.

Johnson

(2007) The Meaning of the Body: Aesthetics of Human Understanding. Chicago, IL: University of Chicago Press.

27.

Kember

Zylinska

(2012) Life after New Media. Cambridge, MA: MIT Press.

28.

Latour

(2004) How to talk about the body? The normative dimension of science studies. Body & Society 10(2–3): 205–229.

29.

Massumi

(2002) Parables for the Virtual: Movement, Affect, Sensation. Durham, NC: Duke University Press.

30.

Merleau-Ponty

(1968) The Visible and the Invisible. Evanston, IL: Northwestern University Press.

31.

Merleau-Ponty

(1970) Phenomenology of Perception. New York: Routledge and Kegan Paul.

32.

Munster

(2006) Materializing New Media: Embodiment in Information Aesthetics. Hanover, NH: Dartmouth College Press.

33.

Murray

(2000) Digital incompossibility: Cruising the aesthetic haze of the new media. CTHEORY. Available at: http://www.ctheory.net/articles.aspx?id=121 (accessed August 2014).

34.

Rose

(2007) The Politics of Life Itself: Biomedicine, Power and Subjectivity in the Twenty-first Century. Princeton, NJ: Princeton University Press.

35.

Stafford

(2009) Thoughts not our own: Whatever happened to selective attention? Theory, Culture & Society 26(2–3): 275–293.

36.

Suchman

(2007) Human–Machine Configurations: Plans, Situated Actions. Cambridge: Cambridge University Press.

37.

Terranova

(2004) Network Culture: Politics for the Information Age. London: Pluto Press.

38.

Thacker

(2003) What is biomedia? Configurations 11(1): 47–79.

39.

van Dijck

(2007) Mediated Memories in the Digital Age. Stanford, CA: Stanford University Press.

40.

van Doorn

(2011) Digital spaces, material traces: How matter comes to matter in online performances of gender, sexuality and embodiment. Media, Culture & Society 33(4): 531–546.

41.

Verbeek

P-P

(2011) Moralizing Technology: Understanding and Designing the Morality of Things. Chicago, IL: University of Chicago Press.