Abstract
Digital interfaces, in the form of websites, mobile apps and other platforms, now mediate user experiences with a variety of economic, cultural and political services and products. To study these digital mediations, researchers have to date followed a range of methodological strategies including the modification of pre-existing qualitative research methods, such as content analysis, discourse analysis and semiotics, among many others, and an experimentation with new methods designed to make visible the operation of data aggregation, analytics and algorithms that are hidden from users. Building upon, while distinct from these strategies, the article sets out a post-phenomenological approach to studying interfaces, websites and apps that explicitly interrogates how they appear as objects. In doing so, the article provides a response to a problem that animates contemporary cultural geography: that new cultural objects are emerging which place in question the habits and practices of analysis that composed the ‘new’ cultural geography. To do this, the paper develops the concepts of unit, vibration and tone to unpack interfaces as sets of entities that work together to shape the experiences and responses of users. As such, the article provides a methodological vocabulary for the analysis of how interfaces operate to modulate user response and action on a series of habitual and un-reflected upon levels and thereby to create outcomes that suit their owners and operators.
Over the past 5 years, cultural geographers and others have highlighted how contemporary life is now mediated through digital interfaces, websites and apps. Across multiple domains of life – including software interfaces used in worldwide financial trading, digital economy platforms, threat detection and anticipation interfaces used by government and emergency services, social media, gaming, place visualisation and locative media – digital technologies are central to the composition of a variety of activities. 1 In response, cultural geographers and others have explored how to research and analyse new digitally enabled and mediated systems of economic, cultural and political communication and practice. Approaches to these so-called ‘digital methods’ 2 have taken a number of forms. For instance, as Rose argues, 3 cultural geographers, sociologists and other social scientists have modified and applied existing types of quantitative and qualitative analysis to digital platforms. This has taken many guises, including the development of ethnographic approaches into netographies, discourse analysis into digital discourse analysis, quantitative content analysis into interactive visualisation and so on. 4
Alongside these modifications, Rose has also pointed to the rise of new forms of digital method that work to take into account the particularity and idiosyncrasies of digital systems, their inherently responsive nature and the rise of big data algorithms that create and order content. 5 As Rose notes, ‘if digital technologies and social and cultural identities and relations are co-produced, it is necessary to look not only at what people do with technologies, but also at what the technologies themselves are doing’. 6 To achieve this, academics have developed a series of methodological tools. For example, Marres and Gerlitz develop the notion of an ‘associational profile’ in order to trace how climate change is discussed on Twitter, 7 and Gerlitz and Helmond use purpose-built tracker tools to understand how the Facebook Like button serves to connect users on multiple websites. 8 In doing so, these kinds of analyses often attempt to reveal that which is hidden from users or, in the words of Deville and Velden, the problem the methods respond to is ‘how to make that, which is rendered invisible, visible’. 9 What is considered to be invisible and require exposure are the logics and techniques used by companies like Facebook or Twitter to aggregate data, the kinds of data they specifically aggregate and analyse, and how this data is passed onto and shared with governmental agencies and the private sector. 10
While diverse, what is clear from examining these forms of digital method is that, as Bates et al. argue, different theorisations of digitality result in the invention and adoption of different methods. 11 Vice versa, these methods also work to produce different understandings of digitality. For instance, from a digital discourse analysis perspective, websites and social media platforms are primarily sites for the production and circulation of meaning. In turn, digitality becomes theorised primarily as a means of discursive communication. Or, in the work of Marres and Gerlitz, in studying Twitter as a set of associations, digitality comes to be theorised as a network of associations.
Despite the differences in the focus of these approaches, whether they theorise and study digital platforms as sites of discursive production, algorithmic governance or surveillance, these processes are often placed in contrast to the surface appearance of interfaces, which are mainly considered to be significant only as a means of obscuring and glossing over these regimes of discursive production, aggregation, analysis and so on. 12 To be clear, by surface appearance of the interface, we are not referring to interfaces as screened images alone. Rather we refer to appearance in a more general sense as that which makes itself manifest in a situation. Interfaces appear as and through physical buttons, sound effects, icons, voice activation and haptic vibrations as well as icons or images on a screen.
Rather than focusing exclusively on how interfaces naturalise the production of social and political discourse, or treating the surface of interfaces as no more than the outcome of the ‘hidden generative mechanisms’ 13 of code and software, we argue that it is through interfaces that digitally mediated and organised life is constituted. For it is through interfaces that people encounter digital content. And interfaces are actively designed to modulate user action with the aim, hope and promise of producing desirable outcomes for those that own and operate interfaces. Most of these design elements do not operate exclusively on a linguistic-representational register but involve other modulations and modifications, such as the placement of buttons and menus, the layout of checkout pages, the colour of backgrounds and the design of sound effects and haptic feedback. Through organising these elements, designers seek to direct how users respond to, interact with and experience these services on a series of affective and emotional registers. 14 For instance, much of the interface-orientated design literature discusses techniques and mechanisms of design that create hooks, which enable ‘behaviours done with little or no conscious thought’ 15 in order to create ‘loyal’ and ‘high-value’ customers. These techniques are not based upon the assumption that users can be simply manipulated into doing what the designers and owners of these interfaces want with complete certainty. Instead, designers work to modulate the action of users. Where manipulation assumes that an interface designer can exert direct control over a user’s actions, modulation recognises that a designer can only set the limits of interaction in order to give the user some degree of choice within a set of prescribed limits in an attempt to increase the chance that they will take the course of action the interface or interface designer intends. 16
A number of researchers are beginning to argue that responding to these design practices requires the development of new methods. For instance, Light et al. utilise a walk-through method to understand how smartphone apps are experienced in practice in order to ‘reveal intricate details about the artefact in question’ and seek to ‘make explicit the otherwise implicit and (by design) apparently seamless process of engag[ement]’. 17 In a similar manner, Jewitt proposes a multi-modal account for thinking through how interfaces are used by developing a ‘fine-grained analysis of artefacts and interactions . . . with a view to the underlying choices available to communicators, the meaning potentials of resources and the purposes for which they are chosen’. 18
Building on this work, the article offers a conceptual vocabulary and a methodological procedure in order to understand how interfaces modulate user action. At the heart of our vocabulary is a treatment of interfaces as compositions of objects, which we term units, whose appearances can never be simply reduced to their visual design or form on a screen. From this perspective, interfaces are not single self-enclosed objects but relational systems composed of multiple parts, each of which communicate with one another and the user to create a range of affective, habitual and often un-reflected upon responses. As we explain across the following sections, each unit of an interface vibrates at a different amplitude, frequency and rhythmic articulation and resonates to emit a tone. Developing a language of unit, vibration and tone offers a method for analysing how interfaces attempt to modulate users’ experiences, allowing us to focus on how interfaces shape, guide or otherwise affect action.
This method develops the insights of, while also being distinct from, pre-existing digital, visual and interface methods in a number of ways. First, the vocabulary developed here allows us to talk about different types of content, such as images, sounds and rumbles from vibration motors etc., using a shared language that is common to all parts of the interface. This is beneficial as it allows us to closely trace where and how modulation occurs, without the difficulty of incorporating separate methodological approaches for sound, images and so on into one hybrid method.
The implications of this first distinction lead to the second key difference; a post-phenomenological approach allows us to think about the way interfaces are structured to modulate action, without reducing this modulation to an effect of how the user perceives or attaches meaning to that interface. While theoretical approaches to digital interfaces regularly point to the way they modulate human action across a series of affective and emotional registers, interface methods tend to return to a model of human meaning making to explain the effect of this modulation. 19 For instance, as Jewitt puts it, digital multi-modal methods focus on ‘the different ways in which people make meaning and how those meanings are interrelated’, 20 and Light et al.’s walk-through method suggests that digital interfaces primarily work to ‘transform meaning through the interaction they invoke’. 21 A post-phenomenological method is at least partially distinct from these positions in that it offers a way of empirically examining and explaining how modulation occurs without assuming a model of human meaning making is necessary to cover the explanatory gap between action and response. As we shall see, a post-phenomenological approach allows us to examine how action is linked to sensation rather than meaning alone and demonstrates how sense is distributed among a variety of units in the interface, rather than located in a discrete human subject.
The article pushes forward debates in digital and cultural geographies, then, by asking geographers not to reify or abstract particular aspects of digital devices, by focusing, for instance, on algorithms, code or data as the most important or primary site of analysis. For instance, rather than constructing ‘data journeys’ to understand ‘the movement of data between different sites’ ‘through Euclidian space’, 22 the article focuses on what appears in a given interface at a given time. In other words, a post-phenomenological approach asks that each part of an interface, including algorithms and data, be considered as units that work and are experienced in varying combinations, rather than standing alone. From this position, the individual importance of a piece of data, an algorithm or an image can only be attributed and understood in relation to the operation of a particular interface.
At the same time, the article supplements existing ways of researching digital life by providing a method for empirically investigating digital devices using a distributed model of human subjectivity, in which the body or subject is not the primary source or location of experience. While digital methods are productive in creating new theories of digitality, they have been less capable of thinking through and responding to post-humanist and post-phenomenological accounts of human experience in relation to digital devices. Developing a post-phenomenological vocabulary for studying digital interfaces is, therefore, one attempt to close the gap between theory and method in relation to both the conceptualisation of digital interfaces and phenomenological geography more broadly.
To examine the appearance of various parts of interfaces in a way that allows us to analyse how they modulate action, the article focuses on three examples: the iTunes digital store (Apple’s digital media storefront interface), the Wonga.com website (a high-cost short-term credit website) and the Facebook iOS App (a social media app developed for Apple iDevices). These examples are chosen to demonstrate how the methodological vocabulary we develop can be applied across a range of different interfaces. As such, the examples are discussed primarily to illustrate and demonstrate the methodology at work, rather than provide an in-depth analysis of the interfaces themselves.
Developing these arguments, the rest of the article proceeds in five parts. The section ‘Post-phenomenology, sound and digital methods’ identifies what is distinctive about a post-phenomenological approach and how developing a methodological vocabulary based on sound provides a way to account for how interfaces modulate user action across a variety of registers. The section ‘Unit’ uses the work of Manovich to understand interfaces as composed of distinct entities, which we term units. The section ‘Vibration’ explores how units relate to one another and users, arguing that units are composed of and emit vibrations of different amplitudes, frequencies and rhythmic articulations and these vibrations can form resonances with existing vibrations from outside of a particular interface. The section ‘Tone’ suggests that, depending on the nature of a unit’s vibrations, different tones are formed that attempt to prime and shape user action. By way of conclusion, the section ‘Conclusion: modulating action’ translates the vocabulary of unit, vibration and tone into a procedure for researching how digital interfaces appear and act.
Post-phenomenology, sound and digital methods
Post-phenomenology offers a distributed account of human experience by expanding what we mean by the human and re-evaluating the role non-human objects play in the construction of experience. 23 In doing so, post-phenomenological approaches understand that objects both proceed and exceed human experience of them while also providing the grounds and means for human thought and cognition. From a post-phenomenological position, objects, digital or otherwise cannot be reduced to the ways they are used or perceived by humans. Instead, objects are considered to share more ontological features with humans than previously thought. Indeed, Shaviro goes as far as suggesting that, like humans, objects partially ‘perceive’ aspects of other objects, rather than encountering them as complete or total lumps of matter. 24 From this position, much like a human perceiver only sees one aspect of an object, such as a house depending on what side of the house they are standing, objects also only partially encounter other things. For instance, when an apple sits on a desk it encounters the hardness of the desk’s surface and its density but does not encounter its colour. 25 With this notion of sensual objects in mind, writers such as Shaviro suggest a language of brute causality or utility to be insufficient to understand how objects communicate and encounter one another and humans. The term communication therefore refers to how objects affect one another and humans in a general sense. For instance, in relation to digital life specifically, communication might involve the way code runs to execute an action on a user’s computer, the way a colour on screen affects a user’s body or how electrical signals from a computer’s sound card become vibrations in the cone of a speaker or headphones connected to the computer. 26
Beginning with this insight, a post-phenomenological approach to interfaces focuses on understanding an iOS app like Facebook not as a single thing made up of multiple parts, such as a map, a comment box and a series of sound effects, but as a set of multiple objects, where each object such as the map or comment box or sound effect exceed their relationship to the interface as a whole. From this position, interfaces are sets of objects that communicate partially with one another in order to produce the effect of being a single or coherent thing for the user. But, in order to understand how interfaces shape user response across multiple registers, we need to understand how the various objects that appear as a single thing work together to create and sustain a singular effect.
To describe and account for how objects appear in interfaces and affect the body on a variety of habitual, un-reflected upon and non-discursive registers, we draw upon vocabularies emerging from sound and sonic geographies. 27 While the majority of accounts of interfaces emphasise their visual character, developing a sonic account of modulation to analyse interfaces is beneficial for three reasons. First, understanding interfaces through the language of sound rather than vision is useful in order to avoid linking particular parts of interfaces to particular or individual human senses. Just as sound can be simultaneously heard and felt and these sounds and feelings depend on the environment the sound travels through, 28 so are parts of interfaces experienced on multiple sensory levels depending on what they are placed in relation to. In other words, a methodological vocabulary based on sound allows us to understand the synaesthetic effect specific parts of interfaces can have on the human user of an interface, in ways that are difficult to articulate using a language of the visual. The objects that make up an iOS app like Facebook, such as emoji’s or news feed item boxes, are not simply visual representations seen by the eye, nor are slider units on interfaces like Wonga.com simply sets of buttons felt by the hand. Rather each part, which in the next section we will call a unit, is simultaneously felt, seen and heard and these senses regularly become mixed together and linked to various sensory memories. For instance, as we will demonstrate in the ‘Vibration’ section, interface designers regularly use colours to create affective associations between their service and other products to increase feelings of confidence or security. Crucially, developing a methodological vocabulary that can attend to the mixed nature of the senses is key because interface designers and developers themselves draw upon multiple aspects of the user’s senses to attempt to modulate action through the interfaces they create.
Second, a methodological vocabulary based upon the sonic helps us to trace connections between technical aspects of interface design and how the features of design modulate user action using a language that is common to both the technical and human parts of the process. The language of unit, vibration and tone developed in the following sections allow us to close the gap of explanation between how interfaces communicate to humans and modulate human practices in a non-deterministic way. This is because language drawn from sound enables us to explain how technical forces can be experienced on various human sensory registers and thus have human effects, while recognising that every force is translated differently depending on how it is registered. As Gallagher suggests, the sonic ‘need not necessarily be perceived, felt or meaningful. On account of its lively motion . . . [the sonic] . . . does tend to activate other registers as it encounters bodies, sparking nervous and motor systems, accruing or entraining additional layers of sense and signification’.
29
Gallagher gives the example of a refrigerator to explain how the vibrational basis of sound can come to signify different things depending on what encounters that sound and how it is encountered:
The hum of my domestic refrigerator, for instance, presumably drones away while I am sleeping or out at work, discharging energy into the surrounding environment in a way which does not ‘mean’ anything. When encountered by human bodies, however, the fridge hum may become soothing, reassuringly familiar, or it may provoke annoyance, perhaps feeding into understandings of power consumption, carbon foot-prints and climate change.
30
As we shall see in the ‘Vibration’ section, using a methodological vocabulary drawn from sound allows us trace how the shape or colour of a button on an interface menu generates a particular vibration, while not assuming that the vibration itself determines, or is determined by, a signifying system that is contained within the human or precedes the effect it attempts to generate. In other words, a methodological vocabulary drawn from sound provides a way of moving beyond a language of simple manipulation. As any vibration of a unit is necessarily a translation, we cannot produce a straightforward account of linear cause and effect between an interface and user. Instead, we have to account for the specific ways particular parts of an interface are translated into vibrations and how these vibrations are experienced. After this move, we can then analyse how an interface attempts to modulate users’ actions through the ways its vibrations are organised and work across multiple bodily registers.
Third, a methodological vocabulary of sound offers one way of avoiding a false distinction between theory and practice or concept and method that can stall innovative analyses of digital objects. As Deleuze and Guattari suggest, concepts are procedures that help us attend to problems, rather than abstract theoretical axioms. In their words, concepts ‘must not be confused with general or abstract ideas’ and ‘can only be assessed as a function of their problem and their plane’.
31
Concepts are thus kinds of tiny autonomous machines of thought that can be employed to examine concrete problems rather than provide a general account of reality as such. As they put it,
concepts are centers of vibration, each in itself and every one in relation to all the others. This is why they all resonate rather than cohere or correspond with each other . . . As fragmentary totalities, concepts are not even the pieces of a puzzle, for their irregular contours do not correspond to each other. They do form a wall, but it is a dry-stone wall, and everything holds together only along diverging lines.
32
The concepts of unit, vibration and tone that are developed in the sections ‘Unit’, ‘Vibration’ and ‘Tone’ are methodological precisely because they are custom-built tools produced in thought that offer a means of researching and understanding how digital interfaces attempt to modulate the action of users. Following this point, the terms outlined in the next sections, such as vibration and tone, should be understood as figurative concepts, in the sense that they are designed to provide a mode of analysis that can cross between a whole range of different interfaces, rather than being literal descriptions of these interfaces. We now turn to discuss the concepts of unit, vibration and tone that make up a post-phenomenological approach to studying interfaces.
Unit
How might we conceptualise the relationships between the different types of content – such as maps, graphs, images, texts and buttons– that compose interfaces? Furthermore, how might we develop a shared methodological vocabulary to understand how different kinds of interfaces modulate user action? Rather than thinking about interfaces as composed of content that can be organised into categorical types such as images or text, we suggest the first step is to consider all parts of an interface as equally modular units. Modularity refers to the idea that each part of an interface is more or less detachable from other parts. Manovich summarises this idea nicely:
a new media object has the same modular structure throughout. Media elements, be it images, sounds, shapes, or behaviors are represented as collections of discrete samples (pixels, polygons, voxels, characters, scripts). These elements are assembled into larger-scale objects but they continue to maintain their separate identity. The objects themselves can be combined into even larger objects – again, without losing their independence . . . An example of modularity is the structure of a HTML document: with the exemption of text, it consists of a number of separate objects – GIF and JPEG images, media clips, VRML scenes, Shockwave and Flash movies – which are all stored independently locally and/or on a network.
33
As Manovich suggests, an interface is not one thing, but many different units that are assembled together to create an overall function or effect. From his perspective, an interface is not a set of representations, texts or discourses (even though it may contain these), rather it is set of small machines, each of which is independent from, but also linked to, one another.
Putting the idea of modularity into methodological practice, how might we identify the particular units of an interface? We begin by asking what appears as a distinct entity for the user of the interface. In visual terms, interfaces are often designed as a series of boxes that are separated using lines, shading, contrasted colouring and other forms of differentiation. 34 This makes different pieces of content clearly identifiable for the user. In turn, this principle and practice of interface design provides a simple way to identify and differentiate between units. For example, upon opening the Facebook iPhone app, a number of units can be identified. At the top of the screen, there is a search box, which is a dark blue rectangular unit with a magnifying glass icon and the word ‘Search’ displayed in a lighter blue colour within it. Below this there is a ‘What’s on your mind?’ unit, which is a white rectangular box with the user’s profile picture on the left and ‘What’s on your mind?’ text in light grey to the right. There are also a number of other boxes and features, including a news feed unit, which contains a series of news items in individual white boxes and the profile picture and name of those who posted these stories. At the bottom of the screen, there are also a series of individual icons that lead to other parts of the app and could also be classified as units, such as a friend request page and a marketplace page.
Of course, a unit is not just what visually appears on a screen. A unit might also be a particular sound effect that forms part of an interface or a mechanical button that activates a particular function of an interface. In this case, a unit would be identified through its particular mode of appearance. For instance, the Apple iOS personal voice assistant Siri could be identified as a unit, which is both distinct from and linked to other units that make up the iPhone interface (such as the home button, which a user must hold down to activate the Siri unit, or the other apps that Siri can access, such as the weather or calendar app units). 35 The Siri unit would be identified then, through the way it speaks to the user, when it responds, how it responds, its tone of voice, cadence of speech and so on. Or, in relation to haptic feedback on a smartphone, the unit would be the particular intensity and length at which a haptic motor rumbled in the smartphone, in relation to a specific function of an app.
Furthermore, it is important to recognise that what is classified as a unit depends on what aspect of an interface you are studying and how you are studying it. For example, you may be interested in how people choose their profile image as a form of self-representation and the ways that the profile image is displayed in different ways in different places on Facebook. 36 Here, the profile picture itself would be a unit that would not be reducible to any of the other units of which it forms a part. Or you may be interested in the role that sharing news stories on Facebook can play in altering and reinforcing peoples’ political attachments and identifications. 37 Here, the profile picture would not be considered as a distinct unit but instead analysed as a key part of the broader news feed item unit because the researcher would be interested in the words and stories shared in the news feed and how they are displayed alongside the profile image.
In both cases, the way of resolving where one unit ends and another begins is to recognise that the concept of modularity allows us to understand that units are essentially bottomless. This is because any unit could always disclose some new or previously undisclosed quality or property depending on what it encounters and how it encounters it. For example, when placed alongside other units on a user’s Facebook page, like their biographical information, the Facebook profile picture discloses the users’ identity by linking their name and what they look like. But if the same photo is added to Microsoft’s How-Old.net website, the site’s machine learning algorithm will disclose an estimated age of the user based on a set of qualities that can only be activated when the photo encounters that site in particular. Unlike a visual method such as compositional analysis, a post-phenomenological analysis is not about identifying the sum total of units that make up an interface. For instance, a compositional analysis is based on the assumption that an image such as a painting or photo is made up of a series of elements that are more or less fixed and can be accounted for. Knowing that these elements are stable then enables a researcher to examine the spatial organisation of the image, its focal point, the colours used and so on, in order to analyse it. But in a digital interface, different units are always encountering one another in a different way, depending on the hardware and software on which they are being accessed, which alters what appears. In the case of Facebook, the profile picture units in the Facebook app will appear quite differently depending on whether the user is accessing the app on an iPhone 6 compared to an iPad, for example. Understanding units as autonomous, but also linked to one another, is important because it allows us to account for how people engage with and respond to the same unit differently depending on how they are accessing it.
In other words, there is no outside perspective from which all of the ways units can encounter one another can be accounted for. While this might appear limiting, such recognition actually aids analysis by encouraging researchers to decide which units are important in relation to the specific research question or phenomenon they are investigating. Of course, the implications of a post-phenomenological position are that the researcher will have to accept that there will be units that are not accounted for from the perspective of the unit they are studying and thus any interface analysis is necessarily partial. For example, in the Facebook profile photo example discussed above, the photo does not encounter the content of the user’s status update and vice versa, but the user does encounter both together as part of one unit. Identifying units and how they relate to one another is important but is only the first step in a post-phenomenological approach.
Vibration
The second step of a post-phenomenological approach is then to identify how each unit that makes up the interface communicates with other units and users. For instance, when loading up a simple website such as Wonga.com, each of the sites’ units, such as photos and headlines, have to communicate with the Cascading Style Sheet code that determines where these units are placed on the page, which in turn shapes what the user sees on screen. To theorise how different units communicate with each other and the user on a series of non-discursive and habitual levels, we suggest developing the concept of vibration. According to McLaren, ‘vibrate can be traced to the Latin word vibratus, which means to “move quickly to and fro”, or shake, which itself has roots in words referring to the wagging of a dog’s tail, swinging or wiping’. 38 Using a sonic term like vibration allows us to understand how particular parts of an interface communicate with the user to prime and shape various responses and actions on a series of material and habitual levels. To make sense of how units vibrate and communicate through their vibrations, we can develop our own methodological vocabulary specific to interfaces. We can identify four aspects of units’ vibration: their amplitude, frequency, rhythmic articulation and resonance.
Amplitude
The term amplitude can help us understand how different units that make up an interface encourage or discourage interaction without explicit linguistic-representational signposting, such as words or labels. In relation to interfaces, amplitude can be defined as the strength of emission of vibrations of a unit. Units can emit stronger amplitude vibrations or weaker amplitude vibrations. A unit could be said to have a weaker amplitude vibration when it is static or non-interactive. Conversely, a unit could be said to have a stronger amplitude vibration when it is interactive and it is clear what the unit is for and how it operates.
An example of a weaker amplitude unit would be the progress indicator on the application pages of Wonga.com. When a user clicks the apply button on Wonga.com, they are taken through to a series of application pages. These pages ask for a range of personal and financial information that are used to help Wonga decide whether to accept a user’s application or not. Progress through these pages is represented by a blue line at the top of the screen with five equally spaced circles along the line. As the user inputs information and moves through the pages, the line and circles associated with each page turn blue, until the application is complete. The progress indicator vibrates with weak amplitude because it is embedded on the application page as a static image, meaning that the user cannot use the indicator to click forward and back through the application process after they have completed each section. If a user could move backwards and forwards using the progress indicator bar, then this would be an example of a high-amplitude unit. The progress indicator bar is presumably designed to emit a weak amplitude in order to give users a sense of control over the application process by letting them know how close they are to completion while simultaneously trying to stop them from returning to previous parts of the form. Keeping the user moving through the process is important in order to minimise any delay that might give rise to doubt or reflection, which might result in the user leaving the application process and not applying for a loan. As such, the weak amplitude of the progress indicator on Wonga.com is designed to minimise thought or reflection on the part of the user while reinforcing a sense of control over the process in order to increase the rates at which users complete the loan application process.
Crucially, the amplitude of a unit does not simply emerge when someone attempts to click on a link. Amplitude can be communicated in a variety of ways, depending on how units are placed alongside other units. One way stronger amplitude might be indicated is through the use of shading or gradients on text or boxes or animation to indicate that the box provides some interaction. For example, within the timeline news unit on the Facebook iOS app, users can react to posts using emoji’s (graphical icons, usually of faces, that denote mood or feeling). When the user taps the reaction button, the emojis are animated and express different emotions through various forms of movement, such as raising their eyebrows or opening their mouths. This would be an example of a unit expressing a stronger amplitude because the unit offers visual feedback to the user that encourages them to engage with the unit. If the emojis were static, as on the iPhone messaging app’s emoji selection unit, then this would be an example of a unit with a weaker amplitude because it would not encourage engagement. Amplitude is thus a key form of vibration that can be used to modulate user engagement with an interface or app. The strong amplitude of the Facebook reaction unit’s vibrations is designed to encourage people to react to posts and in turn generate more traffic to the service, which Facebook relies upon to generate profit through advertising. As such, the amplitude of a unit’s vibrations can be communicated through a variety of design elements including colour, shading, sound and images and through a number of sensory channels, including sight, hearing and touch.
Frequency
The second aspect of a unit’s vibration is the frequency of vibration. The term frequency enables us to analyse how interfaces are designed to modulate the speed at which users engage with them to gloss over or emphasise certain aspects of the interface, depending on the intention of that unit. For example, terms and conditions documents related to purchase or service agreements on interfaces such as Apple iTunes are, in effect, regularly designed to be higher frequency. Terms and conditions documents are a legal requirement in many territories, but Apple knows that most users won’t take the time to read them and furthermore, they might not like what they read if they do make the effort to attend to the documents in detail. The higher frequency of the iTunes terms and conditions page is achieved through a combination of the size of the text, which is small, the layout of the text on the page, which is closely packed together with tight line spacing and the use of colour contrast between the text and background, which uses greys and whites to minimise the distinction between foreground and background. In combination, these aspects create a unit that users are likely to skim through or ignore completely, looking only for the ‘I Agree’ button to make the unit disappear and allow them to engage with other units on the site.
But, this is not to say that all terms and condition pages are high frequency or that the same unit could not be designed with a lower frequency. Distinct from a higher frequency unit, a lower frequency terms and conditions unit would enable clear distinctions between types of text and information to encourage slower forms of engagement. It could achieve this by breaking the text of the agreement into sections with clear headers and icons that identify and explain key aspects of the text. Furthermore, rather than presenting all the text on one page, which requires a lot of scrolling on the part of the user, different parts of the document could be split into separate boxes or stages, each of which was designed to dynamically fit into the available space of users’ screens. As this simple example shows, the frequency of a unit is not intrinsically linked to its function but can be actively modulated in an attempt to generate a range of responses from the user.
Rhythmic articulation
The third way of understanding how a unit communicates is in terms of its rhythmic articulation as either more staccato or legato. Analysing a unit’s rhythmic articulation as either staccato or legato helps us understand how interfaces create experiences of connection and distinction between units in order to encourage feelings of effort or effortlessness. In music, legato means playing notes in a manner that is smooth and connected so that there appears to be no gap or distinction between successive notes or tones. Staccato means playing notes in a sharp, distinct and discontinuous manner with clear temporal gaps between successive notes. Playing a musical instrument such as a guitar, this would be expressed as the difference between strumming a set of strings and letting the sound from the strings ring out (legato) and plucking and then muting individual strings with your hand (staccato).
In relation to analysing interfaces, the term legato refers to how units are experienced as smooth and indivisible, compared to staccato, where units are experienced as jerky and divisible. For instance, consider the use of sliders, a common feature on many money and finance websites selling loans, mortgages and other financial products. More specifically, consider how sliders are used on the short-term credit website Wonga.com. The sliders on Wonga.com are two horizontal bars with a button on each bar that the user can manipulate left or right to decrease or increase the amount of money they would like to borrow and how long they would like to borrow it for. The sliders on Wonga.com emit a very legato rhythmic articulation because as you drag the slider button with your finger when visiting the website on a smartphone, or mouse pointer when visiting on a desktop or laptop, it moves smoothly and indivisibly across the screen. This is very different from a competitor’s credit website Longerloans.com, which uses a similar configuration of sliders. The slider unit on Longerloans.com moves in a much more staccato way. As one slides the buttons, they seem to click between different predefined points on the bar, which creates a sense of spatial and temporal distantiation between the units of money or time being selected. A language of legato and staccato vibrations allows us to analyse how these two simple differences might effect how people engage with and ultimately choose to apply for a loan with these companies. On one hand, the legato rhythm of the Wonga.com slider expresses a smooth feel that encourages interaction and a sense of effortlessness. This might give users reassurance that the whole application process will be as effortless as their engagement with the slider and thus encourage them to apply for a loan. The more staccato vibrations of the Longerloans.com slider, on the other hand, might create a sense of inertia or interruption and thus play some small part in discouraging people from applying for a loan (although we imagine this is certainly not the company’s intention).
Another example to illustrate the distinction between the legato and staccato vibrations of units would be the use of vertical menus as a means of selecting and organising content. On iTunes, for example, a vertical menu on the library tab splits a user’s library into ‘Recently added’, ‘Artists’, Albums’ and ‘Songs’, among other categories. In order to navigate a music library through these categorical units, a user has to click between each of them individually. This would be an example of a staccato vibration as each menu can only be clicked one at a time, creating a haptic and visual distinction between different ways of organising the user’s music content. As with the other types of vibration discussed above, staccato and legato vibrations are not only limited to forms of interactive movement or purely visual elements but can also be expressed through other sensory channels such as sound and sound effects. Returning to the Facebook iOS app, we can state that scrolling through the news feed feels very legato and this legato vibration is reinforced by a sound effect that plays when you hit the top of the page. Rather than stopping dead, the news feed item unit bounces as if carrying the momentum of your movement and makes a strange sucking and popping sound to emphasise the elasticity and smooth motion of the unit. However, if scrolling through to the top of the news feed resulted in a clicking sound, we could state that the unit expressed more of a staccato vibration. Regardless of the mechanism employed, what is important here is that modulating the rhythmic articulation of a unit shapes its distinctness from other units within the interface.
Resonance
The final aspect of a unit’s vibration is its capacity to resonate with other vibrations that are not necessarily present within the interface itself. Developing Jewitt’s work on digital multi-modal methods, understanding the resonance of units helps us analyse how interfaces attempt to draw upon users’ experiences, memories and associations with other products and services and their everyday lives to make the interface more appealing or familiar. 39 When designed intentionally, a unit’s resonance attempts to stimulate specific vibrations from users’ bodies. A simple example of this is Wonga.com. Although a high-cost short-term credit website, Wonga.com’s use of blue colours in the slider and logo units is clearly designed to mimic the colours used by established banks and financial institutions. Within Western society, the colour blue has a strong association with trust, loyalty and competence. 40 By utilising a blue logo throughout the interface, Wonga.com is designed to resonate with peoples’ previous experiences of other financial institutions and in doing so transfer the affective sense of trust they might have for these businesses to Wonga.com itself. In turn, these resonances could create a feeling of positive familiarity and thus are presumably designed to encourage people to use Wonga.com over and above other competitors in the short-term credit market. Like all of the other forms of vibration, resonance is not simply linked to a particular aspect of a unit. While the colour of a unit might create resonant vibrations in some instances, in others, resonance might be generated through the shape or size of an icon or button and thus be experienced on a haptic level. For instance, perhaps a slider feels just like a slider on another website that a user is familiar with, which comforts and reaffirms their decision to use the new site. In any case, resonance is one way that a unit’s vibrations can be organised to attempt to create links between previously unconnected elements and so shape how users engage with and respond to a range of interfaces.
Tone
Now we have established how to identify the units that make up interfaces and how they communicate with each other and the user; this allows us to demonstrate how the various aspects of a unit’s vibrations are designed to shape what could be called that unit’s overall tone. For our purposes, tone can be defined as the effect created by a combination of vibrations and how these combinations are designed to shape intuitive and usually un-reflected upon forms of response and action from users of an interface. 41 While the previous section focused on individual aspects of a unit’s vibration, units express multiple forms of vibration and the specific combinations of amplitude, frequency, rhythmic articulation and resonance shape the overall tone of the unit. Indeed, if we take the language of vibration seriously hundreds of different permutations of vibrations and thus tones are possible, which allow us to understand how many different units are arranged in an attempt to prime and shape user responses to interfaces in different ways.
To make sense of this idea of tone, let us return to the iOS Facebook app’s reaction unit that we discussed in the previous section. We could name the overall tone of this unit as playful or inviting. This particular tone is constituted by the way various aspects of its vibration are organised. For example, the reaction unit could be said to express a strong amplitude, through the way the animation of the emoji faces invites interaction. The winking, nodding and smiling emoji’s call out to the user, effectively asking them to slide their finger along the unit to select one. When the user does select a face, this increases the size of the face, thus amplifying its presence within the unit. At the same time, this unit also expresses low-frequency vibrations through the speed and repetition of the animations of the emoji faces. When opening the unit, each face expresses an emotion through moving its mouth, eyes and so on. This animation is around 2 seconds in length and loops continuously, regardless of whether the user is selecting that particular face. This encourages the user to spend a little time examining what each face does, before choosing a particular reaction to add to their Facebook post. Alongside this, the reaction unit also generates staccato vibrations through the yellow colour of each emoji, which is clearly distinguished from the white background upon which the emojis sit. This enables users to clearly differentiate between the emojis and pick an individual emoji without confusion. Finally, the reaction unit emits strong resonant vibrations through the way the faces draw upon pre-existing emoji design conventions (e.g. their circular yellow design). This resonance might create a sense of ease and familiarity for the user because they are used to seeing and using emojis on other sites and apps, such as the Apple message app, the Skype app or the Snapchat app, and therefore increase the likelihood of individuals using this feature on Facebook.
From this example, it is clear that the particular arrangement of vibrations of a unit, the tone these vibrations express and how the tones of various units work together are not accidental. Each unit and its tones are actively designed to work together to shape or prime specific responses or actions to potentially occur. In the Facebook reaction unit, the emoji faces are designed to encourage their use, thus increasing engagement with items on users’ feeds, including paid adverts. From a design perspective, then, changing just a single aspect of a unit’s vibration can in turn alter the unit’s tone. For instance, if the designers at Facebook decided to increase the speed of the emoji reaction animations, this would alter the frequency of vibrations of the unit. Lower frequency vibrations would become higher frequency as the emoji animation looped more quickly, which might alter the tone of the unit. Rather than expressing a playful tone, the unit might express a tone of irritation as users were unable to perceive what animations the emojis were expressing as they looped too quickly. In turn, if one were to alter two or three aspects of the reaction unit’s vibrations, you might end up with a very different tone. Playfulness could become irritation, could become frustration, could become anxiety and so on.
It is important to note here that in the same way that naming a unit and deciding whether something is internal or external to a unit is dependent on the kind of question the researcher is asking, the name of the tone of the unit is also partially open to the interpretation of the user or researcher. What one researcher may term warmth another might term openness. What matters less here is the name given to the tone, than identifying and being able to account for the specific vibrations that produce that tone. One cannot say that the tone of a unit will always create a response that mirrors the intention of the tone, but one can point to what the interface is trying to communicate and how it communicates this in ways that are not reducible to the particular content of a unit (such as image, sound or text).
Conclusion: modulating action
This article is one response to a problem that animates contemporary cultural geography: that new cultural objects are emerging which place in question the habits and practices of analysis that composed the ‘new’ cultural geography. Leaving aside whether or not cultural objects were ever stable and self-contained, we agree with and start from Rose’s call to attend to the specificity of digital technology as a form and force of mediation now inseparable from everyday living. While approaches that repurpose existing qualitative methods or develop new methods to disclose the operation of machinic life are timely and necessary, our method is a response to a problem that both approaches circle but ultimately bypass: how digital interfaces condition without determining action. Learning from existing visual and digital methods, a post-phenomenological approach attempts to stay in a difficult space between celebrating the voluntarism of human users and decrying or wondering at the hidden manipulations of the technical. In this space, we take as the task of research to understand the specifics of digital modulation, or, put differently, how interfaces shape and guide without wholely determining action. Our emphasis is not, then, on what an interface represents – what it stands in for and masks or hides. Instead, we are interested in what interfaces might do, in their efficacy in relation to forms of (non)human action.
In response to this problem, we have offered a series of concepts that are disclosive in aim – they attempt to disclose the specific operation of interfaces in relation to other technical objects and through people’s encounters with digital devices. Crucially, this vocabulary allows us to describe how interfaces shape and guide action without returning to a presumption of manipulation. While this article has focused on how to research the mode of operation of interfaces, it opens up further questions for considering the habitual (or not) encounters and thus relations between people and interfaces, or, in the terms of this article, ethnographies of units, vibrations and tones. In relation to interfaces specifically, thinking sonically allows us to consider how the multiple forms of communication between units, which we have termed vibrations, attempt to construct particular ideal forms of response from the user, but cannot determine what the user does, due to the complexity of these communications. Furthermore, identifying various aspects of vibration allows us to analyse subtle variations in visually similar units, such as sliders or drop-down menus in order to think through their effects in ways that are specific to particular interfaces.
Allowing an analysis of very different interfaces, the concepts of unit, vibration and tone lead us to a procedure for researching digital interfaces. Like any procedure, we offer it in the hope of use, critique and revision:
1. Unit
What makes up an interface?
Identify and differentiate between the modular ‘units’ (maps, graphs, images, texts, buttons, sounds, haptic feedback etc) that make up a digital interface.
Determine which units are to be focused on.
2. Vibration
How do units relate to each other and users?
a) Amplitude
What type of interaction do units encourage and discourage (e.g. quick click through, attention)?
Through what discursive and extra-discursive means do units encourage or discourage interaction (e.g. colour, sound, volume and tone)?
b) Frequency
How do units encourage speed or slowness and for what end (e.g. speed used to minimise reflection on part of user)?
Through what means are faster or slower modes of engagement enabled (e.g. size of text, amount of scrolling on page, length of audio effect)?
c) Rhythmic articulation
What rhythms of engagement do units encourage and discourage (e.g. smooth or stilted forms of viewing or clicking)?
How do these rhythms create senses of connection and disconnection between various units in the interface and cultivate feelings of effort or effortlessness (e.g. colours or sounds used to create feelings of weight or grip when scrolling or clicking)?
d) Resonance
What associations or connections beyond the interface are made present through units and how are those associations made present (e.g. how do units utilise particular elements from other products and services to create similar feelings or affects)?
3. Tone
What is the overall ‘feel’ of an interface (or of an arrangement of units that compose part of an interface)?
How are different overall effects produced through particular permutations of vibration?
Footnotes
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The research reported in this paper was supported by the award of an ESRC standard grant ‘Digital Interfaces and Debt: understanding mediated decision making processes in high cost short term credit products’ (grant number: ES/N012666/1). Dr Ash (PI), Prof Anderson (Co-I), Dr Gordon (RA), Prof Langley (Co-I).