Abstract
Introduction
It is now more than twenty years since the term ‘multimodality’ emerged in the fields of social semiotic, education, and social interaction research, offering a new and broad framework to understand the connections between communication and learning, the significance of the social in meaning-making and the diversity of modes that are used for meaning-making (Bezemer and Kress, 2016). The starting point for multimodality was Halliday’s social semiotic theory of communication (Halliday, 1978), which highlights the social function of semiotic resources and argues that the meaning potential of a resource is dependent on the context of use. Hodge and Kress (1998), and Kress and van Leeuwen (1996, 2006) draw on Halliday’s theorization of how people exchange meanings through socially situated uses of language, but they extend the focus beyond speech and writing to consider all semiotic modes with the same attention to detail for each mode. A key focus for multimodal research in this area is systematically to document and map the relationship across and between modes in texts, interactions, social practices, artefacts and spaces. The methods used in a social semiotic approach to multimodality therefore involve breaking down the object of study into its component parts, working out how the components – or modes – work together to make meaning, and understanding in what ways particular modal choices are shaped by the interpersonal, social and cultural contexts of their use.
Over time, scholars across many disciplines and theoretical traditions have turned to multimodality to understand contemporary social life, yet not all follow the processes that derive from social semiotic analysis (see Jewitt, 2014). This diversity has led to critiques about the multiple ways in which multimodality has been adapted in different disciplines, with many ‘questions about the ‘epistemological compatibility’ of different approaches, when each carries particular theoretical and methodological histories and associations’ (Dicks et al., 2011: 227). Nonetheless, multimodality has gained particular traction in an era of profound social, economic and technological change where developments have reshaped the communicative landscape, in particular with the rapid growth of digital media, their miniaturization, affordability and subsequent ubiquity.
Whilst communication has arguably always been multimodal (Flewitt, 2012), digital technologies and their use raise new challenges for social science research and analysis, particularly with the increasing complexity they bring to researching interaction, communication and meaning making, both in the new environments for research that they offer, and in the tools they make available for data collection, storage and representation. Multimodal methodologies offer timely ways to address the urgent questions raised by these profound changes, by attending to what people say, write, draw, design, look at, how they navigate physical and virtual worlds, through rooms, websites and other spaces, and how they use their hands and other parts of their bodies to interact with computers, devices and other people in face-to-face or virtual encounters. In short: multimodal methodologies have enabled social scientists across disciplines and theoretical traditions to study how people behave and interact in contemporary life through physically present and digitally remote environments. Yet bringing multimodality (and the social semiotic approach that underpins its methodologies) into dialogue with other disciplines continues to produce ‘provocative issues for qualitative research methodology’ (Dicks et al., 2011: 227).
In this special issue, we seek to add the perspectives of new and established scholars working across disciplines who have connected with the broad field of multimodality to explore methodological developments in the study of contemporary communication and interaction. The articles in this issue have been selected to illustrate how multimodality has been used to develop research methods and theoretical understandings of the current digital landscape, to explore methodological innovation, to point to future methodological directions, and to debate the inevitable tensions of bringing multimodality to the study of human society.
We focus on two major fields where cross-disciplinary synergies with multimodality have been particularly productive. Firstly, multimodality increasingly features in contemporary studies that focus on communication and interaction, notably studies in Conversation Analysis (CA). CA has emerged as a distinct interactional strand from ethnomethodology and the work of Harold Garfinkel, to examine the structure and organisation of people’s communication practices, particularly talk-in-interaction (Sacks et al., 1974). Since the wider availability of video recording technology, there has been a methodological shift in considering how bodies are also used in interactions, including a focus on gaze and gesture. The work of Charles Goodwin (e.g., 1981) and Christian Heath (e.g., 1986), among others, pioneered the examination of body movement within the CA framework. More recently, some researchers (for e.g., Mondada, 2008; Sidnell and Stivers, 2005) have described this shift from the study of talk-based social interactions to multimodal interactions. Both social semiotic and interaction-inspired approaches have been applied in face-to-face settings, where participants draw on a range of resources for communication, moving beyond language as the primary concern for mutual understanding. In this special issue, the articles by Elena Davitti and Lauran Doak demonstrate how multimodality contributes to investigations in settings where the use of spoken language, in particular, can be compromised. Davitti considers how the body shapes the practice of dialogue interpreting, while Doak explores how the use of the body and material objects foreground interactions involving children with communication disorders. In both these articles, the affordances and challenges of video technologies and the transcription of video recordings pose important methodological considerations for multimodality, as do the compatibility of terminologies and theorizations that derive from different disciplinary traditions such as ethnography and multimodality (Kress, 2011).
Secondly, multimodality is extending into the field of Human Computer Interaction, where understanding complex interactions and communication in the real world mediated by digital technologies is critical – not only for understanding embodied and sensory interaction, but also for informing design. Two articles in this special issue look at each of these areas. Lamb et al. draw on multimodality as a working methodology for understanding the complexities of contemporary urban space. The authors use multimodal auto-ethnography as a method for capturing and reconstructing the varying narratives that unfold across the city through the use of mobile phones. This provides a new context for a multimodality approach, and extends the scope of multimodal discourse to engage with the ‘colourful and noisy semiotic material of the street’. Yet it also raises issues around the disciplinary compatibility of auto-ethnography with multimodality. The novel interaction modalities of contemporary technologies present new research challenges, particularly in understanding embodied forms of interaction, not only conceptually and theoretically, but also to inform effective design e.g. of learning environments. The article by Malinverni and colleagues thus engages with a multimodality approach in the context of digital environment design, where typically research methods are designer driven and informed by paradigms of Graphical User Interfaces, which the authors argue are inadequate in their capacity to focus on bodily qualities of interaction. This article illustrates how multimodality lent unique and crucial insight into critical design aspects, which were not accessible through monodisciplinary research approaches.
Our final article considers the insights that multimodality can bring to understanding of the relationship between the semiotics of touch, technology and sensory communication. The compatibility of multimodality with sensory ethnography has long been contested (Dicks et al., 2011; Flewitt, 2011; Dicks and Hurdley, 2011; Pink, 2011; Dicks, 2014), and their reliance on fundamentally distinct epistemological commitments means their union will likely always be fractious. Here, Jewitt and Leder Mackley situate multimodality and sensory ethnography within the research terrain of emergent touch technologies (for e.g., haptics, virtual reality, biosensing technologies), and reflect on the complexities of methodological dialogues across paradigmatic boundaries. Whilst acknowledging the tensions between how touch is conceptualised, categorised and represented in the fields of multimodality and sensory ethnography, they sketch out their take on the tensions that arise, and the provocative and productive questions, themes and directions that can emerge from this dialogue for an emergent multimodal and multisensorial agenda for researching digital touch communication.
This special issue intends to showcase the diversity of the field and how different researchers approach multimodality to explore the social world: addressing complex practical situations in the social world (dialogue interpreting by Davitti; social interaction with a child with complex communication needs by Doak) and increasingly convoluted conceptual questions (digital touch by Jewitt and Leder Mackley), and those that involve methodological compatibility (with auto-ethnography by Lamb et al., with sensory ethnography by Jewitt and Leder Mackley), and technological innovation (digital environment design by Malinervi et al.). The special issue also intends to highlight the opportunities and challenges of such diversity in understanding multimodality as a field of inquiry, and increased interest in its focus for enabling more nuanced and deeper understanding of multimodal communication, especially given the expanding context of digital forms of interaction and communication.