Abstract
In the communication landscape of the new media age we are witnessing a fast transition from ‘the world told’ to ‘the world shown’ (Kress, 2003). Words, either in oral or written modes, become progressively more visual; the screen is now their obvious site of appearance; oneness with sound and movement is now the norm, rather than the exception; sharing is fast, cheap and easy as never before. This scenery poses challenges to any field of research remotely interested in human communication, but it is of profound relevance for corpus linguistics, which aims at contributing to generate objective analysis of the actual patterns of language use. Multimodality and Active Listenership: A Corpus Approach is about how language researchers can face this challenge. It provides the state-of-the-art in multimodal corpus linguistic research and corpus development, presenting practical information on the development and use of multimodal linguistic corpora. Corpus linguistics has been firmly rooted in the analysis of text-based, written or spoken data. In this book, however, Knight champions the notion that multimodal corpora are an important resource for studying and analyzing the principles of human communication.
Monomodal corpora have a long history of use in linguistics. Multimodal corpora, on the other hand, are still scarce due to copyright restrictions, among other reasons. However, a range of multimodal corpora does exist and the book offers an index of such databases. Most of them are not freely available and focus on languages other than English. To give an idea of the variety of compositional characteristics of the corpora presented in the book, it is worth mentioning the AMI Meeting Corpus, which contains 100 hours of recordings from three different meeting rooms, and the IFADV Corpus, a free dialog video corpus composed of face-to-face interaction between close friends/colleagues in 20 15-minute conversations (five hours in total).
The book is divided into eight chapters. Chapter 1 presents an overview of what multimodal corpora are and how to use them. Multimodal corpora are defined as physical repositories of annotated content on communication channels, including speech, gaze, hand gesture and body language. The integration of textual, audio and video records of communicative events in contemporary multimodal corpora, the author argues, provides a platform for the exploration of a range of lexical, prosodic and gestural features of conversation for investigations of the ways in which these features interact in real everyday speech.
Chapter 2 approaches practical issues related to the physical construction of multimodal corpora, discussing in pragmatic terms themes such as design and infrastructure, size and scope, naturalness and availability and reusability. In this chapter the author also discusses multimodal corpora transcription, coding and marking up, presenting as well computer software which enables researchers to transcribe and code digitally audio and video records of communication. The corpus most extensively alluded to in the book to provide evidence and support for claims being made is the Nottingham Multi-modal Corpus (Knight et al., 2008). It comprises 13 videos and audio recordings of academic situations, such as single speaker lectures and sessions between MA or PhD students and their supervisors. The corpus comprises 250,000 words, compiled between 2005 and 2008.
Chapters 3 and 4 are dedicated to demonstrating how multimodal corpora can be used to enhance our understanding of the interplay between language and gesture-in-use in the generation of meaning in interaction. Chapter 3 focuses on non-verbal behavior, specifically head nods. Chapter 4 analyses the roles, forms and functions of linguistic backchanneling behaviors. Both chapters depart from the history of research in non-verbal communication, but move forward, suggesting new frontiers for the possible contribution a multimodal perspective opens for corpus linguistics into language and gesture-in-talk research.
The objective of the next two chapters is to demonstrate how a multimodal corpora approach can fill a gap left by previous studies which investigated real-life conversational contexts. Such studies only provided detailed analysis of either spoken or non-verbal behavior, failing to provide an exhaustive account of the collaborative, simultaneous use of spoken backchannels and nods. Chapters 5 and 6 present a corpus-based methodology for the actual investigation of multimodal corpora, extensively offering examples of how new software and methodologies can be used to enhance the description and understanding of the complex relationship between language and gesture-in-talk.
Chapters 7 and 8 develop new approaches to the analysis of multimodal corpora, distinguishing them from simple video analysis. Here Knight presents a defense of the usefulness, principles and methods of analysis of large datasets. However, the development of movement tracker techniques for the analysis of extensive corpora still needs to be greatly improved. It is clear that the integration of automated gesture - and behavior- detection utilities into the research of human communication is a target on the horizon.
For those researchers who observe the intriguing complexity of the current communicative landscape with a mix of interest and curiosity, Multimodality and Active Listenership: A Corpus Approach will make a good guide for the introduction to the technical, practical and ethical issues and challenges around the construction of multimodal corpora. In conclusion, this book is recommended not only for corpus linguistic researchers, but also for anyone interested in making contributions to the understanding of communication patterns in real contextualized human interaction.