Book review: Charles J. Forceville and Eduardo Urios-Aparisi (eds),Multimodal Metaphor (Applications of Cognitive Linguistics 11). Berlin: Mouton de Gruyter,2009. 470 pp.,98 illus. ISBN 978–3–11–020515–2 (hbk)

This book may rightly be called a milestone publication for its ambitious aspirations alone. It seeks to fuse two of the major research paradigms in late 20th and early 21st century social sciences and linguistics, namely the cognitive and the multimodal. Cognitive linguists have argued that metaphor is a matter of thought and conceptual structure, which are only made manifest through language and other semiotic forms of expression. Multimodality research is a late but natural response to the well-known fact that discourse and communication are hardly ever realized in language alone but more often than not through a combination of signing modalities in various media. The natural connection between these two hitherto separate strands of research is spelt out in the book. Cognitive linguistics uses multimodal texts to study the diverse manifestations of conceptual metaphors in pictures, sound and gestures and to see how metaphors are constructed combining these modes, i.e. multimodally. Multimodality theory looks at the crucial function of metaphors for linking various modes and for meaning making in text and discourse. The attempt at symbiosis is a strategic move designed to be conducive to both domains of enquiry. Yet it is probably fair to say that the key questions and methodologies mainly reflect the concerns of conceptual metaphor theory even though what the individual media case studies reveal is sure to also be profitable for multimodality research at large.

The very idea of multimodal metaphor forms the theoretical cornerstone of the book. In contrast to verbal (and other mono-modal) metaphor, it is defined as those metaphors ‘whose target and source are each represented (also: rendered, p. 4) exclusively or predominantly in different modes’ (p. 24, cf. also p. 4). So for a conceptual metaphor like happiness is light, the text in a commercial or feature film might render the target happiness in a phrase (e.g. she lit up, he radiates joy) whereas the moving image may make manifest the source through a beam of light illuminating the scene, a torch shone at someone or a candle being lit. By the same token, music could signal happiness via an upward moving tune in one of the light-hearted major keys. Multimodal metaphors, then, cue metaphorical meaning through sign combinations from various signing modes. Given that definitions of ‘mode’ are ‘extremely thorny’ (p. 4) as they involve at least the aspects of sensory channel (e.g. auditory vs visual), medium (e.g. speech vs writing) and semiotic rules, the venture to describe, classify and analyse multimodal metaphors cannot be an easy one. Add to this the fact that modes blend (e.g. music into noise or vice versa), overlap (e.g. typography and picture) and mix in myriad, complex ways, and the enterprise must seem daunting and demanding. The book tackles the task in two ways: first, it focuses on a set of recurrent questions central to the cognitive linguistic paradigm and, secondly, it assembles neatly delineated media case studies each dealing with a professional domain of semiotic work, a genre and a combination of modes. The scope of work covers advertising, cartoons, comics/mangas/animated films, co-speech gestures, popular music and film.

Although every chapter would merit individual scrutiny and comments, this review must for reasons of space and for the sake of generalization proceed differently. I have, therefore, decided to single out the most central concerns running through the various contributions in order to explain and critically assess them. What I hope to be able to point out in this fashion are the major strands of thought in the cognitive and multimodal paradigms, their recurring concerns and the problems, blind spots and grey areas this kind of research confronts.

The genre- and/or mode-oriented enquiries into metaphor in chapters 3 to 18 are introduced by two rather general contributions by the editors. Urios-Aparisi and Forceville present a very fine-grained overview of the research topics and questions in the book outlining the connections, synergies and ‘thematic patterns’ (p. 11). Forceville gives a systematic and also partly historical outline of some of the fundamental notions underlying multimodal metaphor research. Apart from providing definitions, the text reflects on the nature of similarities that can underlie non-verbal metaphor and on the essential differences between conventional and idiosyncratic metaphor. However, perhaps symptomatically, it has comparatively little to say about the influence of genre on metaphor production and interpretation – simply because this is an area hardly studied systematically as yet.

In keeping with the tradition of conceptual metaphor theory, the most pervasive element in the majority of chapters is a concern with the kinds of metaphors contained in the texts studied. So the authors invariably set out to identify the targets and sources that are realized in the signing modes, genres and media at hand. We learn among other metaphorical preferences that brands are living organisms (Koller), wines are three-dimensional artefacts (Caballero), a car is a person (Urios-Aparisi), the ballot box is a dice (Yus), us policy is boot crushing other countries (El Refaie), ideas are physical objects (Schilperoord and Maes), anger is a dangerous animal (Eerden), anger is hot fluid in a container (Shinohara and Matsunaka), introducing an idea is producing an object (Müller and Cienki), syntactic structures are geometric physical structures (Mittelberg and Waugh), pitch relationships are relationships in physical space (Zbikowski) and human victim is animal (Eggertsson and Forceville). Consistent with the conceptual metaphor enterprise (cf. Kövecses, 2002: 281 ff.) the book contains a metaphor/metonymy index (p. 467 ff.), which is useful for the initiated scholar as it allows retrieval and comparison of frequent metaphors and their concrete manifestations in diverse semiotic forms of expression. If we disregard for the moment that having to use language to materialize conceptual structure may create circularity and might also involve individual bias and ambiguity in the naming, then, clearly, spelling out target and source of the multimodal metaphors makes up the core business of the kind of research presented in the volume. As multimodality research is a text and discourse analytical discipline and, therefore, necessarily a critical kind of linguistics it will gladly take on board any insight into frequently used targets and sources. This is empirically validated knowledge about what kinds of social representations are construed through metaphor and this in turn allows judgements to be made about perspectives, world-views, power relations and ideologies/myths constructed and pursued in texts. Perhaps the book ought to have emphasised this more in the focus of its individual chapters and as the key to a successful link between the cognitive and the multimodal (but cf. Forceville, p. 34).

A more specific and markedly multimodal enquiry of the book is directed at how sources and targets are distributed across the various modes. We learn that the concreteness and sensory directness of visuals and sound/music make them suited to realizing source domains whereas language preferentially expresses target domains (Koller and Forceville, ch. 16). Some targets, however, like the taste and smell of wine, tend towards the mono-modal, verbal metaphor and use little imagery (Caballero). Not infrequently, source domains cut across language and visuals (Shinohara and Matsunaka) or sources and targets distribute between the modes (Urios-Aparisi, Müller and Cienki), and generally mono- and multimodal metaphors seem to cooperate (Eggertsson and Forceville). As a preliminary conclusion a model of ‘dual encoding’ (Koller, pp. 45, 62) seems plausible, which suggests that the cognitive and communicative effects are optimized if metaphors build up using various modes. Considering that mode inter-linking in multimodality research is generally seen as following the principle of ‘complementarity’ or ‘transcriptivity’, which means that modes each contribute according to their potentials and deficits, too structuralist an approach to multi-modal metaphor must be detrimental. After all, despite varying patterns in the text it is the recipient who ‘zips’ cues from the modes together to arrive at a plausible reading of the metaphorical meanings.

The question I find most fascinating and promising for future multimodality research is how the specificity of the mode constrains and facilitates metaphor construal. Here the book offers a few preliminary insights. So Eerden indicates that it is the sensory and media properties of a mode that determine the way they can contribute to metaphorical meaning, e.g. animated film offers dynamic semiotic resources of the moving image, which do not have an equivalent in language. Forceville highlights that where language has deictic expressions to explicitly link modes, time-based media must rely on ‘well-timed simultaneity’ (p. 395). Zbikowski, finally, pays attention to ‘sonic analogues’ in music (p. 376) like e.g. ascending/rising pitch, which may cue gestures or emotional states. It is these timid beginnings on the way to understanding the semiotic and textual properties of modes, which deserve most attention and future dedication, as they mark an ideal halfway point between the disciplines of the cognitive and the multimodal.

Besides concerns with individual genres and modes, some contributions also focus on the more general question of how metaphorical expressions in various modes are perceived and understood. Starting out from the basic assumption that ‘the comprehension of verbal, visual and multimodal metaphors involves similar mental procedures’ (Yus, p. 147) the relevant chapters (Yu, Yus, El Refaie, Schilperoord and Maes, Zbikowski) set great store by pointing out that metaphor understanding must be more than just the static mapping of conceptual elements. Rather it involves flexible ad-hoc interpretation and a good sense of an optimally relevant reading conditioned by context and situation. Not surprisingly, those authors favour Fauconnier’s and Turner’s more dynamic conceptual blending over mapping as it allows for exactly the kind of flexibility and creativity needed in the modelling of textual and multimodal understanding. Multimodality research can afford to remain blind to the internal cognitive debates about competing models, but it will of necessity have a vested interest in a rich textual interpretation.

It is a great merit that the book adheres to the sharp delineation between metaphor and metonymy as practised in cognitive linguistics (cf. Kövecses, 2002: 143 ff.) and also addresses the specific role metonymy plays in multimodal meaning making. Urios-Aparisi shows various interactions between metonymy and metaphor in TV-commercials (cf. also Forceville, ch. 16), which suggest that metonymy’s main functions are to highlight the target of a metaphor, prepare a metaphor’s conceptual ground by providing rich knowledge about source or target and signpost those elements crucial to the mapping. For gestures, Mittelberg and Waugh argue that ‘these two figures of thought jointly structure multimodal representations of grammatical concepts’ (p. 329). Here, metonymy ‘leads the way into metaphor’ (p. 347) in that first, the gesturing hand is a metonym for an imaginary object, and secondly, this object drawn into the air metaphorically stands for an abstract idea, e.g. a sentence or morphemes. With a view to the future of a cognitively grounded multimodality research, I should like to underline the importance of looking beyond metaphor at the vast repertoire of rhetorical figures. All of them are but manifestations of conceptual structures and operations, which build multimodal texts. Here is a vast field opening up for multimodal rhetoric and cognitive semiotics.

A final issue would seem to be high on the agenda of conceptual metaphor theorists. Yet, but for the exception of Shinohara and Matsunaka, it does not surface in the book. This concerns the question of whether (and how much of) metaphor is culture-specific or rather universal. Because it is early days in cognitive multimodality research this omission can easily be excused. Yet it should be obvious that cultures may prioritize and handle modes differently in multimodal artefacts. It is also known that while embodied metaphors indeed seem universal, various cultures produce differing kinds of experiential knowledge that might motivate different metaphors in other semiotic manifestations. Again this could be a promising focal point for future research.

What I hope to have shown is that the book is rich in strategic thought and analytical detail that fruitfully engage both with multimodality theory and cognitive semiotics. The volume manages to live up to its high aspirations, yet beyond tentatively mapping out a new field it also reveals some of its inherent difficulties. The first concerns the need for transcribing multimodal texts in order to fix their temporally fleeting or spatially complex structures (cf. Schneider and Stöckl, 2011: 28 ff.). If one wants to show how modes interrelate and which roles metaphor/metonymy play in this, multimodal data is needed in a systematic form. A second difficulty is also a methodological one. The book demonstrates that metaphorical meaning can be notoriously complex and hard to model. Rather than just rely on textual structures it would be worth complementing the analysis by media–ethnographic empirical methods, like eye tracking, interviews and questionnaires in order to be able to describe actual recipient interpretations (cf. El Refaie). Finally, conceptual metaphor theory ought to be prepared to acknowledge its limitations. Churning out conceptual metaphors for a sheer infinite variety of metaphorical expressions may be helpful to create order and system. Yet it also disguises the underlying problem: the mentalese A = B shorthand is verbal and reflects the linguistic habits of a culture and the bias of the analyst. Rather than worry too much about concepts in the mind and preconceived static mappings, it would be sound to look closely at the processes of ‘interactive property attribution’ (Glucksberg, 2001; cf. Stöckl, 2010: 190 ff.) in multimodal texts. Also the concrete forms and types of metaphorical expressions and their effects on the meaning potential created deserve more attention – both in language and in other non-verbal semiotic modes.