Abstract
Multimodality has established itself as a robust ‘field of application’ (p. 21) for unpacking the complexity of discourse. As the field proliferates, frameworks and theoretical grounding of multimodal inquiries often vary and occasionally contradict one another. Building Bridges for Multimodal Research is a timely publication to bring together different perspectives, making multimodality more readily accessible for discourse analysts.
The book includes 20 chapters in five sections, providing a comprehensive coverage of theoretical, methodological, empirical and analytical perspectives on multimodality. Section I has four chapters devoted to theoretical issues, addressing the notion and concepts of multimodality and relevant terminology. Wildfeuer, the editor of the book, introduces the heterogeneity of multimodal research in an effort to justify the necessity to build bridges across disciplinary and geographical boundaries, and provides an overview of the volume. Mavers dwells upon the definition, transcription and analysis of multimodal research through the introduction of thought-provoking questions. Stöckl revisits the concept of mode and explores the textuality of multimodal artifacts with the help of a multimodal-genre-analysis model that merges text linguistics with semiotics, discourse analysis and rhetorics. Kwiatkowska takes a cognitive approach toward semiotics and introduces a descriptive framework in an effort to explore similarities between visual and verbal representations.
Section II has five chapters which attend to methodological issues in multimodal research and demonstrate how disciplines ‘cross-fertilize’ (p. 51) each other. Seizov suggests a visually-based annotation framework for images in the web context by means of combining established linguistic tools with the latest visual communication theories. Handl brings together visual rhetoric and cognitive semantics, proposing a visual-rhetoric-cognitive-semantic approach for the reconstruction of multimodal argument. Mattfeldt extends the analysis of agonality, which in German linguistics refers to the competition of opposing ideas and concepts in a discourse, into multimodal discourses and demonstrates its value in the examination of interplays between verbal and visual discourses. El Naggar puts forward a synergy of the discourse historical approach and the theory of visual grammar, both falling within the broader field of critical discourse analysis, for the analysis of televangelism discourse. Bowcher adopts the concept of generic structure potential, developed in text linguistics, for multimodal analysis in an effort to unpack the realization of contextual configuration through structure.
Section III has five chapters, devoted to empirical examinations of theoretical and methodological models in multimodal research. Ponzetto, Wessler, Weiland, Kopf, Effelsberg and Stuckenschmidt construct a model based on semantic and pragmatic information from images for automatic classification of online iconic images. Debus, Stuckenschmidt and Wessler carry out a multimodal corpus analysis of election manifestos, enriching the study of political science with qualitative and quantitative readings of images. Kaltenbacher and Kaltenbacher employ an eye-tracking tool in the study of website viewing behaviors, filling the gap between speculation about reading paths and actual reading behavior. Siefkes and Arielli combine experimental aesthetics with multimodal research to examine intermodal influences through statistical evaluations. Vales brings together systemic functional linguistics, text analysis and XML-based annotation to propose an annotation framework for the study of comic panels.
Section IV includes three concrete example analyses of multimodal artifacts. Hiippala examines the structure and layout of bilingual double pages in in-flight magazines using computer vision and multimodal analysis. Schmitt analyzes the audio-visual metaphors in a TV advertisement, which highlights the notion of mode as a semiotic and phenomenological object. Tseronis offers a detailed analysis of documentary films by combining argumentation studies with multimodal analysis, which further illustrates how these two approaches inform each other.
The final section presents three analytical adventures into less studied multimodal territories. Chapters by Krasni and Maxwell focus on Google News Aggregator and medieval manuscripts, respectively, and Moalla reports a study on visual literacy at tertiary level.
A theme that runs through all the chapters is bridge building, and this is clearly articulated in each chapter. Innovative synergies have been created by bringing together disciplines including linguistics, communication studies, computer science, psychology, visual rhetoric, political science, argumentation theory, aesthetics, musicology and medieval studies. These innovations on the one hand give rise to methodological development of multimodal research, and on the other hand accentuate the intricacy of multimodal documents when these texts are being cross-examined under the lens of multiple disciplines. The latter, in particular, makes this edited volume of great value to discourse analysts, as an increasing number of researchers in the broader field of social science find discourse analysis a tool of significant value.
The bridge across geographical borders is, in my view, less significant than those across disciplinary ones, despite the editor’s claim of internationality. The collection is, to some extent, Eurocentric. Although the language settings under investigation range from European languages (e.g. English, German and Finish) to Eastern ones (Arabic and Chinese), the Arabic and Chinese texts are the only two non-European ones in the volume. The analyses of these two cases do not seem to incorporate methodological perspectives that are unique to these two languages. Neither do the two studies sufficiently address the ‘communicative load’ (Martinec, 2003) of different types of semiotic resource which set the two Eastern cultures apart from their Western counterparts. It would be desirable to see and hear perspectives and voices from a broader range of geographical communities and language settings to further ‘open up to more interdisciplinary work’ (p. 20) as a continuation of the inspiring discussions in this volume on interdisciplinary innovations.
Overall, the volume provides rich information on approaches to multimodal research, which can be readily adopted by discourse analysts. Furthermore, the interdisciplinary theoretical innovations in the book shed light on potential methodological developments for discourse studies.