Deconstructing a kaleidoscope of semiotic modes and audience perception of multimodality in advertising discourse

The term multimodality in linguistics refers to investigating language not in isolation but in interaction with other modes such as music, images, gestures, symbols, etc. (see Kress, Multimodal Discourse, 2001, and Kress, Multimodality: A Social Semiotic Approach to Contemporary Communication, 2009; also O’Halloran, Multimodal Discourse Analysis, 2011). Advertising discourse has been selected as the focal point of this research since it is to be perceived as a kaleidoscopic representation of these elements.

In line with previous works on the multimodality of contemporary discourse, multimodal metaphors, pragmatics and reception studies, the main goal of this research is to investigate the role of language and other semiotic modes (music, gaze, gestures) contributing to the success of advertising discourse, as well as their impact on the audience. ELAN software developed by the Max Planck Institute for Psycholinguistics was used to annotate three television commercials and generate multimodal transcripts to be analysed.

In addition, a quantitative and qualitative analysis of audience data was conducted in accordance with previous observations that the role of the receiver in meaning-making has been neglected in the study of language and multimodal communication (Cook 2001, Kjeldsen and Hess 2021). Fifty educated UK participants were asked to evaluate different persuasive devices (slogan, celebrity endorsement, music and sound effects, product information, humour and fine print) realized through multiple semiotic modes, including spoken language, written text, music, gaze and gestures in an attempt to challenge the entrenched belief that advertising discourse addresses uneducated people and is far less effective in other circumstances (see Goddard’s The Language of Advertising: Written Texts, 1998). The results of this research are expected to highlight the role of multiple semiotic modes in influencing the perception and impact of advertising communication, whose final goal is to influence audience perceptions and lead to product purchase.