The Impact of Cross-Sensory Interaction Between Emotional Images and Musical Modes on the Aesthetic Evaluation of Neutral Stimuli

Abstract

Previous studies have shown that major key music paired with positive emotional images is typically associated with positive emotions, while minor key music paired with negative images tends to be linked to sadness. However, there is still a lack of research on the integrative mechanisms underlying cross-modal emotion transference between music and image. The present study employed a 2 (image: positive vs. negative) × 2 (music: major vs. minor) within-subjects design to investigate how these combinations influence aesthetic preferences for neutral stimuli. The findings revealed the following: (1) Under minor key music, negative emotional images significantly enhanced the aesthetic evaluation of neutral stimuli compared to positive emotional images. This reflects an emotion-congruent effect, facilitating emotion assimilation–contrast processes and enhancing aesthetic experience. (2) Under positive emotional images, major key music significantly increased the aesthetic appraisal of neutral stimuli relative to minor key music, again promoting assimilation–contrast effects and aesthetic enhancement. (3) Under negative emotional images, there was no significant difference in aesthetic evaluation of neutral stimuli between major and minor key music, possibly because the negative images dominated the emotional processing, marginalizing the influence of music. (4) Under major key music, no significant difference in aesthetic evaluation of neutral stimuli was observed between positive and negative emotional images, which may suggest that, within the emotional framework of major key music, the emotional impact of positive and negative images converged and thus failed to significantly alter aesthetic preferences. These findings reveal boundary conditions for the emotion assimilation–contrast effect in the context of cross-modal emotion transference. From an applied perspective, musical mode and emotional image can serve as effective modulators in the perception and aesthetic evaluation of visual information.

Keywords

cross-sensory emotional images musical modes aesthetic evaluation neutral stimuli

Get full access to this article

View all access options for this article.

References

Arielli

(2012). Contrast and assimilation in aesthetic judgments of visual artworks. Empirical Studies of the Arts, 30(1), 59–74. https://doi.org/10.2190/EM.30.1.f

Armitage

Eerola

(2022). Cross-modal transfer of valence or arousal from music to word targets in affective priming? Auditory Perception & Cognition, 5(3-4), 192–210. https://doi.org/10.1080/25742442.2022.2087451

Bai

Tao

(2016). The response effects of Chinese and western music on emotion. Acta Psychologica Sinica, 48(7), 757. https://doi.org/10.3724/SP.J.1041.2016.00757

Barrett

N. F.

Schulkin

Bernacer

(2017). Emotional granularity and the musical enjoyment of sadness itself. Behavioral and Brain Sciences, 40, e351. https://doi.org/10.1017/S0140525X17001595

Baumgartner

Esslen

Jäncke

(2006). From emotion perception to emotion experience: Emotions evoked by pictures and classical music. International journal of psychophysiology, 60(1), 34–43. https://doi.org/10.1016/j.ijpsycho.2005.04.007

Bekhtereva

Craddock

Müller

M. M.

(2021). Emotional content overrides spatial attention. Psychophysiology, 58(8), e13847. https://doi.org/10.1111/psyp.13847

Blizek

(1973). Aesthetics and Psychobiology by DE Berlyne. Journal of Aesthetics and Art Criticism, 31(4), 553. https://doi.org/10.2307/429334

Bradley

M. M.

Lang

P. J.

(2007). The International Affective Picture System (IAPS) in the study of emotion and attention. In J. A. Coan, & J. J. B. Allen (Eds.), Handbook of emotion elicitation and assessment (pp. 29–46). Oxford University Press.

Bruns

Pombo

Ripollés

Pelli

D. G.

(2023). Emotions of subject and object affect beauty differently for images and music. Journal of Vision, 23(13), 6–6. https://doi.org/10.1167/jov.23.13.6

10.

De Gelder

Vroomen

(2000). The perception of emotions by ear and by eye. Cognition & Emotion, 14(3), 289–311. https://doi.org/10.1080/026999300378824

11.

Ethofer

Anders

Erb

Droll

Royen

Saur

, … Wildgruber

(2006). Impact of voice on emotional judgment of faces: An event-related fMRI study. Human Brain Mapping, 27(9), 707–714. https://doi.org/10.1002/hbm.20212

12.

Friedman

R. S.

Song

S. E.

Cox

G. E.

(2024). Reexamining the association between aesthetic sensitivity to musical and visual complexity. Empirical Musicology Review, 19(2), 77–87. https://doi.org/10.18061/emr.v19i2.9705

13.

Gabrielsson

Lindström

(2010). The role of structure in the musical expression of emotions. In Juslin

P. N.

Sloboda

(Eds.), Handbook of music and emotion: Theory, research, applications (2nd ed., pp. 367–400). Oxford Unviersity Press.

14.

Gerdes

A. B.

Wieser

M. J.

Alpers

G. W.

(2014). Emotional pictures and sounds: A review of multimodal interactions of emotion cues in multiple domains. Frontiers in Psychology, 5, 1351. https://doi.org/10.3389/fpsyg.2014.01351

15.

Hunter

P. G.

Schellenberg

E. G.

Schimmack

(2010). Feelings and perceptions of happiness and sadness induced by music: Similarities, differences, and mixed emotions. Psychology of Aesthetics, Creativity, and the Arts, 4(1), 47–56. https://doi.org/10.1037/a0016873

16.

Ishizu

Zeki

(2011). Toward a brain-based theory of beauty. PloS One, 6(7), e21852. https://doi.org/10.1371/journal.pone.0021852

17.

Jeong

J. W.

Diwadkar

V. A.

Chugani

C. D.

Sinsoongsud

Muzik

Behen

M. E.

, … Chugani

D. C.

(2011). Congruence of happy and sad emotion in music and faces modifies cortical audiovisual activation. NeuroImage, 54(4), 2973–2982. https://doi.org/10.1016/j.neuroimage.2010.11.017

18.

Juslin

P. N.

Västfjäll

(2008). Emotional responses to music: The need to consider underlying mechanisms. Behavioral and Brain Sciences, 31(5), 559–575. https://doi.org/10.1017/S0140525X08005505

19.

Kawakami

Furukawa

Katahira

Okanoya

(2013). Sad music induces pleasant emotion. Frontiers in Psychology, 4, 311. https://doi.org/10.3389/fpsyg.2013.00311

20.

Klein

Melnyk

Völckner

(2021). Effects of background music on evaluations of visual images. Psychology & Marketing, 38(12), 2240–2246. https://doi.org/10.1002/mar.21588

21.

Liu

Pei

(2018). Texture-aware emotional color transfer between images. IEEE Access, 6, 31375–31386. https://doi.org/10.1109/ACCESS.2018.2844540

22.

Logeswaran

Bhattacharya

(2009). Crossmodal transfer of emotion by music. Neuroscience Letters, 455(2), 129–133. https://doi.org/10.1016/j.neulet.2009.03.044

23.

Reber

Schwarz

Winkielman

(2004). Processing fluency and aesthetic pleasure: Is beauty in the perceiver’s processing experience? Personality and Social Psychology Review, 8(4), 364–382. https://doi.org/10.1207/s15327957pspr0804_3

24.

Redies

(2008). A universal model of esthetic perception based on the sensory coding of natural stimuli. Spatial Vision, 21(1), 97–117. https://doi.org/10.1163/156856808782713780

25.

Schmitz

T. W.

Rosa

E. D.

Anderson

A. K.

(2009). Opposing influences of affective state valence on visual cortical encoding. The Journal of Neuroscience, 29(22), 7199–7207. https://doi.org/10.1523/JNEUROSCI.5387-08.2009

26.

Sherif

Hovland

C. I.

(1961). Social judgment: Assimilation and contrast effects in communication and attitude change.

27.

Smit

E. A.

Milne

A. J.

Dean

R. T.

Weidemann

(2022). Evaluative conditioning of responses to unfamiliar chords by exposure to valenced images. Psychology of Music, 50(2), 579–595. https://doi.org/10.1177/03057356211008972

28.

Straehley

I. C.

Loebach

J. L.

(2014). The influence of mode and musical experience on the attribution of emotions to melodic sequences. Psychomusicology: Music, Mind, and Brain, 24(1), 21–34. https://doi.org/10.1037/pmu0000032

29.

Sunaga

(2018). How the sound frequency of background music influences consumers’ perceptions and decision making. Psychology & Marketing, 35(4), 253–267. https://doi.org/10.1002/mar.21084

30.

Tiihonen

Haumann

N. T.

Shtyrov

Vuust

Jacobsen

Brattico

(2024). The impact of crossmodal predictions on the neural processing of aesthetic stimuli. Philosophical Transactions of the Royal Society B: Biological Sciences, 379(1895), 20220418. https://doi.org/10.1098/rstb.2022.0418

31.

Timmers

Crook

(2014). Affective priming in music listening: Emotions as a source of musical expectation. Music Perception: An Interdisciplinary Journal, 31(5), 470–484. https://doi.org/10.1525/mp.2014.31.5.470

32.

Weijkamp

Sadakata

(2017). Attention to affective audio-visual information: Comparison between musicians and non-musicians. Psychology of Music, 45(2), 204–215. https://doi.org/10.1177/0305735616654216

33.

Wyczesany

Ligeza

Tymorek

Adamczyk

(2018). The influence of mood on visual perception of neutral material. Acta Neurobiologiae Experimentalis, 78(2), 163–172. https://doi.org/10.21307/ane-2018-015

34.

Zatorre

R. J.

(2005). Music, the food of neuroscience? Nature, 434(7031), 312–315. https://doi.org/10.1038/434312a