Crossmodal Correspondences in the Sounds of Chinese Instruments

Abstract

People tend to associate stimuli from different sensory modalities, a phenomenon known as crossmodal correspondences. We conducted two experiments to investigate how Chinese participants associated musical notes produced by four types of Chinese instruments (bowed strings, plucked strings, winds, and percussion instruments) with different colors, taste terms, and fabric textures. Specifically, the participants were asked to select a sound to match each color patch or taste term in Experiment 1 and to match the experience of touching each fabric in Experiment 2. The results demonstrated some associations between pitch and color, taste term, or the smoothness of fabrics. Moreover, certain types of Chinese instruments were preferentially chosen to match some of the colors, taste terms, and the texture of certain fabrics. These findings therefore provided insights about the perception of Chinese music and shed light on how to apply the multisensory features of sounds to enhance the composition, performance, and appreciation of music.

Keywords

crossmodal correspondence Chinese music pitch music perception timbre

People experience the surrounding world by constantly integrating information and associating stimuli from different sensory modalities (Spence & Squire, 2003), such as sound and color (Spence, 2011). In particular, people tend to associate visual forms in bright colors (e.g., red or yellow) with high-pitched tones or sounds and darker colors (e.g., blue) with low-pitched sounds (Curwen, 2018; P. Walker, 2016). Such crossmodal correspondences between sounds and colors are mediated by common emotional responses to them (Lindborg & Friberg, 2015; Palmer, Schloss, Xu, & Prado-León, 2013). However, to the best of our knowledge, it remains unclear how sounds produced by Chinese instruments are associated with colors. Therefore, this study was conducted to investigate crossmodal correspondences between musical notes produced by Chinese instruments and other features such as color.

Studies have found that people tend to associate sounds with tastes (Knöferle & Spence, 2012; Simner, Cuskley, & Kirby, 2010). For instance, Crisinel and Spence (2009, 2010a, 2010b) conducted a series of studies to demonstrate associations between higher pitched musical notes (from piano or stringed, woodwind, or brass instruments) and a sweet or sour taste along with associations between lower pitched sounds and a salty or bitter taste. These music–taste associations could modulate the influence of music on people’s perception of foods or drinks (Carvalho et al., 2015; Carvalho, Wang, Van Ee, Persoone, & Spence, 2017; Knöferle, Woods, Kappler, & Spence, 2015; Q. Wang, Spence, & Keller, 2017). These findings suggested that music can deliver information regarding nonauditory characteristics such as color and taste (P. Walker, 2016); however, it remains unclear whether colors and tastes associated with the same or similar sounds are congruent in terms of color–taste associations (e.g., Wan et al., 2014).

Moreover, audio–tactile interactions have been found to play a prominent role in musical performances (Gunther & O’Modhrain, 2003), as playing an instrument requires coordinating motions and corresponding auditory outcomes (Occelli, Spence, & Zampini, 2011). One study revealed that tactile stimuli could result in the co-activation of the auditory cortex (Schürmann, Caetano, Hlushchuk, Jousmäki, & Hari, 2006), and sounds could alter haptic perception (Ro, Hsu, Yasar, Elmore, & Beauchamp, 2009). Other studies revealed that sounds could also influence people’s tactile perception. For example, the amplification of sounds could increase the perception of roughness (Guest, Catmur, Lloyd, & Spence, 2002). Moreover, sounds synchronous with hand rubbing could modify the perceived roughness and dryness of the palms (Jousmäki & Hari, 1998). High- and low-frequency sounds could selectively alter participants’ discrimination of high- and low-frequency vibration, respectively (Yau, Olenczak, Dammann, & Bensmaia, 2009). Even nontactile sounds such as white noise could influence the perceived roughness of fingers (Suzuki, Gyoba, & Sakamoto, 2008) or faces (Suzuki & Gyoba, 2011).

Moreover, the timbre of sounds has been found to alter people’s perception of surface stiffness (Liu & Ando, 2016). Timbre consists of a set of auditory attributes that frequency, intensity, and duration cannot account for (McAdams & Giordano, 2009; Saitis, Weinzierl, von Kriegstein, Ystad, & Cuskley, 2018). Timbre interacts with pitch or loudness to influence listeners’ discrimination and classification of sounds (Melara & Marks, 1990; Reiterer, Erb, Grodd, & Wildgruber, 2008). Eitan and Rothschild (2011) examined the semantic associations between timbre and smoothness or softness. Compared with the sounds produced by violin, flute sounds were generally rated as smoother and softer by participants. Mu, Zhao, and Wan (2015) asked participants to match the texture of different fabrics and sounds produced by four types of Western instruments (piano, strings, woodwind, and brass), and their results revealed some associations between the piano sounds and the texture of a fabric sample rated as smooth and soft, such as faux fur. Collectively, these findings provided some empirical evidence regarding the associations between timbre and the sense of touch.

Okamoto, Nagano, and Yamada (2013) defined roughness–smoothness, hardness–softness, and coldness-warmness as three fundamental dimensions of people’s tactile perception with materials. In other words, smoothness and softness are often used as antonyms of roughness and hardness, respectively (Eitan & Rothschild, 2011; Fujisaki, Tokita, & Kariya, 2015; Ludwig & Simner, 2013). Roughness and hardness were the researchers’ main interests in some studies (Guest et al., 2002; Jousmäki & Hari, 1998; Suzuki & Gyoba, 2011; Suzuki et al., 2008). In contrast, smoothness and softness were emphasized more in the touch experience with fabrics (e.g., Peck & Wiggins, 2006). Previous research also demonstrated that the subjective ratings of a fabric’s smoothness and softness were positively correlated, and both were correlated with the pleasantness ratings of the fabric (Mu et al., 2015). However, to the best of our knowledge, it remains unclear how sounds produced by Chinese instruments are associated with the smoothness and softness of fabrics.

Therefore, this study was conducted to investigate crossmodal correspondences between musical notes produced by Chinese instruments and color, taste, and fabric texture. We chose to focus on musical notes produced by Chinese instruments in this study for two reasons. First, it remains unclear whether music from different cultural backgrounds, such as Western and non-Western music, share similar multisensory features. It should be noted that some aspects of music perception are universal across cultures, such as temporal processing (Drake & Bertrand, 2001) and tonality induction (Krumhansl, 2000). By contrast, many other aspects of music perception are culturally dependent, such as phrase perception (Nan, Knösche, & Friederici, 2006), musical synchronization (Drake & Ben El Heni, 2003), attention allocation (Arikan et al., 1999; Zhu et al., 2009), and music comprehension (Morrison et al., 2003). An event-related potential study demonstrated that guqin (one of the oldest Chinese instruments) sounds evoked a greater P300 in the frontal regions of Chinese participants’ brains than did piano sounds (Zhu et al., 2008), hinting at a potential difference in the perception of Chinese and Western music.

Second, a lot of previous studies have used Western music or music notes produced by Western instruments in multisensory research (e.g., Carvalho et al., 2017; Crisinel & Spence, 2010b, 2011; Mu et al., 2015; Q. Wang & Spence, 2015; Q. Wang, Woods, & Spence, 2015), whereas very few studies have tested on Chinese music or music notes produced by Chinese instruments. The concept of music (known as Yue) is one of the fundamental categories of Confucian thought. Performance of Chinese music is closely linked to Chinese philosophy (e.g., Chow-Morris, 2010) and values the harmony between feminine, quiet, and dark elements (referred to as Yin) and masculine, loud, and light elements (referred to as Yang). Traditional Chinese music often uses a pentatonic scale with the notes of Gong, Shang, Jue, Zhi, and Yu (e.g., Chen, 1989). During the 20th century, performance of Chinese music underwent stylistic changes (Y. Wang, 2010), and the technique for playing Western instruments has been applied to playing similar Chinese instruments (van der Linden, 2015). Even though Chinese instruments have been traditionally classified into eight categories (referred to as Bayin), modern Chinese orchestras have learned from Western orchestras about orchestra division and stage layout. Nowadays, a Chinese orchestra often has four sections: bowed strings, plucked strings, winds, and percussion sections (Han & Gray, 1979).

As shown in Figure 1, we chose four Chinese instruments to use in this study: erhu (2-string vertical fiddle), guzheng (21-string plucked zither), dizi (transverse bamboo flute), and yunluo (a set of small tuned gongs mounted in a wooden frame) to represent bowed strings, plucked strings, winds, and percussion instruments, respectively. In Experiment 1, we examined crossmodal correspondences between sounds and colors or taste terms. In Experiment 2, we examined crossmodal correspondences between sounds and the touching experience of varied fabrics.

Figure 1.

The erhu, guzheng, dizi, and yunluo instruments that local musicians used to record sounds for this study.

Experiment 1

The purpose of the present experiment was to examine crossmodal correspondences between musical notes produced by Chinese instruments and varied color patches or taste terms. Using Crisinel and Spence’s (2011) experimental paradigm, we asked participants to choose a sound to match each color patch or taste term. The colors and taste terms were chosen based on Wan et al.’s (2014) study of color–taste associations.

Methods

Participants

Eighty undergraduate or graduate students (mean age = 21.2 ± 2.03 years, ranging from 18 to 25 years; 40 males and 40 females) from a major university in mainland China took part in the present experiment. None of them reported to have color blindness or hearing impairments. The present and the following experiments were approved by a local ethics committee. Moreover, all participants signed informed consent forms before the experiment started and were compensated with 30 Chinese Yuan each for participation.

Apparatus and stimuli

The auditory stimuli used in the study were 28 sounds that local musicians produced using an erhu, guzheng, dizi, or yunluo (see Figure 1). It should be noted that these instruments have narrower range of musical notes than Western instruments such as the piano. Based on the common range of notes available to these four instruments, we chose to use pitch levels ranging from D4 (294 Hz) to D6 (1175 Hz) in intervals of three tones, including D4, #F4 (370 Hz), bB4 (466 Hz), D5 (587 Hz), #F5 (740 Hz), bB5 (932 Hz), and D6. Each sound was edited using Adobe Audition CS6 software. The tones lasted 1.5 seconds and were delivered to the participants through closed-ear headphones (Edifier W800BT) at a loudness of 70 dB.

Similar to Wan et al.’s (2014) study of color–taste associations, we used a total of 11 color patches (each subtending 200 × 200 pixel) in the present experiment, including black (H: 0°, S: 0%, B: 0%), blue (H: 240°, S: 100%, B: 100%), brown (H: 0°, S: 75%, B: 65%), green (H: 120°, S: 100%, B: 100%), gray (H: 0°, S: 0%, B: 50%), orange (H: 39°, S: 100%, B: 100%), pink (H: 350°, S: 25%, B: 100%), purple (H: 300°, S: 100%, B: 50%), red (H: 0°, S: 100%, B: 100%), white (H: 0°, S: 0%, B: 100%), and yellow (H: 60°, S: 100%, B: 100%). We also used five Chinese words to describe sour, sweet, bitter, salty, and umami tastes. Each word was presented in the Song font at a font size of 60 and then fitted within a 200-by-200-pixel box. Inquisit 3.0 software was used to present these color patches and taste terms against a gray background (H: 0°, S: 0%, B: 85%) and to record the data.

Design and procedure

The experimental task was to choose a sound to match each color patch or taste term. We used a within-participants design, such that each participant completed a total of 16 trials—11 color trials and 5 taste trials. The order of these two types of trials was counterbalanced among participants, but all stimuli within each category were presented in random order. Each trial started with one color patch or taste term being presented on the first computer screen. The participants were instructed to choose a sound from the available sound icons shown on the second computer screen to match this color patch or taste term (Figure 2). Similar to Crisinel and Spence’s (2010b) study, we presented these sound symbols on four scales corresponding to four instruments, with the pitch increasing from left to the right. Each scale represented one type of instrument, and we used a Latin-square design to counterbalance the order of the four instruments. The participants were instructed to click each icon to hear the corresponding sound, and they were permitted to hear as many sounds as necessary before making a selection.

Figure 2.

A screenshot of the display presented to the participants to choose music notes.

Results

Associations between musical notes and colors

Similar to Crisinel and Spence’s (2010b, 2011) study, we transformed the selected pitch to musical instrument digital inference (MIDI) note numbers. Specifically, the pitch levels of D4, #F4, bB4, D5, #F5, bB5, and D6 were transformed to MIDI note numbers of 62, 66, 70, 74, 78, 82, and 86, respectively. The mean pitch values selected for color patches are shown in Figure 3. We performed a repeated-measure analysis of variance (ANOVA) on the data, and the results revealed a significant main effect of Color, F(10, 790) = 10.40, p < .001, $η_{p}^{2}$ = .12. In addition, pairwise comparisons with Bonferroni’s correction¹ revealed that the selected pitch for the color red was higher than that for the six other colors of black, blue, brown, gray, pink, or purple, all ts > 3.94, ps < .01. By contrast, the selected pitch for the colors gray and black was lower than that for the four other colors of brown, green, orange, and yellow, all ts > 3.52, ps < .04. Furthermore, the pitch matched to gray was also lower than that for pink, t(79) = 3.70, p = .02, and the pitch matched to blue was lower than that for yellow, t(79) = 3.79, p = .02. None of the other pairwise comparisons reached the significance level, all ts < 3.29, ps > .08.

Figure 3.

Mean pitch and instruments selected for each color in Experiment 1. Error bars in the upper panel show the standard errors of the means. *p < .05. **p < .01. ***p < .001.

The types of instruments chosen for each color patch are also shown in Figure 3. We performed a Color (11 color patches) × Instrument (erhu, guzheng, dizi, or yunluo) log-linear analysis on the data, and the results revealed a significant interaction term, χ² = 195.95, p < .001. χ² tests revealed that the instrument choices were not equally distributed among the four options for the colors black, brown, green, pink, red, and white, all χ² > 14.80, ps < .03. We further performed one-sample t tests to examine whether the top choice was significantly higher than the chance level of 25%. The results revealed significant preferences in the choices of instrument for the five colors: erhu for the color black (45%), t(79) = 3.57, p = .006, and red (43%), t(79) = 3.15, p = .02; dizi for the color brown (43%), t(79) = 3.15, p = .02; and yunluo for colors pink (50%), t(79) = 4.44, p < .001, and white (46%), t(79) = 3.79, p = .003. No such preference was significant for the color green, t(79) = 2.30, p = .17, and the instrument choices were equally distributed among the four options for the colors blue, gray, orange, purple, and yellow, all χ² < 6.31, ps > .90. Moreover, we also calculated the correlation between the brightness level of each color and the MIDI values of the selected pitch for all the chosen pairs. The results revealed a significant positive correlation, r = .25, p < .001.

Associations between musical notes and tastes

The mean pitch matched to each taste term is shown in Figure 4. A one-way repeated-measure ANOVA on the data revealed a significant main effect of Taste, F(4, 316) = 18.23, p < .001, $η_{p}^{2}$ = .19. Pairwise comparisons revealed that the selected pitch for bitterness was the lowest, all ts > 3.30, ps < .02. In contrast, the selected pitch for saltiness was the second lowest, all ts > 3.09, ps < .03. None of the other pairwise comparisons reached the significance level, all ts < 1.42, ps > .90.

Figure 4.

Mean pitch and instruments selected for each taste term in Experiment 1. Error bars in the upper panel show the standard errors of the means. *p < .05. ***p < .001.

The instruments chosen for each taste term are also shown in Figure 4. We performed a Taste (sourness, sweetness, bitterness, saltiness, or umami) × Instrument (erhu, guzheng, dizi, or yunluo) log-linear analysis on the data, and the results revealed a significant interaction term, χ² = 239.29, p < .001. χ² tests revealed that the instrument choices were not equally distributed among the four options for all taste terms, all χ² > 13.20, ps <.03. One-sample t tests revealed that both the yunluo and guzheng were selected more often than the chance level (25%) for sweetness, both ts > 3.35, ps <.01; and there was no significant difference between the frequencies of these two choices (48% vs. 44%), t(79) = 0.35, p = .73. Similarly, both the dizi and erhu were selected more often than the chance level for bitterness, both ts > 3.14, ps < .02; and there was no significant difference in the frequencies of these two choices (44% vs. 43%), t(79) = 0.12, p = .91. The results also revealed that the dizi sounds were preferentially chosen for saltiness (46%), t(79) = 3.79, p = .002. In contrast, the yunluo sounds were preferentially matched to umami taste (56%), t(79) = 5.60, p < .001. By contrast, no significant preference was found for sourness, t(79)= 1.42, p > .90.

Discussion

In this experiment, the results demonstrated that higher pitched sounds were preferentially matched to the color red than to the colors black or gray. These color–pitch associations, found with musical notes produced by Chinese instruments, were consistent with previous findings with simple tones or musical notes produced by Western instruments (Simpson, Quinn, & Ausubel, 1956; Ward, Huckstep, & Tsakanikos, 2006). We also observed a significant positive correlation between the selected pitch values and the brightness of the colors, which was also in line with the previous findings about color–pitch associations (Curwen, 2018; P. Walker, 2016, for reviews). Moreover, we also observed some pitch–taste associations, as lower pitch was more preferentially matched to a bitter or salty taste than sourness, sweetness, or umami. These results were consistent with Crisinel and Spence’s (2009, 2010a, 2010b) findings with musical notes produced by piano, strings, woodwind, or brass. Even though we used a narrower pitch range (starting with D4) compared with previous studies, we were able to identify significant pitch–color and pitch–taste associations, suggesting the robustness of these crossmodal correspondences (also see P. Walker, 2016).

We also demonstrated that certain types of Chinese instruments were preferentially matched to some colors and taste terms. These results therefore provided more empirical evidence of crossmodal correspondences for timbre, as previous research demonstrated timbre–shape associations (Adeli, Rouat, & Molotchnikoff, 2014) and timbre–color associations for sounds produced by Western instruments (Mu et al., 2015). Specifically, our results revealed that erhu sounds were preferentially matched to the colors black and red and bitter taste. The dizi sounds were matched to the color brown and bitter or salty tastes, whereas yunluo sounds were matched to the colors pink and white and the sweet and umami taste. Wan et al. (2014) demonstrated associations between the color black and bitterness and between the color pink and sweetness, whereas Heller (1999) reported brown–bitterness associations. Moreover, Piqueras-Fiszman, Alcaide, Roura, and Spence (2012) demonstrated that the same strawberry mousse was rated as sweeter when it was served in a white plate than when it was served in a black plate, suggesting some associations between white color and sweetness. Therefore, our results demonstrated that some colors and tastes associated with similar sounds were congruent with each other in terms of color–taste associations.

The results of this Experiment 1 also revealed that some Chinese instruments were preferentially chosen to match color and tastes that were incongruent in terms of color–taste associations. First, our participants preferentially matched erhu sounds to the color red and bitter taste separately, whereas the color red was not associated with bitterness in a previous study of color–taste associations (Wan et al., 2014). Second, dizi sounds were matched to the color brown in one task but to the bitter and salty taste in another task. However, saltiness was not associated with the color brown in the previous research (Spence et al., 2015). Third, yunluo sounds were matched to the colors pink and white in the music–color association task but to sweetness and umami in the music–taste task. Nevertheless, previous research did not uncover any pink–umami or white–umami associations (Spence et al., 2015).

Experiment 2

The purpose of the experiment was to examine audio–tactile associations in the musical notes produced by Chinese instruments. As mentioned in the introduction, we chose to use fabrics as experimental stimuli and therefore focused on the smoothness and softness of these stimuli. Based on Mu et al.’s (2015) findings about the smoothness and softness ratings of eight fabrics, we chose four fabrics to test in this experiment: silk (shimmering fabric that feels smooth and soft), faux fur (pile fabric that feels soft and fluffy), terrycloth (woven fabric that feels soft and fluffy), and wool (woven fabric that feels coarse).

Methods

Eighty new Chinese participants (mean age = 21.5 ± 2.10 years, ranging from 18 to 27 years; 40 males and 40 females) took part in the experiment. All aspects of the methods were the same as those used in Experiment 1 except for the following differences. In this experiment, the task was to choose a sound to match the touch experience of silk, faux fur, terrycloth, and wool. Each fabric was cropped to 20 cm × 20 cm in size and hung in a clothes hanger placed on a movable clothes rack. The participants cleaned their hands before the experimenter blindfolded them. Then, they were instructed to touch a paper towel (also cropped to the same size as the fabrics) to familiarize themselves with the procedure of properly touching a fabric by feeling, rubbing, and twisting it.

The main experiment consisted of four trials. During each trial, blindfolded participants first properly touched a fabric. Their blindfolds were removed after the experimenter hid the fabric, and the participants were then asked to choose a sound to match the touching experience. After that, they were also asked to separately rate the extent to which the fabric felt smooth and soft on two 7-point scales (1 = extremely rough and 7 = extremely smooth on one scale, and 1 = extremely hard and 7 = extremely soft on the other scale). This procedure was repeated until all trials were completed. A Latin-squared design was used to counterbalance the order of these four fabrics among the participants.

Results

Smoothness and softness ratings of fabrics

The mean smoothness and softness ratings of each fabric are shown in Figure 5. First, the results of the smoothness and softness ratings revealed a significant positive correlation between these two types of ratings, r = .45, p < .001. Next, repeated-measure ANOVAs on these two types of ratings also revealed a significant main effect of fabric on the smoothness scores, F(3, 237) = 164.26, p < .001, $η_{p}^{2}$ = .68, and on the softness scores, F(3, 237) = 38.70, p < .001, $η_{p}^{2}$ = .33. Pairwise comparisons revealed that silk was considered to be the smoothest, all ts > 5.68, ps < .001. Faux fur was considered to be the second smoothest, both ts > 8.94, ps < .001. However, the difference between the smoothness of wool and terrycloth did not reach the significance level, t(79) = 2.63, p = .06. Furthermore, silk and wool were considered to be the softest and the hardest fabrics, respectively, all ts > 4.72, ps < .001. In addition, the faux fur and terrycloth received comparable softness scores, t(79) = 1.12, p > .90.

Figure 5.

Mean smoothness and softness ratings for each fabric are shown in the upper panel and the lower panel, respectively. Error bars show the standard errors of the means. ***p < .001.

Associations between musical notes and fabric texture

The mean pitch and instruments chosen for each fabric are shown in Figure 6. First, a repeated-measure ANOVA on the selected pitch values revealed a significant main effect of fabric, F(3, 237) = 11.75, p = .001, $η_{p}^{2}$ = .13. Pairwise comparisons revealed that the pitch matched to terrycloth was lower than that for silk or wool, both ts > 2.77, ps < .05. None of any other pairwise comparisons reached the significance level, all ts < 2.20, ps > .18.

Figure 6.

Mean pitch and instruments selected for each fabric in Experiment 2. Error bars in the upper panel show the standard errors of the means. *p < . 05. ***p < .001. MIDI = musical instrument digital inference.

Second, a Fabric (silk, faux fur, terrycloth, or wool) × Instrument (erhu, guzheng, dizi, or yunluo) log-linear analysis on the instrument choice data revealed a significant interaction term, χ² = 90.18, p < .001. χ² tests revealed that the instrument choices were not equally distributed among the four options for all fabrics, all χ² > 25.80, ps < .001. One-sample t tests revealed that both guzheng and yunluo were selected more often than the chance level for silk, both ts > 2.72, ps < .05; and there was no significant difference between the frequencies of these two choices (40% vs. 41%), t(79) = .12, p = .90. The results also revealed that the dizi sounds were preferentially matched to the texture of faux fur (50%), t(79) = 4.44, p < .001; wool (46%), t(79) = 3.79, p = .002; and terrycloth (46%), t(79) = 3.79, p = .002.

Discussion

In summary, the experiment revealed two major results. First, the smoothest and softest fabric in this study (i.e., silk) was associated with the guzheng or yunluo sounds, whereas the dizi sounds were preferentially matched to the texture of faux fur, wool, and terrycloth. These results demonstrated some associations between the timbre and touch of fabrics that were in line with the literature about sounds produced by Western instruments, such as the violin, flute, and piano (Eitan & Rothschild, 2011; Mu et al., 2015). Despite the positive correlations between the softness and smoothness ratings in the present experiment (also see Mu et al., 2015), terrycloth was rated as less smooth than faux fur with comparable softness ratings and softer than wool with comparable smoothness ratings. Collectively, these results suggested that the smoothness and softness of fabrics both contribute to the crossmodal mapping between timbre and the perception of fabric texture.

Second, our results revealed that the selected pitch for terrycloth was significantly lower than that for wool, and terrycloth was rated as softer than wool with comparable smoothness ratings. This result was in line with Eitan and Rothschild’s (2011) findings that high-pitched sounds produced by the piano or flute were rated as being harder, rougher, and lighter than low-pitched sounds (Eitan & Timmers, 2010). However, our results also revealed that the selected pitch for terrycloth was lower than that for a smoother and softer fabric: silk. This result was inconsistent with Eitan and Rothschild’s (2011) findings about semantic associations, but such a discrepancy could be attributed to the possibility that tactile properties other than smoothness or softness, such as weight (P. Walker, Scallon, & Francis, 2017), may also play an important role in the crossmodal mapping between pitch and fabric texture. In this study, the weights of the silk, faux fur, terrycloth, and wool samples were 7 g, 60 g, 14 g, and 20 g, respectively. Therefore, it is possible that our participants selected a higher pitch for silk than for terrycloth based on the weight difference of these two types of fabrics. Future research is needed to test this possibility and the possible interaction between different tactile properties.

General Discussion

Taken together, the results of this study revealed significant crossmodal correspondences between musical notes produced by Chinese instruments and color, taste, and perception of fabric smoothness and softness. The results of Experiment 1 revealed some color–pitch and taste–pitch associations for musical notes produced by bowed strings, plucked strings, winds, and percussion instruments from a Chinese orchestra. These results, observed with Chinese participants, were consistent with previous studies conducted with Western participants and musical notes produced by Western instruments (Crisinel & Spence, 2009, 2010a, 2010b). These findings therefore suggest that crossmodal mapping between pitch and color or taste is universal across varied musical styles or different populations. Considering the mediating role of affective factors in music–taste (Q. Wang & Spence, 2015) and music–color associations (Barbiere, Vidal, & Zellner, 2007; Lindborg & Friberg, 2015; Palmer et al., 2013), it is possible that these crossmodal correspondences for Chinese music may be moderated by other attributes of Chinese music that can trigger different emotions, such as harmony and rhythm. Future research is needed to investigate the influence of these attributes.

Our results also demonstrated that some colors and tastes associated with similar sounds were crossmodally congruent with each other in terms of color–taste associations. These findings suggested that stimuli across three modalities may be congruent with each other, forming a multimodality congruency based on a common mapping to the same connotative meaning (see L. Walker, Walker, & Francis, 2012). By contrast, the results of Experiment 1 also revealed that some Chinese instruments were preferentially chosen to match colors and tastes that were incongruent in terms of color–taste associations. One explanation for this phenomenon is that crossmodal associations across different modalities are transitive but are not necessarily consistently aligned from one modality to another (Deroy, Crisinel, & Spence, 2013). Alternatively, timbre is a multifarious set of abstract sensory attributes (McAdams & Giordano, 2009), so it is also possible for timbre–color and timbre–taste associations to be determined by different sensory attributes of timbre, such as attack sharpness, brightness, nasality, and richness.

The results of Experiment 2 demonstrated some associations between timbre and fabric smoothness and softness. Specifically, the guzheng and yunluo sounds were preferentially selected to match the smoothest and softest fabric in this study, silk, whereas the dizi sounds were preferentially matched to the texture of three other types of fabrics used in this study (i.e., faux fur, wool, and terrycloth). These results, obtained with sounds produced by Chinese instruments, demonstrate some crossmodal associations between timbre and the perception of smoothness and softness that were in line with previous findings with Western instruments (Eitan & Rothschild, 2011; Mu et al., 2015). Collectively, these findings hinted that a crossmodal mapping between timbre and certain tactile properties. Despite the positive correlations between softness and smoothness ratings in this study and Mu et al.’s (2015) study, our results also indicated that different fabrics featured with varied levels of smoothness or softness may be associated with the same or a different timbre or pitch. This finding suggested that different aspects of the touching experience may contribute to audio–tactile interactions. The crossmodal correspondences for musical sounds may stem from the visual or tactile features of the instruments (e.g., size, typical color, and shape), various acoustical correlates of timbre (McAdams, 2013), or cultural contexts of instruments, such as musical genres or associated cultural practices.

As with any study, there were some limitations as far as the interpretation and generalizability of this study are concerned. For one, the stimuli used in this study were simple musical notes, so it would be interesting to test with more complex musical stimuli such as clips (Albertazzi, Canal, & Micciolo, 2015; Guetta & Loui, 2017; Kontukoski et al., 2015; Tan & Kelly, 2004) on participants with varied cultural backgrounds in future research. In addition, the fabrics used in this experiment were generally soft (compared with many other stimuli), so caution should be exercised when one tries to generalize the findings with fabrics to other materials with higher levels of hardness.

In conclusion, the results of this study demonstrated associations between musical notes produced by Chinese instruments and stimuli from other sensory modalities, including color, taste, and fabric texture. These findings provided some insights regarding the perception of traditional Chinese music. Similar to balancing different tastes in cooking, the multisensory features of sounds may be considered in composition (P. Walker, 2016) in order to obtain the harmony between different musical elements that Chinese philosophy and traditional Chinese music value (e.g., Chow-Morris, 2010). Therefore, the findings of this study have direct implications on how to apply the crossmodal features of musical sounds to enhance the composition, performance, and appreciation of music.

Furthermore, understanding the multisensory features of sounds may be also helpful in choosing the most appropriate music to influence consumers’ perception, decision-making, and consumption (North, Hargreaves, & McKendrick, 1997; North, Sheridan, & Areni, 2016; Zellner, Geller, Lyons, Pyper, & Riaz, 2017). Therefore, the findings of this study may also have direct implications on choosing appropriate Chinese music to influence international consumers’ subjective evaluation of (Elder & Krishna, 2012) and purchasing intentions for Chinese products (Hagtvedt & Brasel, 2016; Hultén, 2011). Considering that music has been found to fill in missing information from other sensory modalities (P. Walker, 2016), our findings may also influence how to choose appropriate Chinese music as background music for video introduction of Chinese products to compensate for the lack of gustatory and tactile information available in online shopping.

Footnotes

Acknowledgements

The authors would like to thank Tsinghua University Chinese Orchestra for technical support.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was supported by the National Natural Science Foundation of China (Grant Numbers 71472106 and 71872097) awarded to Xiaoang Wan.

ORCID iD

Xiaoang Wan

Note

References

Adeli

Rouat

Molotchnikoff

(2014). Audiovisual correspondence between musical timbre and visual shapes. Frontiers in Human Neuroscience, 8, 352.

Albertazzi

Canal

Micciolo

(2015). Cross-modal associations between materic painting and classical Spanish music. Frontiers in Psychology, 6, 424.

Arikan

M. K.

Devrim

Oran

Ö.

Inan

Elhih

Demiralp

(1999). Music effects on event-related potentials of humans on the basis of cultural environment. Neuroscience Letters, 268, 21–24.

Barbiere

Vidal

Zellner

(2007). The color of music: Correspondence through emotion. Empirical Studies of the Arts, 25, 193–208.

Carvalho

F. R.

van Ee

Rychtarikova

Touhafi

Steenhaut

Persoone

Spence

(2015). Using sound-taste correspondences to enhance the subjective value of tasting experiences. Frontiers in Psychology, 6, 1309.

Carvalho

F. R.

Wang

van Ee

Persoone

Spence

(2017). “Smooth operator”: Music modulates the perceived creaminess, sweetness, and bitterness of chocolate. Appetite, 108, 383–390.

Chen

(1989). Ancient Chinese music notation. Anuario Musical, 44, 239–257.

Chow-Morris

(2010). Going with the flow: Embracing the Tao to China’s Jiangnan Sizhu. Asian Music, 41, 59–87.

Crisinel

A. S.

Spence

(2009). Implicit association between basic tastes and pitch. Neuroscience Letters, 464, 39–42.

10.

Crisinel

A. S.

Spence

(2010a). A sweet sound? Food names reveal implicit associations between taste and pitch. Perception, 39, 417–425.

11.

Crisinel

A. S.

Spence

(2010b). As bitter as a trombone: Synesthetic correspondences in nonsynesthetes between tastes/flavors and musical notes. Attention Perception & Psychophysics, 72, 1994–2002.

12.

Crisinel

A. S.

Spence

(2011). Crossmodal associations between flavoured milk solutions and musical notes. Acta Psychologica, 138, 155–161.

13.

Curwen

(2018). Music-colour synaesthesia: Concept, context and qualia. Consciousness and Cognition, 61, 94–106.

14.

Deroy

Crisinel

A.-S.

Spence

(2013). Crossmodal correspondences between odors and contingent features: Odors, musical notes, and geometrical shapes. Psychonomic Bulletin & Review, 20, 878–896.

15.

Drake

Ben El Heni

(2003). Synchronizing with music: Intercultural differences. Annals of the New York Academy of Sciences, 999, 429–437.

16.

Drake

Bertrand

(2001). The quest for universals in temporal processing in music. Annals of the New York Academy of Sciences, 930, 17–27.

17.

Eitan

Rothschild

(2011). How music touches: Musical parameters and listeners’ audio-tactile metaphorical mappings. Psychology of Music, 39, 449–467.

18.

Eitan

Timmers

(2010). Beethoven’s last piano sonata and those who follow crocodiles: Cross-domain mappings of auditory pitch in a musical context. Cognition, 114, 405–422.

19.

Elder

R. S.

Krishna

(2012). The “visual depiction effect” in advertising: Facilitating embodied mental simulation through product orientation. Journal of Consumer Research, 38, 988–1003.

20.

Fujisaki

Tokita

Kariya

(2015). Perception of the material properties of wood based on vision, audition, and touch. Vision Research, 109, 185–200.

21.

Guest

Catmur

Lloyd

Spence

(2002). Audiotactile interactions in roughness perception. Experimental Brain Research, 146, 161–171.

22.

Guetta

Loui

(2017). When music is salty: The crossmodal associations between sound and taste. PLoS One, 12, e0173366.

23.

Gunther

O’Modhrain

(2003). Cutaneous grooves: Composing for the sense of touch. Journal of New Music Research, 32, 369–381.

24.

Hagtvedt

Brasel

S. A.

(2016). Crossmodal communication: Sound frequency influences consumer responses to color lightness. Journal of Marketing Research, 53, 551–562.

25.

Han

K. H.

Gray

(1979). The modern Chinese orchestra. Asian Music, 11, 1–43.

26.

Heller

(1999). Wie farben wirken. Farbpsychologie, farbsymbolik, kreative farbgestaltung [How colour works. Colour psychology, colour symbolism, working creatively with colour]. Hamburg, Germany: Reinbek Press.

27.

Hultén, B. (2011). Sensory marketing: the multi-sensory brand-experience concept. European Business Review, 23(3), 256–273.

28.

Jousmäki

Hari

(1998). Parchment-skin illusion: Sound-biased touch. Current Biology, 8, R190–R191.

29.

Knöferle

K. M.

Spence

(2012). Crossmodal correspondences between sounds and tastes. Psychonomic Bulletin & Review, 19, 1–15.

30.

Knöferle

K. M.

Woods

Kappler

Spence

(2015). That sounds sweet: Sing cross-modal correspondences to communicate gustatory attributes. Psychology & Marketing, 32, 107–120.

31.

Kontukoski

Luomala

Mesz

Sigman

Trevisan

Rotola-Pukkila

Hopia

A. I.

(2015). Sweet and sour: Music and taste associations. Nutrition & Food Science, 45, 357–376.

32.

Krumhansl

C. L.

(2000). Tonality induction: A statistical approach applied cross-culturally. Music Perception, 17, 461–479.

33.

Lindborg

Friberg

A. K.

(2015). Colour association with music is mediated by emotion: Evidence from an experiment using a CIE Lab interface and interviews. PLoS One, 10, e0144013.

34.

Liu

Ando

(2016). Metal sounds stiffer than drums for ears, but not always for hands: Low-level auditory features affect multisensory stiffness perception more than high-level categorical information. PLoS One, 11, e0167023.

35.

Ludwig

V. U.

Simner

(2013). What colour does that feel? Tactile-visual mapping and the development of cross-modality. Cortex, 49, 1089–1099.

36.

McAdams

(2013). Musical timbre perception. In Deutsch

(Ed.), The psychology of music (pp. 35–67). San Diego, CA: Academic Press.

37.

McAdams

Giordano

B. L.

(2009). The perception of musical timbre. In S. Hallam, I. Cross, & M. Thaut (Eds.), Oxford handbook of music psychology (pp. 72–80). Oxford, England: Oxford University Press.

38.

Melara

R. D.

Marks

L. E.

(1990). Interaction among auditory dimensions: Timbre, pitch, and loudness. Perception & Psychophysics, 48, 169–178.

39.

Morrison

S. J.

Demorest

S. M.

Aylward

E. H.

Cramer

S. C.

Maravilla

K. R.

(2003). fMRI investigation of cross-cultural music comprehension. NeuroImage, 20, 378–384.

40.

Zhao

Wan

(2015). Crossmodal correspondences for tactile texture and influence on consumers’ ratings. Talk given at the 2015 Annual Conference of the JMS Marketing Science, October 24–25, 2018, Shanghai, China.

41.

Nan

Knösche

T. R.

Friederici

A. D.

(2006). The perception of musical phrase structure: A cross-cultural ERP study. Brain Research, 1084, 179–191.

42.

North

A. C.

Hargreaves

D. J.

McKendrick

(1997). In-store music affects product choice. Nature, 390, 132.

43.

North

A. C.

Sheridan

L. P.

Areni

C. S.

(2016). Music congruity effects on product memory, perception, and choice. Journal of Retailing, 92, 83–95.

44.

Occelli

Spence

Zampini

(2011). Audiotactile interactions in temporal perception. Psychonomic Bulletin & Review, 18, 429–454.

45.

Okamoto

Nagano

Yamada

(2013). Psychophysical dimensions of tactile perception of textures. IEEE Transactions on Haptics, 6, 81–93.

46.

Palmer

S. E.

Schloss

K. B.

Prado-León

L. R.

(2013). Music-color associations are mediated by emotion. Proceedings of the National Academy of Sciences, 110, 8836–8841.

47.

Peck

Wiggins

(2006). It just feels good: Customers’ affective response to touch and its influence on persuasion. Journal of Marketing, 70, 56–69.

48.

Piqueras-Fiszman

Alcaide

Roura

Spence

(2012). Is it the plate or is it the food? Assessing the influence of the color (black or white) and shape of the plate on the perception of the food placed on it. Food Quality and Preference, 24, 205–208.

49.

Reiterer

Erb

Grodd

Wildgruber

(2008). Cerebral processing of timbre and loudness: FMRI evidence for a contribution of Broca’s area to basic auditory discrimination. Brain Imaging & Behavior, 2, 1–10.

50.

Hsu

Yasar

N. E.

Elmore

L. C.

Beauchamp

M. S.

(2009). Sound enhances touch perception. Experimental Brain Research, 195, 135–143.

51.

Saitis

Weinzierl

von Kriegstein

Ystad

Cuskley

(2018). Timbre semantics through the lens of crossmodal correspondences: A new way of asking old questions. Paper presented at the International Symposium on Universal Acoustical Communication, Sendai, Japan.

52.

Schürmann

Caetano

Hlushchuk

Jousmäki

Hari

(2006). Touch activates human auditory cortex. NeuroImage, 30, 1325–1331.

53.

Simner

Cuskley

Kirby

(2010). What sound does that taste? Cross-modal mappings across gustation and audition. Perception, 39, 553–569.

54.

Simpson

R. H.

Quinn

Ausubel

D. P.

(1956). Synesthesia in children: Association of colors with pure tone frequencies. Journal of Genetic Psychology, 89, 95–103.

55.

Spence

(2011). Crossmodal correspondences: A tutorial review. Attention, Perception, & Psychophysics, 73, 971–995.

56.

Spence

Squire

(2003). Multisensory integration: Maintaining the perception of synchrony. Current Biology, 13, R519–R521.

57.

Spence

Wan

Woods

Velasco

Deng

Youssef

Deroy

(2015). On tasty colours and colourful tastes? Assessing, explaining, and utilizing crossmodal correspondences between colours and basic tastes. Flavour, 4, 23.

58.

Suzuki

Gyoba

(2011). Effects of sound on the tactile perception of roughness in peri-head space. Seeing and Perceiving, 24, 471–483.

59.

Suzuki

Gyoba

Sakamoto

(2008). Selective effects of auditory stimuli on tactile roughness perception. Brain Research, 1242, 87–94.

60.

Tan

Kelly

(2004). Graphic representations of short musical compositions. Psychology of Music, 32, 191–212.

61.

van der Linden

(2015). Non-Western national music and empire in global history: Interactions, uniformities, and comparisons. Journal of Global History, 10, 431–456.

62.

Walker

(2016). Cross-sensory correspondences: A theoretical framework and their relevance to music. Psychomusicology: Music, Mind, and Brain, 26, 103–116.

63.

Walker

Scallon

Francis

(2017). Cross-sensory correspondences: Heaviness is dark and low-pitched. Perception, 46, 772–792.

64.

Walker

Francis

(2012). A common scheme for cross-sensory correspondences across stimulus domains. Perception, 41, 1186–1192.

65.

Wan

Woods

A. T.

van den Bosch

J. J.

McKenzie

K. J.

Velasco

Spence

(2014). Cross-cultural differences in crossmodal correspondences between basic tastes and visual features. Frontiers in Psychology, 5, 1365.

66.

Wang

Spence

(2015). Assessing the effect of musical congruency on wine tasting in a live performance setting. i-Perception, 6(3), 1–13. 2041669515593027.

67.

Wang

Spence

Keller

(2017). Sounds spicy: Enhancing the evaluation of piquancy by means of a customised crossmodally congruent soundtrack. Food Quality and Preference, 58, 1–9.

68.

Wang

Woods

A. T.

Spence

(2015). “ What’s your taste in music?” A comparison of the effectiveness of various soundscapes in evoking specific tastes. i-Perception, 6(6), 1–23. 2041669515622001.

69.

Wang

(2010). Expressiveness in the premodern performance style of Chinese music: “Equanimity” in Abing. Asian Music, 41, 127–165.

70.

Ward

Huckstep

Tsakanikos

(2006). Sound-colour synaesthesia: To what extent does it use cross-modal mechanisms common to us all? Cortex, 42, 264–280.

71.

Yau

J. M.

Olenczak

J. B.

Dammann

J. F.

Bensmaia

S. J.

(2009). Temporal frequency channels are linked across audition and touch. Current Biology, 19, 561–566.

72.

Zellner

Geller

Lyons

Pyper

Riaz

(2017). Ethnic congruence of music and food affects food selection but not liking. Food Quality & Preference, 56, 126–129.

73.

Zhu

Zhang

Ding

Zhou

(2009). Crossmodal effects of Guqin and piano music on selective attention: An event-related potential study. Neuroscience Letters, 466, 21–26.

74.

Zhu

Zhang

Liu

Ding

Zhou

(2008). Differential cognitive responses to guqin music and piano music in Chinese subjects: An event-related potential study. Neuroscience Bulletin, 24, 21–28.