Abstract
Crossmodal correspondences between seemingly independent sensory modalities are often observed in normal participants. For instance, colours commonly map consistently onto pure tones. In this study, we investigated colour-tone mapping in both normal trichromats and in people with congenital blindness. Participants were asked to match tones of differing pitch to named colours. In both cases, the tones selected varied consistently with the colour. The blind responses were similar to those of the trichromats, except in the case of red and green; the blind did not differentiate these colours, whereas the trichromats associated red with a higher tone and green with a lower tone. Otherwise, the results are consistent with a well-established association between pitch and lightness, with lighter colours associated with higher tones, and darker colours with lower tones. Because the blind never had any sensory experience of colour, their matching of colour to pitch is most likely based on semantic understanding.
Introduction
The mapping of one stimulus, whether physically present or imagined, onto another stimulus is known as crossmodal correspondence, this means that a sensory feature in one modality is associated with a sensory feature in another modality without necessarily evoking a conscious sensory experience (Spence & Deroy, 2013; Spence & Parise, 2012). For instance, it has been shown that people consistently match high-pitched sounds with small and bright entities (Spence, 2011). It is also common for people to map low tones onto darker colours and high tones onto lighter ones, and this need not depend on presentation of physical stimuli (de Thornley Head, 2006; Ward, Huckstep, & Tsakanikos, 2006).
People who are congenitally blind, and who thus have no experience of light or colour, are nonetheless able to develop a sophisticated understanding of colour that, at least in some instances, resembles that of people with normal, trichromatic colour vision (e.g., Marmor, 1978; Saysani, Corballis, & Corballis, 2018). Even when people with congenital blindness do not fully understand colour, multidimensional scaling studies have produced “colour spaces” that separate warm from cool colours (Shepard and Cooper, 1992), suggesting at least a degree of understanding of colour. This raises the question of how—or whether—the congenitally blind, who have never experienced sensory colours at all, map colour terms onto tones? How would their mapping compare to that from trichromatic observers? This study seeks to address these questions and provides some possible explanations in the context of crossmodal correspondences and semantics.
Methods
Participants
A total of 33 people participated in this study. Of these, 20 had normal trichromatic vision (10 females and 10 males; M age = 29, range = 21–35), while the remaining 13 were congenitally blind (7 females and 6 males; M age = 40.4, range = 18–69). 1 No hearing impairments, cases of synaesthesia, chromaesthesia, including hereditary synaesthesia were reported. While the aetiology of blindness varied, all of the blind participants were completely blind and had no residual experience of vision. The causes included: leber congenital amaurosis, septo-optic dysplasia, retinal detachment, retrolental fibroplasia, and other congenital malformations.
Stimuli
Visual stimuli consisted of nine basic colour terms: red, orange, yellow, green, blue, brown, purple, black, and white. Each colour term was presented to participants verbally and in a randomised order.
Auditory stimuli consisted of seven individual pure tones: 100 Hz, 200 Hz, 300 Hz, 400 Hz, 500 Hz, 600 Hz, and 700 Hz. All tones were of equal amplitudes. Individual tones were presented to participants in a randomised order, through two external computer speakers.
Procedure
Each participant was first told a colour and then played a pure tone through two external computer speakers. On a computer keyboard, using the left and right arrow keys, the participant was asked to increase (left key) or decrease (right key) the level of Hertz, which varied each time by ±100 Hz, until they were satisfied that a tone corresponds with the target colour. Participants were presented with each colour only once and were at liberty to assign the same tone to more than one colour.
Results
A two-way analysis of variance (ANOVA) 2 was carried out with colour as a within-subjects variable (nine levels: colours) and group as between-subjects variable (trichromats and congenitally blind). The analysis revealed that there was a significant main effect of colour on the tone scale, F(5,157) = 47.05, p < .001, and of group, F(1,31) = 12.15, p < .001. The interaction between colour and group was also significant, F(5,157) = 8.22, p = <.001, indicating that the difference between the groups depends on how particular colours are associated to different tones. Bonferroni-corrected comparisons between groups for each colour showed that the blind differed from trichromats in that they associated significantly lower tones with red and green (p < .001), as shown in Figure 1, but apart from this the groups’ plots largely overlap.
Two audio-visual profiles based on groups’ average scores associating nine basic colour terms with pure tones ranging from 100 Hz to 700 Hz. Error bars represent 95% confidence intervals.
Given the significant interaction between colour and group we followed with a one-way ANOVA across colours for each group separately. The analysis revealed a significant main effect of colour for both the blind F(3.9,46.8) = 12.04, p < .001 and the trichromatic group F(3.4,64.3) = 59.54, p < .001. This showed that, despite the interaction, both groups did associate different colours with different tones.
Discussion and Summary
Audio-visual crossmodal correspondence was explored in samples of congenitally blind and trichromatic participants. The task involved mapping between basic colour terms and pure tones ranging from 100 Hz to 700 Hz. Despite having never had visual experience of colour, and with the exception of red and green, blind participants associated colours with tones similarly to their trichromatic counterparts (Figure 1). For both groups, the mapping suggests that a “lightness scale” may be an underlying factor in how colours are ordered; lighter colours are associated with higher tones and darker colours with lower tones, which seems to be a consistent and longstanding association (Marks, 1975). Thus, for both groups the light colours yellow and white are associated with high tones, and the dark colours blue, brown, purple, and black with low tones. For the blind, however, red and green did not seem to be differentiated, perhaps because these colours had no semantic associations with lightness, whereas for the trichromats red and green were clearly separated, presumably because they were able to draw on experiential associations unavailable to the blind. Notably, there were fewer tones than colours in this study, which may have artificially forced red and green to be mapped similarly. A future extension of this research could include equal number of colours and tones, offering the possibility of a unique mapping for each pair of stimuli. Similarly, increasing the number of repetitions of each colour would be useful in reducing the possibility of hysteresis or order effects and in determining the within-subjects consistency of the mappings between colours and tones.
These data reveal a surprising crossmodal link between colour and sound, even in people who have no direct experience of colour. One factor which has been consistently shown to influence multisensory integration is semantic congruency; in which pairs of visual and auditory stimuli either “match” or “mismatch” in regards to their identity or meaning (Chen & Spence, 2010; Spence, 2011). The mapping of colours onto tones by the blind shows that crossmodal correspondences can occur even in circumstances when a relative sense never existed. While it is conceivable that this correspondence has a basis in overlapping neural mechanisms, as in chromesthesia (which has been reported in the adventitiously blind; Niccolai, van Leeuwen, Blakemore, & Stoerig, 2012; Steven & Blakemore, 2004), it is perhaps more likely that it originates in experience with language—in particular semantics and metaphor. For the congenitally blind, “colour” necessarily begins as a semantic concept, which may then begin to develop associations with other perceptual, cognitive, and semantic objects as a result of experience. To obtain a deeper understanding of the underlying mechanisms related to colour-sound associations, future studies could be conducted with congenitally blind people from different cultural or linguistic backgrounds, or with blind children, who presumably have yet to develop a strong semantic understanding of colour.
Footnotes
Acknowledgments
I would like to express my gratitude to Professor Michael C. Corballis and Associate Professor Paul M. Corballis for their guidance and constructive feedback on earlier versions of this manuscript. Any faults or miscalculations are of my own and should not reflect on the supervision of these esteemed persons.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.