Coloured hearing,colour music,colour organs,and the search for perceptually meaningful correspondences between colour and sound

1.1. Metathetic, Prothetic, and Circular Dimensions

More than half a century ago now, Stevens (1957) highlighted an important distinction between ‘metathetic’ and ‘prothetic’ stimulus dimensions. Loudness, brightness, lightness, heaviness, duration, roughness, area, and apparent length etc., are all prothetic dimensions, with a clear ‘more than’ and ‘less than’ end. ² Stevens refers to these as quantitative perceptual continua (i.e., having to do with how much) with ratio properties. According to Panek and Stevens (1966), the saturation of red can also be arranged as a prothetic continuum. Metathetic dimensions, by contrast, obey a well-structured organization without necessarily having a ‘more than’ or ‘less than’ end. In adults, prothetic dimensions tend to possess a unitary and well-ordered psychophysics, whereas metathetic stimulus dimensions do not. Stevens classified the latter perceptual continua as ‘what kind’ or ‘where (position)’. Relevant here, both auditory pitch and hue are metathetic dimensions, as are visual position, inclination, and proportion, according to Stevens. Smith and Sera (1992) subsequently added shape to the list. Interestingly, however, pitch and hue can also both be represented as circular dimensions. In fact, according to Pridmore (1992), they are the only perceptual dimensions that can be so represented (see Table 1 for a summary).

Polar dimensions should also be considered here. According to Clark (1970), the latter are fundamental to children’s development of relational concepts. Dimensions such as big vs. little, high vs. low, but presumably also left vs. right, sweet vs. sour/bitter are all polar. Notice here how a polar organization can be applied to prothetic dimensions (big vs. little; spatial high vs. low), to metathetic dimensions (such as left vs. right; high vs. low pitch), and to pairs of stimuli that cannot be organized as part of either prothetic or metathetic dimensions, such as between pairs of basic tastes (though see Crisinel et al., 2012; and Watson & Gunter, 2017) or olfactory stimuli. Polar dimensions can presumably be aligned with other polar dimensions although which poles people choose to align may well sometimes turn out to be arbitrary. Intriguingly, certain of the basic tastes have recently been shown to correspond to the polar dimension of elevation (e.g., high vs. low; see Velasco et al., 2019).

While it is undoubtedly possible to establish an analogy between pitch and hue at more of a cognitive, or abstract, level, and by so doing relate, or figure out, the relative position of auditory and visual stimuli along their respective unisensory dimensions (cf. Cohen, 1934; Mellers & Birnbaum, 1982; Moul, 1930; Simpson et al., 1956, p. 100), ³ there is little empirical evidence supporting the existence of any kind of perceptual ‘resonance’ (cf. Muecke & Zach, 2007), or unification ⁴ that necessarily follows from the correct structural alignment of stimulus dimensions (e.g., Sebba, 1991). That is, the dimensional alignment of pitch and hue does not necessarily lead to any perceptually apparent correspondence based on the perceived similarity of the component stimuli. ⁵

Intriguingly, von Hornbostel (1931, 1950) was convinced that sensory brightness represented one of the universal dimensions of sensory experience. His participants were required to match sounds of different pitches to points along a greyscale. He also had them crossmodally match scents with grayscale values. He took his results, and, in particular, the transitivity between different crossmodal comparisons that they apparently revealed, to demonstrate that the concept or quality of ‘sensory brightness’ was one that was common to all of the senses. Were this to be the case, one could easily imagine how simply matching the brightness of auditory and visual stimuli would provide a means of objectively aligning the senses. However, subsequent researchers were not convinced (see Cohen, 1934), arguing instead for a relative/relational judgment rather than necessarily a crossmodal perceptual mapping (see also Hartshorne, 1934). As Cohen (1934, p. 119) put it, the stimuli were ‘analogous’ rather than ‘identical’. Cohen explains as follows: “It would not be unreasonable then to suppose that cross-modality comparison should be based (physiologically, if not introspectively) upon relative positions within different ‘absolute’ scales. According to this view equation with respect to brightness of two experiences of different modalities would involve nothing more than the identity of relative positions upon two wholly independent scales.”

Echoing von Hornbostel’s theoretical speculation, Smith (1987, p. 96) writes that: “we have a lot to gain if we can get beyond mapping specific dimensions one to another and instead delineate the amodal dimensions.” The possibility for amodal concept(s) to exist is apparently linked to the existence of absolute correspondences, as Smith (1987, pp. 97–98) observed: “This suggestion of a trend from dichotomous, categorical treatments of continua to more relativistic ones ought not to be confused with the issue of absolute versus relative correspondences across dimensions. The notion of absolute correspondences between dimensions is that particular values on one dimension map onto particular values on another—for example, higher is not like brighter; rather, a specific pitch matches a specific brightness. As Marks et al. point out, there is little evidence for such absolute correspondences.” (Italics in original). Interestingly, Mellers and Birnbaum (1982, p. 600) came to much the same conclusion, namely that: “In cross-modality judgments, the scale values are influenced by the stimulus distribution: It appears that subjects compare the relative position of a stimulus in its distribution with the relative position of a stimulus of another modality to its distribution, going on to suggest that their results were consistent with a psychological relativity theory of cross-modality judgment.”

To summarize, beyond shedding light on the way in which stimuli are organized within perceptual dimensions in different sensory modalities, the distinction between metathetic and prothetic stimulus dimensions leads to the related distinction between absolute vs. relative crossmodal correspondences. The latter distinction is, in turn, instrumental when it comes to assuming the existence of an amodal concept, conceived of as the same physical property (such as shape) being picked up via multiple senses (see Lewkowicz & Turkewitz, 1980). This, it should be noted, is subtly different from von Hornbostel’s (1931, 1950) notion of universal dimensions of perceptual experience. The emphasis in the latter case would appear to be on the perceptual experience itself (i.e., what it is like), whereas the emphasis for those who have proposed amodal dimensions is on the multiple routes to picking-up information about physical properties out there, regardless of the perceptual qualities that may be associated with that information.

Researchers have long known that people can scale sensations in different sensory modalities across a wide range of prothetic stimulus dimensions (Stevens, 1966; see also Bond & Stevens, 1969). And while some have wanted to argue from such matching behaviours that there may be shared attributes of experience, such as sensory brightness in von Hornbostel’s (1931) case, it turns out that it is hard to rule out alternative, more cognitive, accounts of the underlying process, as we have just seen (see Cohen, 1934). That said, Marks et al. (1987, p. 84) conclude that: “In some fundamental sense, the similarities between pitch and brightness and between loudness and brightness are personal, internal, and subjective; they reside in perception per se and probably depend on common processes of neural coding.” Note the strong claim here, albeit with multiple provisos, that similarity relations are perceptual in nature. Nevertheless, the ‘personal, internal, and subjective’ element did not stop Marks (1974a, b) from trying to establish a robust psychophysics based on the crossmodal matching of the colour lightness of grey surfaces with the pitch of pure tones. ⁶ However, the ability to crossmodally match stimuli is presumably possible between any pair of prothetic stimulus dimensions, only a few of which might be argued to pick-up on the same stimulus, or perceptual, property (cf. Cohen, 1934). ⁷ Elsewhere, Marks et al. (1987, p. 5) talk of the “perceptual, cross-modal equivalence with respect to intensity”. As such, it may be difficult to assert anything concrete about the perceptual similarity of pairs of stimuli based merely on the fact that participants can engage in robust crossmodal matching of stimuli relative to the dimensions to which they belong. In contrast to Marks and colleagues’ suggestion that the similarities are fundamentally perceptual in nature, we will argue for an emotional mediation account of colour-sound correspondences instead (cf. Wilms & Oberfeld, 2018).

Here it may also be worthwhile to contrast hue and pitch with other dimensions of unisensory experience, namely basic tastes and odour qualities which can neither be organized prothetically nor metathetically. Notice how the latter can both be distinguished from one another (i.e., sweet can be distinguished from salty and sour etc.) without there necessarily being any obvious organizational structure linking the various stimuli within each category (i.e., tastes and odours, respectively). ⁸ The organization of smell and taste qualities are, in other words, neither prothetic nor metathetic. Nevertheless, research on the crossmodal correspondences now clearly reveals the existence of robust associations between colour hue categories (such as red, white, etc.) and basic taste qualities, such as sweet, bitter, salty, sour, salty, and umami (Spence & Levitan, 2021; Spence et al., 2015; see also Ikeda, 2002), as well as with odour qualities (see Spence, 2020b, for a review). ⁹ Crossmodal correspondences have also been documented between auditory pitch and both basic taste qualities (e.g., Knöferle et al., 2015), and various olfactory stimuli (e.g., Belkin et al., 1997; Crisinel & Spence, 2012; see Di Stefano et al., 2022; Spence, 2021, for reviews).

Given the latter observation, it would appear that the nature, or structure, of the underlying stimulus/perceptual dimension (see Table 1) does not necessarily constrain the likelihood of observing, or establishing, crossmodal correspondences, nor does it automatically connect to the notion of perceptual similarity. Taken together, therefore, the research that has been reviewed in this section would appear to support the claim that the strength and quality of crossmodal correspondences, such as, for example, between pitch and hue, does not necessarily bear any meaningful relation to the structure, or organization, of the underlying perceptual continua. This conclusion obviously runs counter to the intuitions of those who first commented on the connection between colours and musical sounds. ¹⁰ Indeed, it should be noted that the exact correspondence between these two classes of continua and their underlying neural representations is by no means clear (Lewkowicz & Turkewitz, 1980; Stevens, 1971).

It is worth noting how the conclusion that stimulus dimensional structure has no implications for crossmodal perceptual similarity would appear to contradict the claims made by the psychologist Lawrence Marks in his influential book, The unity of the senses, when he asserts that: “Sensory correspondence is not a domain of inquiry restricted to scientists, a matter solely for experimental scrutiny and empirically based theory. The plain fact is that sensory analogies do exist; they are important to the ways that we sense, perceive, and cognize; they are significant properties of the bodies and minds of people” (Marks, 1978, p. 7). In contrast, according to the view espoused here, while ‘sensory analogies’ may bias (or be based on) the kinds of sensory connections that people find it easy to establish cognitively, we do not believe that they play any role in determining the nature, or strength, of the crossmodal correspondences that are based on perceived similarity. However, while apparently contradicting Marks’ view, our own conclusion would appear to be much more in tune with the art historian Gombrich’s suggestion that we should focus our attention on the structural relationships in the system rather than on the similarity of the elements: “The problem of synesthetic equivalences will cease to look embarrassingly arbitrary and subjective if . . . we fix our attention not on likeness of elements but on structural relationships within a scale or matrix” (Gombrich, 1960, p. 314).

1.3. On the Multiplicity/Hierarchy of Crossmodal Correspondences

One other important point to bear in mind here is that, outside of the psychophysics laboratory, perceptual stimuli typically vary along several dimensions simultaneously (e.g., two visual stimuli may well differ in terms of their hue, but also in terms of their size, shape, texture, etc.). There is presumably a hierarchy of crossmodal correspondences, such that certain perceptual dimensions (or attributes) may only obviously be aligned if the more natural, or perceptually salient, matching dimensions (or attributes) are not available (cf. Gardner, 1974; Parise, 2016). That is, the participants in laboratory studies will presumably simply go with the best of the available response options when, say, in an experimental setting, they are instructed to match a colour to a musical sound. In the audiovisual context, note only how auditory pitch, a correlate to visual brightness, is overall at least as strong as the crossmodal correspondence with loudness. In fact, according to Marks (1989, p. 598), it may even be stronger, or more dominant. In other words, one can ask both what is the spontaneously chosen dimension corresponding to a given perceptual experience, and whether it is consensually chosen across individuals. Separately, for a given specific pairing of attributes or dimensions one can ask how consistent it is across individuals (see Spence, submitted). It is interesting to note how the former question has rarely been posed by researchers, with the latter normally assuming what the comparison dimension should be.

As such, merely demonstrating a statistically significant crossmodal correspondence between stimuli in two arbitrary perceptual dimensions (such as pitch and hue) does not necessarily mean that the correspondence so documented reflects the best of all possible matches with either of the component attributes, merely that it is the best of the options that happened to be available to participants at the time they were asked (see Spence & Levitan, 2021; Spence et al., 2015, on this point). Thus, the mere fact that a robust crossmodal correspondence can be established between two stimuli, or stimulus dimensions, does not mean that this will necessarily be the most natural, or obvious, association that an observer will make, or be drawn to, spontaneously should there be a better-connected dimension available (cf. O’Mahony, 1983, on the multiple correspondences with basic tastes, of which hue is but one). Notice here also how the frequency, or consensuality, with which specific crossmodal associations occur is not necessarily linked to the strength, or vividness, of the association itself (e.g., Marks, 1975; Rader & Tellegen, 1987; Spence, submitted). Or, to give another example, consider only how people may draw an analogy between spatial patterns in vision and temporal patterns in audition if those happen to be the only dimensions in which the auditory and visual stimuli differ/vary (Julesz & Hirsh, 1972). However, as soon as a temporal pattern is presented visually then this may come to dominate as the more natural, or intuitive, crossmodal match for an auditory temporal pattern than a spatial pattern.

1.4. On the Popularity of Colour-Sound Correspondences

So why, then, is it that people have been so interested in colour-pitch correspondences for so long? The topic is undoubtedly of theoretical interest in terms of the discussion of crossmodal perceptual similarity, with early interest seemingly stemming from the various structural and physical similarities that have been highlighted between the dimensions of stimuli/experience, though as we will see later (see Section 4), interest was undoubtedly revived as a result of parallel developments in the emerging artistic field of colour-music (e.g., Klein [Cornwall-Clyne], 1937; see Zilczer, 1987, p. 2016, for reviews). It is, though, worth stressing how many of the artists working in the field of colour-music (not to mention many of the prominent research scientists working in the area) have, over the years, been distracted by the search for crossmodal mappings between musical features and colour (and/or form) in the idiosyncratic experiences reported by synaesthetes (see Donnell-Kotrozo, 1978; Galeyev, 1976, 2003; Itoh et al., 2017; Kandinsky, 1977; von Erhardt-Siebold, 1932; Zilczer, 1987). ¹¹ Indeed, the florid concurrents that have so often been experienced, and reported, by coloured-hearing synaesthetes have undoubtedly helped to raise awareness/interest in crossmodal mappings between this particular pair of sensory attributes (Hänggi et al., 2008; Menouti et al., 2015). In recent decades, the growing interest in sensory substitution systems/devices (e.g., for the blind) has also drawn attention to the question of how best to ‘translate’ visual attributes, such as colour, into sound in an intuitively meaningful manner (see Hamilton-Fletcher et al., 2016; Marks, 1983; see also Lupton, 2018).

1.5. Coloured Hearing Synaesthesia

While some commentators have wanted to align the visible colour spectrum with the auditory pitch scale (Newton, 1704; see also Arnheim, 1986), others have focused instead on specific matches between particular pairs of stimuli, often based on the idiosyncratic inducer-concurrent relations experienced by those synaesthetes who report experiencing coloured music (see Suarez de Mendoza, 1890). For example, just take the Bauhaus/Abstract artist Kandinsky’s (1977) suggestion that the sound of the trumpet is scarlet (see Ione & Tyler, 2003, 2004; Just, 2017; though see also O’Regan, 2011). At one point, Kandinsky (1977, p. 40) writes that: “Light warm red …In music, it is a sound of trumpets, strong, harsh, and ringing.” In this particular case, one can only wonder whether it is mere coincidence that a couple of centuries earlier both Locke (1690) and Leibniz (1704/1896) had written about the blind man who apparently understands what scarlet is because of its being compared to the sound of the trumpet. Such consistency might be taken to suggest the existence of a meaningful (i.e., fundamental) mapping between a specific timbre and a particular hue category (see also Lavignac, 1899; Wallmark, 2019). Other commentators, though, have wanted to put forward an account in terms of learned associations instead. Just take, for example, the eminent neurologist MacDonald Critchley (1900-1997), who once apparently suggested that “the familiar story of trumpet blasts provoking a photism of red, may stem from the fact that such a sound immediately calls up in some persons an imagery of soldiers on parade. Ordinarily they shall be in dress uniform. This evokes a mental picture of scarlet. Should the middle part of this notion eventually become submerged, there will remain a synaesthetic linkage of trumpet-calls with redness.” (as quoted in Harrison, 2001, p. 209; cf. Barilari et al., 2018, for the suggestion that crossmodal correspondences involving visual stimuli may be internalized by the early blind in terms of the structural regularities in language). It is perhaps also worth considering the effect of key colour on the recognition of absolute pitch on the piano (Marvin & Brinkman, 2000).

Kandinsky (1977) referred to a number of specific colour-sound mappings in his writing, though it is often unclear whether the examples he gives constitute examples of the artist’s own synaesthesia, or should be better considered as examples of emotionally-mediated crossmodal correspondences (and hence might perhaps be expected to be experienced by us all; Spence, 2020a). Kandinsky often refers to the latter in terms of ‘inner harmony’ (cf. Harrison, 2001). Something of a similar challenge has faced those interested in trying to understand more about the idiosyncratic crossmodal mappings that have been suggested by synaesthetic Russian artists, namely the composers Rimsky-Korsakov (who reported ‘seeing’ music in the key of A-major as yellow; Myers, 1911), and Scriabin (Galeyev & Vanechkina, 2001; Myers, 1914). Once again, though, it has long been the subject of debate as to what exactly the relationship, if any, was between Scriabin’s personal repertoire of idiosyncratic audiovisual inducer-concurrent mappings, and those that he chose to incorporate into his colour circle/score/luce (Galeyev & Vanechkina, 2001; Triarhou, 2016). ¹²

‘Coloured hearing’ turns out to be one of the most commonly-mentioned forms of synaesthesia, and often appeared in the scientific literature in the decades either side of 1900 (e.g., Dauriac, 1902; de Parville, 1883; English, 1923; Flournoy, 1893; Ginsberg, 1923; Jewanski et al., 2009; Jewanski et al., 2011; Jewanski, Simner, Day, Rothen, & Ward, 2020; Underwood, 1893). ¹³ Marks (1975) provides a particularly thorough review of the multitude of early studies of coloured music and coloured speech sounds. ¹⁴ Interestingly, both pitch and timbre appear to be salient features (i.e., sensory inducers) driving the coloured musical concurrents that have been reported to date. At the same time, however, it has also been acknowledged that there may be a strong visual mental imagery component to many coloured responses to music (Karwoski et al., 1942; Mudge, 1920; see also Spence & Deroy, 2013). And while it has been suggested that the inducer-concurrent mappings experienced by synaesthetes do typically tend to be appreciated by non-synaesthetes (e.g., see Ward et al., 2008), that presumably has no necessary implications regarding the question of whether the inducer is perceptually (or necessarily even affectively) similar to the concurrent in the case of synaesthesia.

3.1. Colour-Pitch Correspondences

During the middle decades of the twentieth century, a number of experimental psychologists, often without any reference to the earlier studies (such as those mentioned above), set themselves the task of trying to establish the nature (and consequences) of any crossmodal correspondences that might exist between colour and pitch. According to Kubovy and Van Valkenburg (2001, p. 114), the justification for this line of empirical research was simply that: “early psychologists concluded that colour gets mapped onto pitch, since they are non-spatial and non-temporal, and since they are both caused by waves.” Notice here how the justification is in terms of the physical nature of the stimuli themselves.

In terms of empirical research, one of the first studies to assess colour-tone associations systematically in a non-synaesthetic population was reported by Simpson et al. (1956). These researchers investigated the nature of any crossmodal associations between colours and pure tones in 995 elementary school children (between third and sixth grade). The children were presented with each of six pure tones (125, 250, 1,000, 4,000, 8,000, and 12,000 Hz, or cycles per second, c.p.s.) ¹⁸ at 40 and 50 dB and forced to choose which of six spectral colours (violet, blue, green, yellow, orange, and red) they ‘thought of’ immediately upon hearing each of the tones. Yellow and green were predominantly associated with high pitch. They suggested red and orange could be categorized as ‘middle-pitched’ colours, while violet and blue were predominantly ‘low-pitched’ colours. ¹⁹ Here, though, it is worth mentioning the point made earlier concerning the fact that the participants were only ever able to choose the best of the six available options. Hence, were it to be the case that pink were to have been the optimal match to one of the pure tones, say, then this would not be apparent from the data.

Simpson et al.’s (1956) study was phrased as a game with the children themselves told to guess if they were unsure. It is therefore unclear whether the mappings that were captured were cognitive in origin or anything more (i.e., perceptual crossmodal correspondences). And while Simpson and colleagues claimed to have demonstrated crossmodal pitch-hue correspondences (or ‘synaesthesia’ in their misleading terminology), ²⁰ several researchers have subsequently questioned whether these researchers actually controlled the brightness and/or saturation of their stimuli (cf. Wicker, 1968). In the absence of certainty on this latter point, the possibility must remain that the crossmodal correspondences reported in this early study might reflect nothing more than a pitch-brightness correspondence (see Bleuler & Lehmann, 1881; Flournoy, 1893; Marks, 1982; and Riggs & Karwoski, 1934) rather than necessarily having anything to do with hue categories per se (Spence, 2011). Certainly, when other experimenters have subsequently looked for correspondences between hue and pitch, it is striking how many of them have failed in their efforts (e.g., Bernstein et al., 1971; Wicker, 1968) (see Table 3 for a summary).

A couple of the studies that have been reported by Marks and his colleagues are interesting inasmuch as they appear to highlight the existence of a crossmodal correspondence between colour words and the associated, or corresponding, pitch in adults (Marks, 1982) (see Figure 3). A few years later, Marks et al. (1987) went on to demonstrate much the same crossmodal associations in children aged between 9 and 13 years (see Marks et al., 1987, Figure 47). Such results presumably suggest that people may associate different (prototypical) or imagined colours as having a specific brightness level, and moreover that they do so from early in development. One final point to note here concerns Simpson et al.’s (1956) suggestion that: “A third interpretation, most consistent with our finding of selective pitch-colour associations in children, is that particular hues and pure tone frequencies are associated with each other because of a certain inherent “belongingness” between each member of a given pair and a particular mood.” (Simpson et al., 1956, p. 102). As we will see later, this mood, affect, or emotion-based account of crossmodal mappings is one that many other scientists, as well as a number of artists have seemingly also stumbled across (cf. Kandinsky’s, 1977, ideas around the notion of ‘inner harmony’).

OPEN IN VIEWER

Figure 3.

Mean ratings of pitch versus mean ratings of brightness to colour words. (Open symbols: results of Marks (1982), Experiment 2; visual scale, dark-bright. Filled symbols: results of Marks (1982), Experiment 4; visual scale, dim-bright.) [Reprinted with permission from Marks (1982, Figure 6).].

Wicker (1968) conducted a pair of studies investigating what he referred to as the “intersensory dimensions in perceptual or cognitive space, i.e. of dimensions which are significantly descriptive of sensory inputs from more than one modality.” [italics in original] (Wicker, 1968, p. 178). Wicker was particularly interested in “behaviour which can be conceptualized as reflecting perceptual or associative alignment between sensory attributes from different sense-modalities” (Wicker, 1968, p. 178). In a first experiment, the participants were presented with a range of 13 pure tones (300, 400, 500, 600, or 700 c.p.s.; this, note, a more sensible choice of tones than used in Simpson et al.’s, 1956, study, given that the researchers were potentially able to investigate also another variable/dimension (such as different notes) beyond higher-lower frequency…) of varying loudness (53-84 dB); they were also presented with 13 coloured Munsell colour squares (green, red, blue, and yellow) of varying brightness and saturation. The participants were instructed to rate the similarity of all pairs of tones and, in a separate experimental session, to rate the similarity of all possible pairs of colour patches. They also had to rate every individual tone and colour patch in terms of 25 semantic differential adjective scales (cf. Moller et al., 2009). They were further encouraged to rate the similarity of the auditory and visual stimuli in another part of the study. In particular, according to Wicker (1968, p. 180): “They were asked to rate the similarity of every tone to every colour on a nine-point scale.”

Multivariate scaling revealed two orthogonal alignments underlying the intersensory and cognitive space: pitch-brightness and loudness-contrast. These were established using multidimensional-scaling, semantic-differential scaling, and an intersensory transfer of training paradigm. At the same time, however, Wicker (1968) also noted how several other ‘alignments’, what in today’s parlance might well be called crossmodal correspondences (Spence, 2011), were not evidenced by his analysis of the data, namely pitch-saturation, loudness-brightness, and loudness-darkness. The failure to find the systematic relationship between pitch and hue was especially surprising given the results of Simpson et al.’s (1956) previous developmental study. Taken together, the studies reported by Simpson et al. and Wicker would therefore seem to suggest that colour-sound mapping is likely based on frequency (in terms of low-high continuum) but ignores octave repetition. In fact, if the mappings were to be based on octave repetition, then most experimental stimuli would have been matched to the same colour (as they are all octaves apart).

A few years later, Bernstein et al. (1971) conducted a small study in which four participants made speeded discrimination responses to the colour (red vs. blue) of a light which was either presented by itself, or else was accompanied by a task-irrelevant tone (of either 100 or 1,000 Hz). However, no significant effect of crossmodal mapping was observed as a function of the colour-pitch mapping. ²¹ As outlined by Bernstein and colleagues, the particular motivation for undertaking their research was to test the hypothesis that: “a high frequency tone would have greater facilitatory effect for a stimulus from the high (short wavelength) end of the spectrum and a low tone would have greater facilitatory effect for a stimulus from the low (long wavelength) end of the spectrum.” (Bernstein et al., 1971, p. 1327). However, the tiny sample size means that it is unclear whether Bernstein et al.’s study was adequately powered to find an effect in the first place.

Finally in this section, Melara’s (1989) studies on dimensional interactions between colour and pitch should also be mentioned. As a cognitive psychologist, Melara was particularly interested in studying dimensional interactions between colour and pitch using the well-established Garner interference paradigm. Indeed, there had been much research interest in the preceding decades concerning Garner’s notion that there was a meaningful distinction between separable and integral stimulus dimensions. The participants in Melara’s studies had to make speeded classification responses to either black and white stimuli or to triangle waveform tones having a fundamental frequency of either 1046.5 Hz (high) or 174.6 Hz (low). That is, the participants made speeded classification responses to a sequence of stimuli presented in one sensory modality while attempting to ignore the task-irrelevant stimuli that were sometimes presented in the other modality. The results of a series of experiments suggested that the crossmodal connection between these achromatic colours (what might be described as brightness) and pitch was partly strategic and partly mandatory (cf. Sun et al., 2018). It is, though, important to bear in mind here that evidencing congruency effects in speeded classification tasks such as these does not, in-and-of-itself, necessarily tell us anything about the perceptual similarity of the stimuli that were used. That is, congruency effects in behavioural studies have multiple causes, only a minority of which are likely to be perceptual in origin.

One other point to note here, in passing, is the very different pitch ranges that have been used in the studies reported above. That is, while the label ‘high-pitched’ might be associated with a pure tone of 1,000Hz in one study (e.g., Bernstein et al., 1971) the same label has been associated with a 12,000Hz tone in another (Simpson et al., 1956). The assumption amongst many of the experimental psychologists would therefore implicitly seem to have been it is the relative position of stimuli along their respective continua that matters (cf. Spence, 2019, for a review). But, one might ask, is that necessarily the case? ²² What is also worth noting is that most (if not all) of the above-mentioned studies used pure tones, which are not properly musical sounds, as they lack overtones (and therefore timbre). Since fundamental aspects of frequency perception (e.g., discrimination) are shown to be facilitated (in healthy participants) with complex or harmonic tones rather than pure tones (e.g., Novitski et al., 2004; Zeitlin, 1964), this would make the choice of pure tones even more problematic, for those wanting to focus on audiovisual correspondences based specifically on frequency. It is relevant here to note that with his ‘Gamut of Odors’ (see Figure 4), Septimus Piesse has also been taken to have highlighted a direct one-to-one correspondence between musical notes and scents. Relative correspondences presumably suggest a belief in some kind of structural analogy rather than a direct perceptual mapping based on the phenomenological similarity of the component stimuli.

OPEN IN VIEWER

Figure 4.

Scale of crossmodal correspondences between sound and odours reproduced from Piesse (1867, pp. 42–43).

Taken together, the experimental research that has been conducted to date to test the psychological correspondence between colour (hue) and pitch has failed to reveal any convincing evidence of such a mapping (or alignment of dimensions), at least not when brightness and saturation have been carefully controlled (cf. Dailey et al., 1997; Hubbard, 1996). Given these null results, researchers would subsequently appear to have been dissuaded from further searching for hue-pitch correspondences. Indeed, as Kubovy and Van Valkenburg (2001, p. 115) put it: “As profound differences between light and sound became clear in the twentieth century, psychologists abandoned the explorations of parallels between pitch and colour.” Instead, it would appear from the psychophysical research reviewed here (see Wicker, 1968) that the more natural correspondence may instead actually be between pitch and brightness (cf. Bleuler & Lehmann, 1881; Flournoy, 1893; Marks, 1982) and between loudness and contrast. ²³

3.3. Pitch-Brightness Correspondences

Crossmodal correspondences between auditory pitch and visual brightness/lightness have often been reported in the psychology literature (Marks, 1987; see also Sabaneev & Pring, 1929; see Table 4). What is more, according to Marks (1978, 2011; Marks & Bornstein, 1987), a neurophysiological explanation for this particular mapping may be based on the existence of common, underlying sensorineural codes (of intensity, brightness, duration, etc.). So, for example, Marks points to the fact that the auditory system appears to use temporal patterning (neural response frequency) in the coding of both pitch and loudness, ²⁴ and believes that this may help to explain the commonalities between both of these dimensions and visual brightness (e.g., see Marks, 1991, p. 193). This suggestion, which echoes an earlier one made by Stevens (1957) regarding the common coding of increased intensity in different sensory modalities in terms of increased neural firing, can be seen as an account of at least certain correspondences motivated on the basis of a similarity in the neural processing/encoding principles (cf. Ellermeier et al., 2021; Walsh, 2003). Spence (2011) labelled those correspondences that are putatively based on such neural similarities as ‘structural correspondences’. ²⁵ Nevertheless, despite the absence of robust empirical evidence supporting the existence of pitch-hue correspondences, a number of other crossmodal correspondences between dimensions of colour and sound have been documented over the last decade or so, including between timbre and hue (e.g., Adeli et al., 2014). For instance, Hamilton-Fletcher et al. (2017) examined how sound influences chroma or hue when properly controlling for lightness. To address this question, they had their participants physically adjust equiluminant colours until they matched with certain sounds. They found that, for pure tones, increases in frequency were associated with increases in chroma. Increasing the loudness of pure tones also increased chroma. Intriguingly, for complex sounds that share the same bandwidth of frequencies (100-3,200 Hz), but that vary in terms of which frequencies have the most power, all of the stimuli were associated with yellow hues. Based on their results, the authors suggest that the presence of frequencies higher than a certain threshold (i.e., above 800 Hz) consistently yields yellow hues.

Anikin and Johansson (2019) reported on a series of 22 experiments in which they investigated crossmodal correspondences between visual (luminance, hue [R-G, B-Y], saturation) and acoustic dimensions (loudness, pitch, amongst various others). Their results revealed that loudness is associated with saturation, while pitch is associated with both luminance and saturation.

As well as investigating any pitch-hue correspondences, experimental psychologists have, over the years, also assessed the nature and consistency of any timbre-colour (hue) crossmodal correspondences. In the latter case, though, the motivation behind researchers’ attempts to demonstrate such correspondences would appear to have been driven more by, or at least related to, the particularities of the idiosyncratic reports of coloured music synaesthetes rather than anything else (see Donnell-Kotrozo, 1978). For example, the composer Raff reported that he perceived the colour of the sound of the trumpet to be scarlet (other people apparently report it to be bright red; Ortmann, 1933); for Kandinsky, meanwhile, the sound of the tuba was also red. Or take Ginsberg’s (1923, p. 589) suggestion that: “Most, if not all of us, seem to agree with the following descriptive phrases; the silvery tone of the violin, the red blare of the cornet, the golden voice of the tenor, etc.” In 1899, the French musicologist, Albert Lavignac suggested a number of instrument-colour relationships such as Flute—azure blue; Oboe—green; Clarinet—red-brown; Horn—yellow; English horn—violet; Trumpet—crimson with orange; Trombone—crimson with orange; Cornet—red; Bassoon—dark brown; Timpani—black; Side drum—grey; Triangle—silver; and Violin—blue (see Donnell-Kotrozo, 1978; Lavignac, 1899). ²⁶

In an early study, entitled ‘The common synaesthesia of music’, Mudge (1920) assessed visual responses to music in a group of 50 mature students. The latter were instructed to report the colours or brightnesses that they associated with certain tones, keys, instruments and familiar musical compositions. While Mudge notes that there was little uniformity as to the particular colour associations (and that eight of his participants reported a total lack of such colour associations), Mudge does admit that some tentative generalizations appear to be warranted. In particular, 34 of the students thought that low tones yielded dark colours, 26 that medium tones yield medium bright colours, and 36 that high tones yielded bright colours. In terms of timbre-colour correspondences, the sound of the violin was typically reported as blue or violet (or related colours), the trombone was reported to be dark, often brown (though for some it was yellow), and the clarinet and flute were reported to be bright. Mudge concludes with the suggestion that there appears to be an association between the complexity of tones, or timbre, and the ‘richness’ of the associated colour.

In recent years, a number of other studies have been published in which significantly non-random associations have been documented between timbre and colour/hue in non-synaesthetic individuals (cf. Adeli et al., 2014) (see Table 4). For instance, Reuter et al. (2018) claim (albeit only in a conference paper thus far) to have demonstrated consistent colour-timbre mappings in non-synesthetic individuals. Meanwhile, in terms of auditory timbre-colour/shape crossmodal correspondences, Adeli and colleagues concluded that their participants: “strongly associated soft timbres with blue, green or light grey rounded shapes, harsh timbres with red, yellow or dark grey sharp angular shapes and timbres having elements of softness and harshness together with a mixture of the two previous shapes.” (see also Arai et al., 2021; Gurman, McCormick, & Klein, in press, on timbre-visual shape correspondences). It should, though, be noted that in much of the modern research, colour and visual brightness are conceptualized as a couple amongst many of the crossmodal correspondences that people may hold with specific timbres, including with basic tastes, olfactory stimuli, textures, volumes etc. (Crisinel & Spence, 2010, 2012; see also Giannos et al., 2021; Saitis et al., 2020). ²⁷ That is, hue-brightness to timbre crossmodal correspondences would not seem to be granted any kind of preferential status relative to the others. This state of affairs obviously contrasts with the disproportionate amount of theorizing that has seemingly been given over to the putative crossmodal correspondences between pitch and hue.

Alexander Wallace Rimington (1854–1918), a British inventor and professor of fine arts, based in London, built one of the first colour organs, patented in 1893. The first public performance of his colour organ took place in June, 1895, at St. James Hall, London (Rimington, 1895). ²⁹ Rimington believed in the physical equivalence of light and sound, and in his compositions, attempted to apply three musical functions, namely time (possibly referring to tempo), rhythm, and instantaneous combination (slow or rapid and varied) to colour. Zilczer (1987, pp. 118–119) notes that: “Rimington equated the seven spectrum bands of natural light with diatonic intervals which composed the musical octave.” (see Figure 6). ³⁰

OPEN IN VIEWER

Figure 6.

Illustration of the colour-musical note mapping that was adopted by Rimington (1895) for the first performance of his colour organ in London.

Interestingly, however, a closer inspection of Rimington’s original address, given at the start of his first public performance in London, reveals that while the inventor stresses the physical analogies between vision and hearing, he leaves open the question of what he calls the psychic consequences. As Rimington puts it: “It will be a question of opinion, and of further experiment, whether the close physical analogy between the octaves of colour and sound has its physiological and psychical counter-part.” In fact, Rimington’s (1895) address is worth quoting at some length here, given the clarity with which he pursues the physical similarity and structural analogies between colour and music:

“Taking the spectrum band as the basis of all colour, there are two remarkable points of resemblance between it and the musical octave, which have long been commented upon and discussed. The first of them is that the different colours of the one, and the different notes of the other are both due to various rates of vibration, acting on the eye or the ear. This is very simply and clearly put by Professor Schellen in his great work upon spectrum analysis. ‘Different colours,’ he says, ‘are produced by the different degrees of rapidity with which the ether vibrations recur, just as the various notes in music depend upon the rapidity of the succession of vibrations of air.’ In a word, ‘colours are to the eye what musical tones are to the ear….

It is pertinent to note that the only claim Rimington makes in support of his suggestion that his colour organ successfully translated music into visual form was the fact that apparently great pieces of music played on the colour organ give rise to pleasing effects. ³¹ Here, one is reminded of similar claims concerning the scent organ (see Spence, 2021, for a review). Rimington wrote: “That colour, like sound, is capable of expressing artistic emotion there can, I think, be no question, but whether it expresses it in the same way as music is doubtful. It is, however, a somewhat strong argument in favour of the existence of the physiological and psychical analogy, that when we avail ourselves of the works of great musical composers for the interpretation of the new art, the results are vastly superior in variety, delicacy, and beauty of colour to those hitherto obtainable by other methods.” One can also find a similar appeal to the colourfulness of the transposition in Pridmore (1992, p. 60) when discussing what the Beatles’ music would look like when transduced by one of his sound-to-light convertors: “Interestingly, all-time famous pop bands, e.g., The Beatles, play very “colourful” music.” (see also Garner, 1978, for speculation on what the opening notes of “God save the Queen” would look like if rendered sequentially in colour). Again, though, one is left wondering how pleasant a control group of random sounds/music would be?

The artist Max Weber (Weber, 1916), just like Kandinsky (1977), was also inspired by synaesthesia, ³² and by the emotional correspondences that he though existed between painting and music (note here Kandinsky’s notion of ‘inner harmony’). A similar appeal to emotion can also be found in Sabaneyev (1911, p. 200; trans. in Galeyev & Vanechkina, 2001, p. 358) who writes that: “Colours, on the one hand, and sounds, on the other hand, engender various moods, often similar to one another, therefore—the associations of colours and sounds arises.”

According to Zilczer (1987), the North American architect Claude Bragdon, who was also a fan of colour music, put on a couple of colour music shows in New York (one in 1915 and the other in 1916). Bragdon based his notion of colour music on the psychological, rather than the purely physical analogy between light and sound. In his 1918 book, Architecture and democracy, Bragdon writes: “If we are to have colour symphonies, the best are not likely to be those based on a literal translation of some musical masterpiece into colour according to this or any theory, but those created by persons who are emotionally reactive to this medium, able to imagine in colour, and to treat it imaginatively.” (Bragdon, 1918, p. 139). In his work, Bragdon was therefore keen to enlarge the concept of colour music beyond a ‘simple’ physical correspondence between colour and music (see also Bragdon, 1916). Meanwhile, according to Zilczer (1987, p. 122), the Philadelphia pianist, May Hallock Greenewalt (active at around the same time) also promoted colour music. She apparently agreed with Brandon on the importance of the psychological basis for musical analogy, by which we might infer the common mood associations of colour and music.

Artist Stanton Macdonald-Wright, who together with Morgan Russell invented the style ‘Synchromism’ (see South, 2001; Zilczer, 1987), believed that an exact correspondence between the colour spectrum and the musical scale would provide a key to their new art. He later wrote: “For many years there has been growing a conviction that there is some deeply rooted, recondite analogy between colour and sound. Both are demonstrably vibratory; both have a varied and defined emotional stimulus for us, and each is used as a medium for an art.” (Macdonald-Wright, 1924, p. 14). This suggestion is curious inasmuch as it appeals both to an exact correspondence between colour and music while also invoking the importance of emotion.

Meanwhile, in the Soviet Union (as was), the Group 'Prometei' carried forward Scriabin’s dream of synthesizing electric light and music in the intervening decades since the composer’s death. According to one commentator: “They explored correlations between the two fields in a number of different ways: (a) correlations with individual qualities of music (pitch, key, timbre, and harmony); (b) correlations with musical themes; (c) correlations with different qualities and themes of music; (d) the polyphonic (contrapuntal) audio-visual integration approach.” (see Galeyev, 1976; Pridmore, 1992).

According to Zilczer’s (1987) excellent summary of the literature on colour music, those working in the field of contemporary art would largely seem to have lost their former fascination with the synaesthetic coloured hearing. As Moritz’s (1997) review makes clear, a number of other artists/film-makers worked with developing abstract colour displays to pair with music. One might think of Sergei Eisenstein’s (1898-1948) interest in sound-colour montage (see Harrison, 2001, p. 133). Eisenstein also wrote at one point that there is no “pervading law of absolute meanings and correspondences between colours and sound.” (quoted in Harrison, 2001, pp. 133–134). Disney’s Fantasia, with music conducted by Leopold Stokowski, can perhaps also be seen as a natural extension of the colour music movement (Pridmore, 1992), though a key strand of the emerging interest was linked to the notion of audiovisual Gestalten, which relied as much on synchrony as correspondence (see also Alves, 2005; Battey & Fischman, 2016; Corra, 1973; HaCohen, 2016; Haverkamp, 2020; Kaduri, 2016; Russet & Starr, 1988; Whitney, 1980). At the same time, however, it has also become increasingly clear that there is no straightforward structural or psychological mapping between this particular pair of senses. Indeed, Vanechkina (1973) noted almost half a century ago how almost all of those he spoke to specially emphasized the associative, metaphoric nature of “colour hearing” in music and excluded from the artistic sphere any clinical cases of colour hearing.

In the 18^th century, Castel confidently predicted that every home in Paris would one day have an Ocular Harpsichord for recreation purposes (Moritz, 1997). Meanwhile, according to Marks (1975, p. 313): “By the late nineteenth century and early twentieth century, interest in colour organs had blossomed into what might almost be called an epidemic (e.g., Sullivan, 1914).” Nevertheless, the great enthusiasm that colour music once aroused rapidly faded, presumably because no agreed perceptual match exists – just consider the wide variety of potential mappings that have been put forward over the years.

Synaesthesia was once a popular source of inspiration for those interested in colour music. However, other accounts based on natural, physical relations (the latter based on the physical properties of the stimuli themselves and the idiosyncratic nature of the perceptual continua), or analogous mapping in Rimington’s matching on the basis of complexity have also, on occasion been proposed. And yet it is noticeable that the common explanation, mentioned, or supported, by so much of the research, both scientific and artistic in origin, is the affective correspondences resulting from shared affect, mood, or emotion of the component stimuli. That said, there would appear to be a growing realization of the potential relevance of crossmodal correspondences to the study of music (Eitan, 2017; Timmers, 2022; Walker, 2016), or as an additional means of illustrating the ‘meaning’ of music for music students (Donnell-Kotrozo, 1978; Wells, 1980; see Bresin, 2005, for an attempt to illustrate the expressivity in music performance by means of the use of colour to link to specific emotions; cf. Clarke & Costall, 2008; Kandinsky, 1977).

Hereinafter, we provide a list of issues regarding colour-sound correspondences that apparently limits the possibility of explaining such correspondences (or analogies; see Jewanski, 2010) by assuming any physical similarities between the two sensory stimuli.

Based on the above considerations, it seems hard to explain colour-sound correspondences based on structural similarities between the component stimuli (cf. Helmholtz, 1867). Thus, it seems that two further possibilities remain. One is that stimuli are matched based on their relative position along the continuum in each sensory modality. Such an explanation (would) fit(s) with results from studies that used isolated tones, which listeners can perceive as higher or lower (such dimension might parallel, as some suggested, brightness for colours). However, when considering complex musical stimuli (e.g., Cutietta & Haggerty, 1987; see Spence, 2020a, for a review), pitch is a much more blurred perceptual correlate of sound. In such cases, probably, one needs to put forward an alternative explanation, according to which the mapping can be conceived of in terms of emotional similarities (or mediation), or even ‘inner harmony’, to use Kandinsky’s (1977) preferred terminology. Although such explanation leaves open the question about the nature of such emotional/inner harmonies, it suggests avoiding simplistic conception of mapping based on elementary properties of visual and auditory stimuli, such as brightness and pitch that seem unable to explain consistent mapping, if any.

While the popularity of colour music has undoubtedly declined, since its peak in the years around 1900, the last decade or so has seen something of a resurgence of interest in the topic of colour-sound correspondences amongst those researchers interested in the development of more intuitive sensory substitution systems (cf. Hamilton-Fletcher et al., 2016; Marks, 1983; Pridmore, 1992). In one recent study, for instance, Cho et al. (2020) investigated whether sound can be used to code colour in order to improve artwork appreciation by those individuals with visual impairments (see also Cavazos Quero et al., 2021). The question immediately crops up though as to whether certain dimension(s) of sound should be used to convey the different specific salient properties of visual images, such as colours. Is the choice an arbitrary one? Or can such systems be made more intuitive? Interestingly, in their recent research, Cavazos Quero and colleagues chose to combine sound with scent, thus presumably adding to the complexity of the process of sensory translation.

6.1. The Emotional Mediation Account(s)

In conclusion, what started out as intuition based on analogous mappings led to a search for exact natural, physical mappings between colour and musical notes. However, the evidence revealed a visual brightness-pitch mapping and, more recently, seemingly robust hue-timbre associations too. At the same time, however, while some have been convinced of the existence of a natural, physical, mapping perhaps based on perceived similarity between matching stimuli in the respective perceptual continua, it is striking how many of those working in this area over recent centuries have ended-up arguing for emotional mediation, sometimes referred to in terms of affect, mood, or what Kandinsky (1977) referred to as ‘inner harmony’, instead. In this regard, the crossmodal correspondence between simple stimuli bears close relation to the search for crossmodal correspondences amongst more complex visual stimuli, such as paintings and pieces of music, documented elsewhere (see Spence, 2020a, for a review).

Indeed, in recent years, there has undoubtedly been growing interest amongst psychologists and practitioners in emotional correspondences between music and colour (or paintings), or emotionally-mediated (or affective) correspondences, as they are sometimes known (Spence, 2020a). The interest in the emotional mediation of colour-to-music mappings is undoubtedly one that provides one additional means of translating between sensory impressions (though perhaps is not as rigorous and objective as some commentators would have liked). Looking back, it is interesting to note how Simpson et al. (1956, p. 100) already hinted at the emotional mediation account as an “indirect” association: “And lastly, before direct associations between hue and pitch could be postulated, it would first be necessary to eliminate the possibility of indirect associations based upon parallel relationships to a common mediating variable such as mood.” The primacy of the emotional mediation account also fits with Werner’s (1934) view of perceptual experiences in infancy and early childhood as largely syncretic – that is, functionally undifferentiated and as physiognomic – that is, imbued with expression and affective properties.

And, in terms of the question with which we started, why should it be that the musical sound pitch/timbre-hue/brightness mappings have attracted so much interest over the centuries, given the many other crossmodal correspondences that are now known to exist (see Spence, 2011), it would seem that there have been multiple different drivers helping to sustain the interest in the topic at different points in time. This interest can be seen as originating from the possibility of structural analogies, from the possibility of perceptual analogies, as well as with artistic interest, technological developments (colour organs previously, and sensory substitution devices more recently), and the florid concurrents reported by coloured-music synaesthetes. The fact that we are all such visually-dominant creatures (Hutmacher, 2019), and that colour appears to play a particularly important part in driving human behaviour (Elliott et al., 2015), may also be relevant here. Ultimately, though, the fact that it is affect, mood, or emotion that appears to explain the particular affinity people experience between musical sounds and colours, no matter whether the stimuli are simple or complex (e.g., paintings and musical composition; see Spence, 2020a), means that pitch/timbre-hue correspondences are, in no way, intrinsically special.

To conclude, it is perhaps worth returning to the opening sentences of the article written by Sabaneev and Pring (1929, p. 266) almost a century ago to consider how far (or not) we have come:

“THERE was a time when it would have been regarded as absurd, or at all events as a symptom of decadence, to raise the question of a correspondence between sound and colour. And yet amongst musicians this is by no means a new problem. We know that a very long time ago there were persons, very often musicians, to whom sounds presented themselves as coloured, so to speak. Resonance evoked a colour association, and this was not a fortuitous happening but was repeated with the invariability of a law. Later on, of course, this phenomenon or, if you prefer it, this faculty, attracted a certain amount of attention amongst scholars, and the name of synopsy or colour-ear was bestowed upon it. So far it appears to be an open question as to whether this connection is organic, conditioned by certain causes of a physiological character, certain proximities or contacts of the optic and auditory nervous ramifications; or whether it is merely associative, a kind of conditional reflex, and therefore may vary with different persons, maintaining nevertheless within the given individual a certain stability.”