Perception

The auditory cortex of nonhuman primates is comprised of a constellation of at least twelve interconnected areas distributed across three major regions on the superior temporal gyrus: core, belt, and parabelt. Individual areas are distinguished on the basis of unique profiles comprising architectonic features, thalamic and cortical connections, and neuron response properties. Recent demonstrations of convergent auditory – somatosensory interactions in the caudomedial (CM) and caudolateral (CL) belt areas prompted us to pursue anatomical studies to identify the source(s) of somatic input to auditory cortex. Corticocortical and thalamocortical connections were revealed by injecting neuroanatomical tracers into CM, CL, and adjoining fields of marmoset (Callithrix jacchus jacchus) and macaque (Macaca mulatta) monkeys. In addition to auditory cortex, the cortical connections of CM and CL included somatosensory (retroinsular, Ri; granular insula, Ig) and multisensory areas (temporal parietal occipital, temporal parietal temporal). Thalamic inputs included the medial geniculate complex and several multisensory nuclei (supra- geniculate, posterior, limitans, medial pulvinar), but not the ventroposterior complex. Injections of the core (A1, R) and rostromedial areas of auditory cortex revealed sparse multisensory connections. The results suggest that areas Ri and Ig are the principle sources of somatosensory input to the caudal belt, while multisensory regions of cortex and thalamus may also contribute. The present data add to growing evidence of multisensory convergence in cortical areas previously considered to be ‘unimodal’, and also indicate that auditory cortical areas differ in this respect.

Much of the information about multisensory integration is derived from studies of the cat superior colliculus (SC), a midbrain structure involved in orientation behaviors. This integration is apparent in the enhanced responses of SC neurons to cross-modal stimuli, responses that exceed those to any of the modality-specific component stimuli. The simplest model of multisensory integration is one in which the SC neuron simply sums its various sensory inputs. However, a number of empirical findings reveal the inadequacy of such a model; for example, the finding that deactivation of cortico-collicular inputs eliminates the enhanced response to a cross-modal stimulus without eliminating responses to the modality-specific component stimuli. These and other empirical findings inform a computational model that accounts for all of the most fundamental aspects of SC multisensory integration. The model is presented in two forms: an algebraic form that conveys the essential insights, and a compartmental form that represents the neuronal computations in a more biologically realistic way.

Although humans are almost constantly exposed to stimuli from multiple sensory modalities during daily life, the processes by which we learn to integrate information from multiple senses to acquire knowledge of multisensory objects are not well understood. Here, we present results of a novel audio – visual statistical learning procedure where participants are passively exposed to a rapid serial presentation of arbitrary audio — visual pairings (comprised of artificial/synthetic audio and visual stimuli). Following this exposure, participants were tested with a two-interval forced-choice procedure in which their degree of familiarity with the experienced audio-visual pairings was evaluated against novel audio — visual combinations drawn from the same stimulus set. Our results show that subjects acquire knowledge of visual — visual, audio — audio, and audio — visual stimulus associations and that the learning of these types of associations occurs in an independent manner.

Gestalt rules that describe how visual stimuli are grouped also apply to sounds, but it is unknown if the Gestalt rules also apply to tactile or uniquely multimodal stimuli. To investigate these rules, we used lights, touches, and a combination of lights and touches, arranged in a classic Ternus configuration. Three stimuli (A, B, C) were arranged in a row across three fingers. A and B were presented for 50 ms and, after a delay, B and C were presented for 50 ms. Subjects were asked whether they perceived AB moving to BC (group motion) or A moving to C (element motion). For all three types of stimuli, at short delays, A to C dominated, while at longer delays AB to BC dominated. The critical delay, where perception changed from group to element motion, was significantly different for the visual Ternus (3 lights, 162 ms) and the tactile Ternus (3 touches, 195 ms). The critical delay for the multimodal Ternus (3 light – touch pairs, 161 ms) was not different from the visual or tactile Ternus effects. In a second experiment, subjects were exposed to 2.5 min of visual group motion (stimulus onset asynchrony = 300 ms). The exposure caused a shift in the critical delay of the visual Ternus, a trend in the same direction for the multimodal Ternus, but no shift in the tactile Ternus. These results suggest separate but similar grouping rules for visual, tactile, and multimodal stimuli.

We tested the hypothesis that long-term adaptation to the normal contingencies between walking and its multisensory consequences (including optic flow) leads to enhanced discrimination of appropriate visual speeds during self-motion. In experiments 1 (task 1) and 2 a two-interval forced-choice procedure was used to compare the perceived speed of a simulated visual flow field viewed while walking with the perceived speed of a flow field viewed while standing. Both experiments demonstrated subtractive reductions in apparent speed. In experiments 1 and 3 discrimination thresholds were measured for optic flow speed while walking and while standing. Consistent with the optimal-coding hypothesis, speed discrimination for visual speeds near walking speed was enhanced during walking. Reduced sensitivity was found for slower visual speeds. The multisensory context of walking alters the coding of optic flow in a way that enhances speed discrimination in the expected range of flow speeds.

Multisensory integration is a powerful mechanism for maximizing sensitivity to sensory events. We examined its effects on auditory localization in healthy human subjects. The specific objective was to test whether the relative intensity and location of a seemingly irrelevant visual stimulus would influence auditory localization in accordance with the inverse effectiveness and spatial rules of multisensory integration that have been developed from neurophysiological studies with animals [Stein and Meredith, 1993 The Merging of the Senses (Cambridge, MA: MIT Press)]. Subjects were asked to localize a sound in one condition in which a neutral visual stimulus was either above threshold (supra-threshold) or at threshold. In both cases the spatial disparity of the visual and auditory stimuli was systematically varied. The results reveal that stimulus salience is a critical factor in determining the effect of a neutral visual cue on auditory localization. Visual bias and, hence, perceptual translocation of the auditory stimulus appeared when the visual stimulus was supra-threshold, regardless of its location. However, this was not the case when the visual stimulus was at threshold. In this case, the influence of the visual cue was apparent only when the two cues were spatially coincident and resulted in an enhancement of stimulus localization. These data suggest that the brain uses multiple strategies to integrate multisensory information.

We investigated the frame of reference involved in audio – visual (AV) fusion over space. This multisensory phenomenon refers to the perception of unity resulting from visual and auditory stimuli despite their potential spatial disparity. The extent of this illusion depends on the eccentricity in azimuth of the bimodal stimulus (Godfroy et al, 2003 Perception 32 1233–1245). In a previous study, conducted in a luminous environment, Roumes et al 2004 (Perception 33 Supplement, 142) have shown that variation of AV fusion is gaze-dependent. Here we examine the contribution of ego- or allocentric visual cues by conducting the experiment in total darkness. Auditory and visual stimuli were displayed in synchrony with various spatial disparities. Subjects had to judge their unity (‘fusion’ or ‘no fusion’). Results showed that AV fusion in darkness remains gaze-dependent despite the lack of any allocentric cues and confirmed the hypothesis that the reference frame of the bimodal space is neither head-centred nor eye-centred.

We examined whether or not abrupt tactile onsets are capable of exogenously capturing tactile spatial attention when visual spatial attention is focused elsewhere. In experiment 1, we compared performance under dual-task conditions (where participants performed a tactile exogenous cuing task and a rapid serial visual presentation—RSVP—task at the same time) with their performance under single-task conditions (where the participants had to perform only the cuing task, although the RSVP stream was still presented in the background) and to a no-stream condition (where only the cuing task was presented). Tactile cuing was completely suppressed in both the dual-task and single-task conditions, showing that exogenous tactile spatial orienting is modulated by visual-spatial attention, which hence appears to be far from truly automatic. In experiment 2, we demonstrated that the abolishment of exogenous tactile orienting was not caused by the transient presentation of abrupt onset stimuli (letters). These results therefore show that exogenous spatial attentional orienting toward abrupt peripheral tactile stimuli is possible as long as perceptual resources are not depleted by a perceptually demanding (RSVP) task.

Previous studies have shown that the perceived location of visual stimuli briefly flashed during smooth pursuit, saccades, or optokinetic nystagmus (OKN) is not veridical. We investigated whether these mislocalisations can also be observed for brief auditory stimuli presented during OKN. Experiments were carried out in a lightproof sound-attenuated chamber. Participants performed eye movements elicited by visual stimuli. An auditory target (white noise) was presented for 5 ms. Our data clearly indicate that auditory targets are mislocalised during reflexive eye movements. OKN induces a shift of perceived location in the direction of the slow eye movement and is modulated in the temporal vicinity of the fast phase. The mislocalisation is stronger for look- as compared to stare-nystagmus. The size and temporal pattern of the observed mislocalisation are different from that found for visual targets. This suggests that different neural mechanisms are at play to integrate oculomotor signals and information on the spatial location of visual as well as auditory stimuli.

The relationship between visually and haptically derived representations of objects is an important question in multisensory processing and, increasingly, in mental representation. We review evidence for the format and properties of these representations, and address possible theoretical models. We explore the relevance of visual imagery processes and highlight areas for further research, including the neglected question of asymmetric performance in the visuo – haptic cross-modal memory paradigm. We conclude that the weight of evidence suggests the existence of a multisensory representation, spatial in format, and flexibly accessible by both bottom — up and top — down inputs, although efficient comparison between modality-specific representations cannot entirely be ruled out.

The brain integrates object information from multiple sensory systems to form a unique representation of our environment. Temporal synchrony and spatial coincidence are important factors for multisensory integration, indicating that the multisensory signals come from a common source. Spatial separations can lead to a decline of visual – haptic integration (Gepshtein et al, 2005 Journal of Vision 5 1013–1023). Here we tested whether prior knowledge that two signals arise from the same object can promote integration even when the signals are spatially discrepant. In one condition, participants had direct view of the object they touched. In a second condition, mirrors were used to create a spatial separation between the seen and the felt object. Participants saw the mirror and their hand in the mirror exploring the object and thus knew that they were seeing and touching the same object. To determine the visual — haptic interaction we created a conflict between the seen and the felt shape using an optically distorting lens that made a rectangle look like a square. Participants judged the shape of the probe by selecting a comparison object matching in shape. We found a mutual biasing effect of shape information from vision and touch, independent of whether participants directly looked at the object they touched or whether the seen and the felt object information was spatially separated with the aid of a mirror. This finding suggests that prior knowledge about object identity can promote integration, even when information from vision and touch is provided at spatially discrepant locations.

Speech perception under natural conditions entails integration of auditory and visual information. Understanding how visual and auditory speech information are integrated requires detailed descriptions of the nature and processing of visual speech information. To understand better the process of gathering visual information, we studied the distribution of face-directed fixations of humans performing an audiovisual speech perception task to characterise the degree of asymmetrical viewing and its relationship to speech intelligibility. Participants showed stronger gaze fixation asymmetries while viewing dynamic faces, compared to static faces or face-like objects, especially when gaze was directed to the talkers' eyes. Although speech perception accuracy was significantly enhanced by the viewing of congruent, dynamic faces, we found no correlation between task performance and gaze fixation asymmetry. Most participants preferentially fixated the right side of the faces and their preferences persisted while viewing horizontally mirrored stimuli, different talkers, or static faces. These results suggest that the asymmetrical distributions of gaze fixations reflect the participants' viewing preferences, rather than being a product of asymmetrical faces, but that this behavioural bias does not predict correct audiovisual speech perception.

When a hand (either real or fake) is stimulated in synchrony with our own hand concealed from view, the felt position of our own hand can be biased toward the location of the seen hand. This intriguing phenomenon relies on the brain's ability to detect statistical correlations in the multisensory inputs (ie visual, tactile, and proprioceptive), but it is also modulated by the pre-existing representation of one's own body. Nonetheless, researchers appear to have accepted the assumption that the size of the seen hand does not matter for this illusion to occur. Here we used a real-time video image of the participant's own hand to elicit the illusion, but we varied the hand size in the video image so that the seen hand was either reduced, veridical, or enlarged in comparison to the participant's own hand. The results showed that visible-hand size modulated the illusion, which was present for veridical and enlarged images of the hand, but absent when the visible hand was reduced. These findings indicate that very specific aspects of our own body image (ie hand size) can constrain the multisensory modulation of the body schema highlighted by the fake-hand illusion paradigm. In addition, they suggest an asymmetric tendency to acknowledge enlarged (but not reduced) images of body parts within our body representation.