The Roles of Physical and Physiological Simultaneity in Audiovisual Multisensory Facilitation

Abstract

A series of experiments measured the audiovisual stimulus onset asynchrony (SOA_AV), yielding facilitative multisensory integration. We evaluated (1) the range of SOA_AV over which facilitation occurred when unisensory stimuli were weak; (2) whether the range of SOA_AV producing facilitation supported the hypothesis that physiological simultaneity of unisensory activity governs multisensory facilitation; and (3) whether AV multisensory facilitation depended on relative stimulus intensity. We compared response-time distributions to unisensory auditory (A) and visual (V) stimuli with those to AV stimuli over a wide range (300 and 20 ms increments) of SOA_AV, across four conditions of varying stimulus intensity. In condition 1, the intensity of unisensory stimuli was adjusted such that d′ ≈ 2. In condition 2, V stimulus intensity was increased (d′ > 4), while A stimulus intensity was as in condition 1. In condition 3, A stimulus intensity was increased (d′ > 4) while V stimulus intensity was as in condition 1. In condition 4, both A and V stimulus intensities were increased to clearly suprathreshold levels (d′ > 4). Across all conditions of stimulus intensity, significant multisensory facilitation occurred exclusively for simultaneously presented A and V stimuli. In addition, facilitation increased as stimulus intensity increased, in disagreement with inverse effectiveness. These results indicate that the requirements for facilitative multisensory integration include both physical and physiological simultaneity.

Keywords

multisensory integration neural coactivation inverse effectiveness race model simultaneity reaction time d′

References

Alvarado

J. C.

Vaughan

J. W.

Stanford

T. R.

Stein

B. E.

(2007). Multisensory versus unisensory integration: Contrasting modes in the superior colliculus. Journal of Neurophysiology, 97, 3193–3205. doi:10.1152/jn.00018.2007

Barlow

R. B.

Snodderly

D. M.

Swadlow

H. A.

(1978). Intensity coding in primate visual system. Experimental Brain Research, 31, 163–177. doi:10.1007/BF00237597

Callan

D. E.

Callan

A. M.

Kroos

Vatikiotis-Bateson

(2001). Multimodal contribution to speech perception revealed by independent component analysis: A single sweep EEG case study. Cognitive Brain Research, 10, 349–353. doi:10.1016/S0926-6410(00)00054-9

Calvert

G. A.

(2001). Crossmodal processing in the human brain: Insights from functional neuroimaging studies. Cerebral Cortex, 11, 1110–1123. doi:10.1093/cercor/11.12.1110

Campbell

F. W.

Green

D. G.

(1965). Monocular vs. binocular visual acuity. Nature, 208, 191–192. doi:10.1038/208191a0

Celesia

G. G.

Puletti

(1971). Auditory input to the human cortex during states of drowsiness and surgical anesthesia. Electroencephalography and Clinical Neurophysiology, 31, 603–609. doi:10.1016/0013-4694(71)90076-9

Clark

V. P.

Hillyard

S. A.

(1996). Spatial selective attention affects early extrastriate but not striate components of the visual evoked potential. Journal of Cognitive Neuroscience, 8, 387–402. doi:10.1162/jocn.1996.8.5.387

Diederich

Colonius

(2004). Bimodal and trimodal multisensory enhancement: Effects of stimulus onset and intensity on reaction time. Perception and Psychophysics, 66, 1388–1404. doi:10.3758/BF03195006

Eggermont

J. J.

(1998). Azimuth coding in primary auditory cortex of the cat: Relative latency and interspike interval representation. Journal of Neurophysiology, 80, 2151–2161

10.

Foster

D. H.

Bischof

W. F.

(1991). Bootstrap estimates of the statistical accuracy of thresholds obtained from psychometric functions. Spatial Vision, 11, 135–139

11.

Foxe

J. J.

Simpson

G. V.

(2002). Flow of activation from V1 to frontal cortex in humans: A framework for defining “early” visual processing. Experimental Brain Research, 142, 139–150. doi:10.1007/s00221-001-0906-7

12.

Foxe

J. J.

Strugstad

E. C.

Sehatpour

Molholm

Pasieka

Schroeder

C. E.

McCourt

M. E.

(2008). Parvocellular and magnocellular contributions to the initial generators of the visual evoked potential: High-density electrical mapping of the “C1” component. Brain Topography, 21, 11–21. doi:10.1007/s10548-008-0063-4

13.

Frassinetti

Bolognini

Ladàvas

(2002). Enhancement of visual perception by crossmodal visuo–auditory interaction. Experimental Brain Research, 147, 332–343. doi:10.1007/s00221-002-1262-y

14.

Giard

M. H.

Peronnet

(1999). Auditory–visual integration during multi-modal object recognition in humans: A behavioral and electrophysiological study. Journal of Cognitive Neuroscience, 11, 473–490. doi:10.1162/089892999563544

15.

Hershenson

(1962). Reaction time as a measure of intersensory facilitation. Journal of Experimental Psychology, 63, 289–293. doi:10.1037/h0039516

16.

Holmes

N. P.

(2007). The law of inverse effectiveness in neurons and behavior: Multisensory integration versus normal variability. Neuropsychologia, 45, 3340–3345. doi:10.1016/j.neuropsychologia.2007.05.025

17.

Holmes

N. P.

(2009). The principle of inverse effectiveness in multisensory integration: Some statistical considerations. Brain Topography, 21, 168–176. doi:10.1007/s10548-009-0097-2

18.

Holmes

N. P.

Spence

(2005). Multisensory integration: Space, time and superadditivity. Current Biology, 15, R762–R764. doi:10.1016/j.cub.2005.08.058

19.

Iacoboni

Zaidel

(2003). Interhemispheric visuo-motor integration in humans: The effect of redundant targets. European Journal of Neuroscience, 17, 1981–1986. doi:10.1046/j.1460-9568.2003.02602.x

20.

Jaskowski

(1985). The effect of visual adaptation on simple motor reaction time: Pt. 1. Studia Psychologica, 27, 191–201

21.

Jeffress

L. A.

(1948). A place theory of sound localization. Journal of Comparative and Physiological Psychology, 41, 35–39. doi:10.1037/h0061495

22.

Jeffreys

D. A.

Axford

J. G.

(1972). Source locations of pattern-specific components of human visual evoked potentials. II. Component of extrastriate cortical origin. Experimental Brain Research, 16, 22–40. doi:10.1007/BF00233372

23.

Lakatos

Chen

C. M.

O'Connell

M. N.

Mills

Schroeder

C. E.

(2007). Neuronal oscillations and multisensory interaction in primary auditory cortex. Neuron, 53, 279–292. doi:10.1016/j.neuron.2006.12.011

24.

Legge

G. E.

(1984). Binocular contrast summation II. Quadratic summation. Vision Research, 24, 385–394. doi:10.1016/0042-6989(84)90064-6

25.

Mansfield

R. J. W.

(1973). Latency functions in human vision. Vision Research, 13, 2219–2234. doi:10.1016/0042-6989(73)90224-1

26.

Maunsell

J. H. R.

Ghose

G. M.

Assad

J. A.

McAdams

C. J.

Boudreau

C. E.

Noerager

B. D.

(1999). Visual response latencies of magnocellular and parvocellular LGN neurons in macaque monkeys. Visual Neuroscience, 16, 1–14

27.

Meredith

M. A.

(2002). On the neuronal basis for multisensory convergence: A brief overview. Cognitive Brain Research, 14, 1–40. doi:10.1016/S0926-6410(02)00059-9

28.

Meredith

M. A.

Allman

(2009). Subthreshold processing in cat auditory cortex. NeuroReport, 20, 126–131. doi:10.1097/WNR.0b013e32831d7bb6

29.

Meredith

M. A.

Nemitz

J. W.

Stein

B. E.

(1987). Determinants of multisensory integration in superior colliculus neurons. I. Temporal factors. Journal of Neuroscience, 7, 3215–3229

30.

Meredith

M. A.

Stein

B. E.

(1986). Visual auditory and somatosensory convergence on cells in superior colliculus results in multisensory integration. Journal of Neuroscience, 56, 640–662

31.

Miller

(1982). Divided attention: Evidence for coactivation with redundant signals. Cognitive Psychology, 14, 247–279. doi:10.1016/0010-0285(82)90010-X

32.

Miller

(1986). Timecourse of coactivation in bimodal divided attention. Perception and Psychophysics, 40, 331–343. doi:10.3758/BF03203025

33.

Miniussi

Girelli

Marzi

C. A.

(1998). Neural site of the redundant target effect: Electrophysiological evidence. Journal of Cognitive Neuroscience, 10, 216–225. doi:10.1162/089892998562663

34.

Molholm

Ritter

Murray

M. M.

Javitt

D. C.

Schroeder

C. E.

Foxe

J. J.

(2002). Multisensory auditory–visual interactions during early sensory processing in humans: A high-density electrical mapping study. Cognitive Brain Research, 14, 115–128. doi:10.1016/S0926-6410(02)00066-6

35.

Molholm

Sehatpour

Mehta

A. D.

Shpaner

Gomez-Ramirez

Ortigue

Foxe

J. J.

(2006). Audio-visual multisensory integration in superior parietal lobule revealed by human intracranial recordings. Journal of Neurophysiology, 96, 721–729. doi:10.1152/jn.00285.2006

36.

Mordkoff

J. T.

Yantis

(1991). An interactive race model of divided attention. Journal of Experimental Psychology, 17(2), 520–538. doi:10.1037/0096-1523.17.2.520

37.

Murray

M. M.

Foxe

J. J.

Higgins

B. A.

Javitt

D. C.

Schroeder

C. E.

(2001). Visuo-spatial neural response interactions in early cortical processing during a simple reaction time task: A high-density electrical mapping study. Neuropsychologia, 39, 828–844. doi:10.1016/S0028-3932(01)00004-5

38.

Otto

T. U.

Mamassian

(2012). Noise and correlations in parallel perceptual decision making. Current Biology, 22, 1–6. doi:10.1016/j.cub.2012.05.031

39.

Phillips

D. P.

(1998). Factors shaping the response latencies of neurons in the cat's auditory cortex. Behavioural Brain Research, 93, 33–41. doi:10.1016/S0166-4328(97)00139-3

40.

Piéron

(1952). The Sensations: Their Functions, Processes and Mechanisms. London: Frederick Muller Ltd

41.

Prestrude

A. M.

(1971). Visual latencies at photopic levels of retinal illuminance. Vision Research, 11, 351–361. doi:10.1016/0042-6989(71)90246-X

42.

Raab

D. H.

(1962). Statistical facilitation of simple reaction times. Transactions of the New York Academy of Sciences, 24, 574–590. doi:10.1111/j.2164-0947.1962.tb01433.x

43.

Ross

L. A.

Saint-Amour

Leavitt

V. M.

Javitt

D. C.

Foxe

J. J.

(2007). Do you see what I'm saying? Exploring visual enhancement of speech comprehension in noisy environments. Cerebral Cortex, 17, 1147–1153. doi:10.1093/cercor/bhl024

44.

Roufs

J. A. J.

(1963). Perception lag as a function of stimulus luminance. Vision Research, 3, 81–91. doi:10.1016/0042-6989(63)90070-1

45.

Senkowski

Saint-Amour

Höfle

Foxe

J. J.

(2011). Multisensory interactions in early evoked brain activity follow the principle of inverse effectiveness. NeuroImage, 56, 2200–2208. doi:10.1016/j.neuroimage.2011.03.075

46.

Serino

Farnè

Rinaldesi

M. L.

Haggard

Làdavas

(2007). Can vision of the body ameliorate impaired somatosensory function? Neuropsychologia, 45, 1101–1107. doi:10.1016/j.neuropsychologia.2006.09.013

47.

Song

Miller

K. D.

Abbott

L. F.

(2000). Competitive Hebbian learning through spike-timing-dependent synaptic plasticity. Nature Neuroscience, 3, 919–926. doi:10.1038/78829

48.

Stein

B. E.

Meredith

M. A.

(1993). The Merging of the Senses. Cambridge: MIT Press

49.

Stufflebeam

S. M.

Poeppel

Rowley

H. A.

Roberts

T. P. L.

(1998). Peri-threshold encoding of stimulus frequency and intensity in the M100 latency. NeuroReport, 9, 91–94

50.

Supek

Aines

C. J.

Ranken

Best

Flynn

E. R.

Wood

C. C.

(1999). Single vs. paired visual stimuli: Superposition of early neuromagnetic responses and retinotopy in extrastriate cortex in humans. Brain Research, 830, 43–55. doi:10.1016/S0006-8993(99)01316-5

51.

Turrigiano

G. G.

Leslie

K. R.

Desai

N. S.

Rutherford

L. C.

Nelson

S. B.

(1998). Activity-dependent scaling of quantal amplitude in neocortical neurons. Nature, 391, 892–896. doi:10.1038/36103

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.26 MB