Bimodal Displays Improve Speech Comprehension in Environments with Multiple Speakers

Abstract

Attending to a single voice when multiple voices are present is a challenging but common occurrence. An experiment was conducted to determine (a) whether presenting a video display of the target speaker aided speech comprehension in an environment with competing voices, and (b) whether the “ventriloquism effect” could be used to enhance comprehension, as found by Driver (1996), using ecologically valid stimuli. Participants listened for target words from videos of an actress reading while simultaneously ignoring the voices of 2 to 4 different actresses. Target-word detection declined as participants had to ignore more distracting voices; however, this decline was reduced when a video display of the target speaker was provided. Neither a signal-detection analysis of performance data nor a gaze-contingent analysis revealed a ventriloquism effect. Providing a video display of a speaker when competing voices are present improves comprehension, but obtaining the ventriloquism effect appears elusive in naturalistic circumstances. Actual or potential applications of this research include those circumstances in which a listener must filter a relevant stream of speech from among multiple, competing voices, such as air traffic control and military environments.

Get full access to this article

View all access options for this article.

References

Bertelson, P. , & Radeau, M. (1976). Ventriloquism, sensory interaction, and response bias: Remarks on the paper by Choe, Welch, Gilford and Juola. Perception and Psychophysics, 19, 531–535.

Blauert, J. (1997). Spatial hearing: The psychophysics of human sound localization (rev. ed.). Cambridge: MIT Press.

Campbell, C. S. , & Massaro, D. W. (1997). Perception of visible speech: Influence of spatial quantization. Perception, 26, 626–644.

Driver, J. (1996). Enhancement of selective listening by illusory mislocation of speech sounds due to lip-reading. Nature, 381, 66–68.

Handel, S. (1989). Listening: An introduction to the perception of auditory events. Cambridge: MIT Press.

Jack, C. E. , & Thurlow, W. R. (1973). Effects of degree of visual association and angle of displacement on the "ventriloquism" effect. Perceptual and Motor Skills, 37, 967–979.

Kryter, K. D. (1972). Speech communications. In H. P. Van Cott & R. G. Kinkade (Eds.), Human engineering guide to system design (pp. 161–226). Washington, DC: U.S. Government Printing Office.

Lansing, C. R. , & McConkie, G. (1994). A new method for speech-reading research: Tracking observer's eye movements. Journal of the Academy of Rehabilitative Audiology, 27, 25–43.

Lee, M. D. (2001). Multichannel auditory search: Toward understanding control processes in polychotic auditory listening. Human Factors, 43, 328–342.

10.

MacDonald, J. , Andersen, S. , & Bachmann, T. (2000). Hearing by eye: How much spatial degradation can be tolerated? Perception, 29, 1155–1168.

11.

MacLeod, A. , & Summerfield, Q. (1987). A procedure for measuring auditory and audiovisual speech-reception thresholds for sentences in noise: Rationale, evaluation, and recommendations for use. British Journal of Audiology, 24, 29–43.

12.

Massaro, D. W. (1998). Perceiving talking faces: From speech perception to a behavioral principle. Cambridge: MIT Press.

13.

McGurk, H. , & MacDonald, J. (1976). Hearing lips and seeing voices. Nature, 264, 747–748.

14.

Middlebrooks, J. C. , & Green, D. M. (1991). Sound localization by human listeners. Annual Review of Psychology, 42, 135–159.

15.

Proctor, R. W. , & Van Zandt, T. (1994). Human factors in simple and complex systems. Boston: Allyn & Bacon.

16.

Rosenblum, L. D. , & Saldaña, H. M. (1996). An audiovisual test of kinematic primitives for visual speech perception. Journal of Experimental Psychology: Human Perception and Performance, 22, 318–331.

17.

Thurlow, W. R. , & Jack, C. E. (1973). Certain determinants of the "ventriloquism effect". Perceptual and Motor Skills, 36, 1171–1184.

18.

Vatikiotis-Bateson, E. , Eigsti, I.-M. , Yano, S. , & Munhall, K. G. (1998). Eye movement of perceivers during audiovisual speech perception. Perception and Psychophysics, 60, 926–940.

19.

Viktovitch, M. , & Barber, P. (1994). Effect of video frame rate on subjects' ability to shadow one of two competing verbal passages. Journal of Speech & Hearing Research, 37, 1204–1210.

20.

Wickens, T. D. (2002) Elementary signal detection theory. Oxford, England: Oxford University Press.

21.

Yost, W. A. (1997). The cocktail party problem: Forty years later. In R. H. Gilkey , & T. R. Anderson (Eds.), Binaural and spatial hearing in real and virtual environments (pp. 329–348). Mahwah, NJ: Erlbaum.