Prosodic contrastive focus is used to attract the listener's attention to a specific part of the utterance. Mostly conceived of as auditory/acoustic, it also has visible correlates which have been shown to be perceived. This study aimed at analyzing auditory-visual perception of prosodic focus by elaborating a paradigm enabling an auditory-visual advantage measurement (avoiding the ceiling effect) and by examining the interaction between audition and vision. A first experiment proved the efficiency of a whispered speech paradigm to measure an auditory-visual advantage for the perception of prosodic features. A second experiment used this paradigm to examine and characterize the auditory-visual perceptual processes. It combined performance assessment (focus detection score) to reaction time measurements and confirmed and extended the results from the first experiment. This study showed that adding vision to audition for perception of prosodic focus can not only improve focus detection but also reduce reaction times. A further analysis suggested that audition and vision are actually integrated for the perception of prosodic focus. Visual-only perception appeared to be facilitated for whispered speech suggesting an enhancement of visual cues in whispering. Moreover, the potential influence of the presence of facial markers on perception is discussed.
Audouy, M. (2000). Traitement d'images vidéo pour la capture des mouvements labiaux. Engineering Master Thesis, Institut National Polytechnique de Grenoble, France.
2.
Bailly, G., Elisei, F., & Raidt, S., (2007). Virtual talking heads and ambiant face-to-face communication. In A. Esposito, E. Keller, M. Marinaro, & M. Bratanic (Eds.), The fundamentals of verbal and non-verbal communication and the biometrical issue (pp.302-316). Amsterdam: IOS Press.
3.
Bartels, C., & Kingston, J. (1994). Salient pitch cues in the perception of contrastive focus. In P. Bosch & R. Van Der Sandt (Eds.), Proceedings of the Conference on Focus and Natural Language Processing, Working Papers of the Institute for Logic and Linguistics (pp.94-106). Heidelberg: IBM.
4.
Baum, S.R., Kelsch Daniloff, J., Daniloff, R., & Lewis, J. (1982). Sentence comprehension by Broca's aphasics: Effects of some suprasegmental variables. Brain and Language, 17, 261-271.
5.
Bernstein, L.E., Eberhardt, S.P., & Demorest, M.E. (1989). Single-channel vibrotactile supplements to visual perception of intonation and stress. The Journal of the Acoustical Society of America, 85(1), 397-405.
6.
Beskow, J., Granström, B., & House, D. (2006). Visual correlates to prominence in several expressive modes. In Proceedings of Interspeech 2006 - ICSLP (pp.1272-1275). Pittsburgh, PA.
7.
Birch, S., & Clifton, C. Jr. (1995). Focus, accent, and argument structure: effects on language comprehension. Language and speech, 38, 365-391.
8.
Brådvik, B., Dravins, C., Holtås, S., Rosén, I., Ryding, E., & Ingvar, D. (1991). Disturbances of speech prosody following right hemisphere infarcts. Acta Neurologica Scandinavica, 84, 114-126.
9.
Bryan, K. (1989). Language prosody and the right hemisphere. Aphasiology, 3, 285-299.
10.
Cavé, C., Guaïtella, I., Bertrand, R., Santi, S., Harlay, F., & Espesser, R. (1996). About the relationship between eyebrow movements and F0 variations. Proceedings of the ICSLP 1996, 4, 2175-2179.
11.
Cho, T. ( 2005). Prosodic strengthening and featural enhancement: Evidence from acoustic and articulatory realizations of /a,i/ in English. The Journal of the Acoustical Society of America, 117(6), 3867-3878.
12.
Christophe, A., Gout, A., Peperkamp, S., & Morgan, J. (2003). Discovering words in the continuous speech stream: The role of prosody. Journal of Phonetics, 31, 585-598.
13.
Dahan, D., & Bernard, J.-M. (1996). Interspeaker variability in emphatic accent production in French. Language and Speech , 39(4), 341-374.
14.
De Jong, K. (1995). The supraglottal articulation of prominence in English: Linguistic stress as localized hyperarticulation. The Journal of the Acoustical Society of America, 97(1), 491-504.
15.
Di Cristo, A. (2000). Vers une modélisation de l'accentuation du français (deuxième partie). Journal of French Language Studies, 10, 27-44.
16.
D'Imperio, M. (2001). Focus and tonal structure in Neapolitan Italian . Speech Communication, 33(4), 339-356.
17.
Dohen, M., & Lœvenbruck, H. (2005). Audiovisual production and perception of contrastive focus in French: A multispeaker study. In Proceedings of Interspeech 2005 - EUROSPEECH (pp.2413-2416). Lisbon, Portugal.
18.
Dohen, M., Lœvenbruck, H., Cathiard, M.-A., & Schwartz, J.-L. (2004). Visual perception of contrastive focus in reiterant French speech. Speech Communication, 44, 155-172.
19.
Dohen, M., Lœvenbruck, H., & Hill, H. (2006). Visual correlates of prosodic contrastive focus in French: Description and inter-speaker variabilities. Proceedings of Speech Prosody 2006, I, 221-224.
20.
Erickson, D. (2002). Articulation of extreme formant patterns for emphasized vowels. Phonetica, 59, 134-149.
21.
Erickson, D., Maekawa, K., Hashi, M., & Dang, J. (2000). Some articulatory and acoustic changes associated with emphasis in spoken English. Proceedings of the ICSLP 2000, 3, 247-250.
22.
Graf, H.P., Cosatto, E., Strom, V., & Huang, F.J. (2002). Visual prosody: Facial movements accompanying speech. In Proceedings of the 5th IEEE International Conference on Automatic Face and Gesture Recognition (FGR'02) (pp.381-386). Washington, DC.
23.
Granström, B., & House, D. (2005). Audiovisual representation of prosody in expressive speech communication. Speech Communication, 46, 473-484.
24.
Gussenhoven, C. (1983). Testing the reality of focus domains. Language and Speech, 26(1), 61-80.
25.
Hadar, U., Steiner, T.J., Grant, E.C., & Rose, F.C. (1983). Head movement correlates of juncture and stress at sentence level. Language and Speech, 26, 117-129.
26.
Halliday, M.A.K. (1967). Intonation and grammar in British English. The Hague: Mouton.
27.
Harrington, J., Fletcher, J., & Roberts, C. (1995). Coarticulation and the accented/ unaccented distinction: Evidence from jaw movement data. Journal of Phonetics, 23, 305-322.
28.
Higashikawa, M., & Minifie, F.D. (1999). Acoustical perceptual correlates of `whispered pitch' in synthetically generated vowels. Journal of Speech, Language, and Hearing Research, 42, 583-591.
29.
Keating, P., Baroni, M., Mattys, S., Scarborough, R., Alwan, A., Auer, E.T., ET AL. (2003). Optical phonetics and visual perception of lexical and phrasal stress in English. In M. J. Solé, D. Recasens, & J. Romero (Eds.), Proceedings of the ICPhS 2003 (pp.2071-2074). Barcelona.
30.
Kehoe, M., Stoel-Gammon, C., & Buder, E.H. (1995). Acoustic correlates of stress in young children's speech. Journal of Speech, Language, and Hearing Research , 38, 338-350.
31.
Kelso, J.A.S., Vatikiotis-Bateson, E., Saltzman, E., & Kay, B.A. (1985). A qualitative dynamic analysis of reiterant speech production: Phase portraits, kinematics, and dynamic modeling. The Journal of the Acoustical Society of America, 77(1), 266-280.
32.
Konopczinsky, G. (1986). Du prélangage au langage: acquisition de la structuration prosodique. PhD thesis, Strasbourg University, France.
33.
Krahmer, E., Ruttkay, Z., Swerts, M., & Wesselink, W. (2002a). Perceptual evaluation of audiovisual cues for prominence. Proceedings of the ICSLP 2002 (pp.1933-1936). Denver, CO .
34.
Krahmer, E., Ruttkay, Z., Swerts, M., & Wesselink, W. (2002b). Pitch, eyebrows and the perception of focus. In Proceedings of Speech Prosody 2002 (pp.443-446). Aixen-Provence, France.
35.
Krahmer, E., & Swerts, M. (2006). Perceiving focus. In C.-M. Lee (Ed.), Topic and focus: A cross-linguistic perspective (pp.121-137). Dordrecht, The Netherlands: Kluwer Academic Publishers .
36.
Ladd, R.D. (1996). Intonational phonology. Cambridge : Cambridge University Press (Cambridge Studies in Linguistics).
37.
Lallouache, M.-T. (1991). Un poste visage-parole couleur. Acquisition et traitement automatique des contours de lèvres. PhD thesis, Institut National Polytechnique de Grenoble, France.
38.
Lœvenbruck, H. (1996). Pistes pour le contrôle d'un robot parlant capable de réduction vocalique. PhD thesis, Institut National Polytechnique de Grenoble, France.
39.
Lœvenbruck, H. (1999). An investigation of articulatory correlates of the accentual phrase in French. Proceedings of the 14th ICPhS, 1, 667-670.
40.
Lœvenbruck, H. (2000). Effets articulatoires de l'emphase contrastive sur la phrase accentuelle en français. In Proceedings of the XXIIIth JEP (pp.165-168). Aussois, France.
41.
MacLeod, A., & Summerfield, A.Q. (1987). Quantifying the contribution of vision to speech perception in noise. British Journal of Audiology, 21, 131-141.
42.
McGurk, H., & MacDonald, J. (1976). Hearing lips and seeing voices. Nature, 264, 746-748.
43.
Miller, G.A., & Nicely, P. (1955). An analysis of perceptual confusions among some English consonants. The Journal of the Acoustical Society of America, 27(2), 338-352.
44.
Morel, M.-A., & Danon-Boileau, L. (1998). Grammaire de l'intonation. L'exemple du français oral. Paris/ Gap France: Ophrys, Bibliothèque de Faits de Langues .
45.
Munhall, K.G., Jones, J.A., Callan, D.E., Kuratate, T., & Vatikiotis-Bateson, E. (2004). Visual prosody and speech intelligibility - head movement improves auditory speech perception. Psychological Science, 15(2), 133-137.
46.
Nlke, H. (1994). Linguistique modulaire: de la forme au sens. Louvain/Paris: Peeters.
47.
Pierrehumbert, J., & Hirshberg, J. (1990). The meaning of intonational contours in discourse . In P. Cohen, J. Morgan, & M. Pollack (Eds.), Intentions in communication (pp.271-311). Cambridge, MA: The MIT Press.
48.
Reisberg, D., Mclean, J., & Goldfield, A. (1987). Easy to hear but hard to understand: A lip-reading advantage with intact auditory stimuli. In B. Dodd & R. Campbell (Eds.), Hearing by eye: The psychology of lip-reading (pp.97-114). Hillsdale, NJ: Lawrence Erlbaum Associates.
49.
Risberg, A., & Agelfors, E. (1978). On the identification of intonation contours by hearing impaired listeners. Speech Transmission Laboratory - Quarterly Progress Report and Status Report, 19(2-3), 51-61.
50.
Risberg, A., & Lubker, J. (1978). Prosody and speechreading. Speech Transmission Laboratory - Quarterly Progress Report and Status Report, 19(4), 1-16.
51.
Rossi, M. (1999). La focalisation. In L'intonation, le système du français: description et modélisation (pp.116-128). Paris : Ophrys.
52.
Séguinot, A. (1976). L'accent d'insistance en français standard. Studia Phonetica, 12, 1-58.
53.
Selkirk, E.O. (1984). The grammar of intonation. In E. O. Selkirk (Ed.), Phonology and syntax: The relation between sound and structure (pp.197-296). Cambridge, MA: The MIT Press.
54.
Sumby, W.H., & Pollack, I. (1954). Visual contribution to speech intelligibility in noise. The Journal of the Acoustical Society of America , 26(2), 212-215.
55.
Summers, W.V. (1987). Effects of stress and final-consonant voicing on vowel production: Articulatory and acoustic analyses. The Journal of the Acoustical Society of America, 82(3), 847-863.
56.
Swerts, M., & Krahmer, E. (2004). Congruent and incongruent audiovisual cues to prominence. Proceedings of Speech Prosody 2004 (pp.69-72). Nara, Japan.
57.
Swerts, M., & Krahmer, E. (2008). Facial expression and prosodic prominence: Effects of modality and facial area. Journal of Phonetics, 36(2), 219-238.
58.
Thompson, D.M. (1934). On the detection of emphasis in spoken sentences by means of visual, tactual, and visual-tactual cues. Journal of General Psychology, 11, 160-172.
59.
Touati, P. (1987). Structures prosodiques du suédois et du français. SwedenLund University Press.
60.
Touati, P. (1989). De la prosodie française du dialogue. Rapport du projet KIPROS. Working Papers in Linguistics, 35, 203-214.
61.
Touratier, C. (2000). La sémantique. Paris: Armand Collin.
62.
Vatikiotis-Bateson, E., & Kelso, J.A.S. (1993). Rhythm type and articulatory dynamics in English, French and Japanese. Journal of Phonetics, 21, 231-265.
63.
Vihman, M.M. (1996). Phonological development: The origins of language in the child. Oxford: Blackwell .
64.
Weintraub, S., Mesulam, M.-M., & Krahmer, L. (1981). Disturbances in prosody: A right-hemisphere contribution to language. Archives of Neurology, 38, 742-744.
