In a study of optical cues to the visual perception of stress, three American English talkers spoke words that differed in lexical stress and sentences that differed in phrasal stress, while video and movements of the face were recorded. The production of stressed and unstressed syllables from these utterances was analyzed along many measures of facial movement, which were generally larger and faster in the stressed condition. In a visual perception experiment, 16 perceivers identified the location of stress in forced-choice judgments of video clips of these utterances (without audio). Phrasal stress was better perceived than lexical stress. The relation of the visual intelligibility of the prosody of these utterances to the optical characteristics of their production was analyzed to determine which cues are associated with successful visual perception. While most optical measures were correlated with perception performance, chin measures, especially Chin Opening Displacement, contributed the most to correct perception independently of the other measures. Thus, our results indicate that the information for visual stress perception is mainly associated with mouth opening movements.
Alexandre, M.-A., & Gérard, C. (2002, April). Perception of emphatic stress: Multiple regression analyses. Paper presented at Temporal Integration in Perception of Speech, Aix-en-Provence, France.
2.
Auer, E.T., JR., Bernstein, L.E., & Coulter, D.C. (1998). Temporal and spatiotemporal vibrotactile displays for voice fundamental frequency: An initial evaluation of a new vibrotactile speech perception aid with normal-hearing and hearing-impaired individuals . Journal of the Acoustical Society of America, 104, 2477-2489.
3.
Beckman, M., & Elam, G.A. (1997). Guidelines for ToBI labelling (Version 3). Unpublished manuscript, Ohio State University, USA.
4.
Beckman, M.E. (1986). Stress and non-stress accent. Dordrecht: Foris.
5.
Beckman, M.E., & Edwards, J. (1994). Articulatory evidence for differentiating stress categories. In P. A. Keating (Ed.), Papers in laboratory phonology III: Phonological structure and phonetic form (pp.7-33). Cambridge: Cambridge University Press.
6.
Bernstein, L.E., Auer, E.T., Chaney, B., Alwan, A., & Keating, P.A. (2000). Development of a facility for simultaneous recordings of acoustic, optical (3-D motion and video), and physiological speech data [abstract]. Journal of the Acoustical Society of America, 107, 2887.
7.
Bernstein, L.E., Eberhardt, S.P., & Demorest, M.E. (1989). Single-channel vibrotactile supplements to visual perception of intonation and stress. Journal of the Acoustical Society of America, 85, 397-405.
8.
Bolinger, D.L. (1958). A theory of pitch accent in English. Word, 14, 109-149.
9.
Campbell, W.N. (1993). Automatic detection of prosodic boundaries in speech. Speech Communication, 13, 343-354.
10.
Cavé, C., Guaitella, I., Bertrand, R., Santi, S., Harlay, F., & Espesser, R. (1996). About the relationship between eyebrow movements and F0 variation. ICSLP96, 4, 2175- 2179.
11.
Cho, T. ( 2005). Prosodic strengthening and featural enhancement: Evidence from acoustic and articulatory realizations of /a,i/ in English. Journal of the Acoustical Society of America, 117, 3867-3878.
12.
Cho, T. ( 2006). Manifestation of prosodic structure in articulation: Evidence from lip kinematics in English. In L. M. Goldstein, D. H. Whalen, & C. T. Best (Eds.), Laboratory phonology 8: Varieties of phonological competence (pp.519-548). Berlin/New York: Mouton de Gruyter .
13.
Choi, J-Y., Hasegawa-Johnson, M., & Cole, J. (2005). Finding intonational boundaries using acoustic cues related to the voice source. Journal of the Acoustical Society of America, 118, 2579-2587.
14.
Condon, W. (1976). An analysis of behavioral organization. Sign Language Studies, 13, 285-318.
15.
de Jong, K. (1995). The supraglottal articulation of prominence in English: Linguistic stress as localized hyperarticulation. Journal of the Acoustical Society of America, 97, 491-504.
16.
Dohen, M., Loevenbruck, H., Cathiard, M.-A., & Schwartz, J.-L. (2004a). Visual perception of contrastive focus in reiterant French speech. Speech Communication , 44, 155-172.
17.
Dohen, M., Loevenbruck, H., Cathiard, M.-A., & Schwartz, J.-L. (2004b). Identification of the possible visible correlates of contrastive focus in French. In B. Bel and I. Marlien (Eds.), Speech prosody 2004 (pp.77-80). Retrieved February 2009, from http://www.isca-speech.org/archive/sp2004
18.
Dohen, M., Loevenbruck, H., Cathiard, M.-A., & Schwartz, J.-L. (2004c). Can we see focus? A visual perception study of contrastive focus in French. In B. Bel & I. Marlien (Eds.), Speech prosody 2004 (pp.73-76). Retrieved February 2009, from http://www.iscaspeech.org/archive/sp2004
19.
Dohen, M., & Loevenbruck, H. (2005). Audiovisual production and perception of contrastive focus in French: A multispeaker study. In Interspeech 2005 (pp.2413-2416). Retrieved February 2009, from http://www.isca-speech.org/archive/interspeech_2005
20.
Dohen, M., Loevenbruck, H., & Hill, H. (2005). A multi-measurement approach to the identification of the audiovisual facial correlates of contrastive focus in French. In E. Vatikiotis-Bateson (Ed.), Auditory-visual processing workshop (pp.115-116). Retrieved February 2009, from http://www.isca-speech.org/archive/avsp05
21.
Dohen, M., Loevenbruck, H., & Hill, H. (2006). Visual correlates of prosodic contrastive focus in French: Description and inter-speaker variability. In R. Hoffmann & H. Mixdorff (Eds.), Speech prosody 2006 (pp.221-224). Retrieved February 11, 2008, from http://www.isca-speech.org/archive/sp2006
22.
Ekman, P. (1979). About brows: Emotional and conversational signals . In M. von Cranach, K. Foppa, W. Lepenies , & D. Ploog (Eds.), Human ethology: Claims and limits of a new discipline (pp.169-202). Cambridge University Press.
23.
Erickson, D. (2002). Articulation of extreme formant patterns for emphasized vowels. Phonetica, 59, 134-149.
24.
Erickson, D., Fujimura, O., & Pardo, B. (1998). Articulatory correlates of prosodic control: emotion and emphasis. Language and Speech, 41, 395-413.
25.
Fant, G., Kruckenberg, A., & Liljencrants, J. (2000). Acoustic-phonetic analysis of prominence in Swedish. In A. Botinis (Ed.), Intonation (pp.55-86). Kluwer Academic Publishers.
26.
Fry, D.B. (1958). Experiments in the perception of stress. Language and Speech, 1, 126-152.
27.
Granström, B., House, D., & Lundeberg, M. (1999). Prosodic cues to multimodal speech perception . In Proceedings of the 14th International Congress on Phonetic Sciences (pp.655-658). San Francisco.
28.
Grant, K.W., Ardell, L.A., Kuhl, P.K., & Sparks, D.W. (1986). The transmission of prosodic information via an electrotactile speech reading aid. Ear and Hearing, 7, 328-335.
29.
Hadar, U., Steiner, T.J., Grant, E.C., & Clifford Rose, F. (1983). Head movement correlates of juncture and stress at sentence level. Language and Speech , 26, 117-129.
30.
Harrington, J., Fletcher, J., & Beckman, M. (2000). Manner and place conflicts in the articulation of accent in Australian English. In M. B. Broe & J. Pierrehumbert (Eds.), Papers in laboratory phonology V (pp. 40-51). Cambridge : Cambridge University Press.
31.
Hayes, B. (1995). Metrical stress theory: Principles and case studies . Chicago: University of Chicago Press.
32.
Heldner, M. (2003). On the reliability of overall intensity and spectral emphasis as acoustic correlates of focal accents in Swedish. Journal of Phonetics, 31, 39-62.
33.
Herment-Dujardin, S., & Hirst, D. (2002). Emphasis in English: A perceptual study based on modified synthetic speech. In B. Bel & I. Marlien (Eds.), Speech prosody 2004 (pp.379-382). Retrieved February 2009, from http://www.isca-speech.org/archive/sp2004
34.
Hirst, D., & Di Cristo, A. (1998). A survey of intonation systems. In D. Hirst & A. Di Cristo (Eds.), Intonation systems: A survey of twenty languages (pp.1-44). Cambridge: Cambridge University Press.
35.
House, D., Beskow, J., & Granström, B. (2001). Timing and interaction of visual cues for prominence in audiovisual speech perception. In by P. Dalsgaard, B. Lindberg, H. Benner, & Z. Tan (Eds.), Proceedings of Eurospeech 2001 (pp.387-390). Retrieved February, 2009, from http://www.isca-speech.org/archive/eurospeech_2001
36.
Jensen, C. (2003). Perception of prominence in Standard British English . In Proceedings of the 15th International Congress on Phonetic Sciences (pp.1815-1819). Barcelona.
37.
Jiang, J., Alwan, A., Keating, P.A., Auer, E.T., & Bernstein, L.E. (2002). On the relationship between face movements, tongue movements and speech acoustics. EURASIP Journal on Applied Signal Processing, 11, 1174-1188.
38.
Jiang, J., Auer, E.T., Alwan, A., Keating, P.A., & Bernstein, L.E. (2007). Similarity structure in visual speech perception and optical phonetic signals. Perception and Psychophysics, 69, 1070-1083.
39.
Keating, P.A., Cho, T., Baroni, M., Mattys, S., Bernstein, L.E., Chaney, B., ET AL. (2000). Articulation of word and sentence stress [abstract]. Journal of the Acoustical Society of America, 108, 2466.
40.
Kendon, A. (1978). Differential perception and attentional frame: Two problems for investigation. Semiotica, 24, 305-315.
41.
Kochanski, G., Grabe, E., Coleman, J., & Rosner, B. (2005). Loudness predicts prominence: Fundamental frequency lends little. Journal of the Acoustical Society of America , 118, 1038-1054.
42.
Krahmer, E., Ruttkay, Z., Swerts, M., & Wesselink, W. (2002). Pitch, eyebrows and the perception of focus. In B. Bel & I. Marlien (Eds.), Speech prosody 2004 (pp.443-446). Retrieved February 2009, from http://www.isca-speech.org/archive/sp2004
43.
Krahmer, E., & Swerts, M. (2006). Hearing and seeing beats. In R. Hoffmann & H. Mixdorff (Eds.), Speech prosody 2006. Retrieved February 2009, from http://www.iscaspeech.org/archive/sp2006
44.
Ladd, D.R. (1996). Intonational phonology. Cambridge : Cambridge University Press.
45.
Lansing, C.R., & McConkie, G.W. (1999). Attention to facial regions in segmental and prosodic visual speech perception tasks. Journal of Speech, Language, and Hearing Research, 42, 526-539.
46.
Lehiste, I. (1970). Suprasegmentals. Cambridge, MA: The MIT Press.
47.
Massaro, D.W., & Beskow, J. (2002). Multimodal speech perception: A paradigm for speech science. In B. Granström, D. House, & I. Karlsson (Eds.), Multimodality in language and speech systems (pp.45-71). Dordrecht: Kluwer Academic Publishers.
48.
Munhall, K.G., Jones, J.A., Callan, D.E., Kuratate, T., & Vatikiotis-Bateson, E. (2004). Visual prosody and speech intelligibility: Head movement improves auditory speech perception. Psychological Science, 15, 133-138.
49.
Pentland, A.P., & Darell, T. (1994). Visual perception of human bodies and faces for multi-modal interfaces. ICSLP 94, Yokohama , 543-546.
50.
Rietveld, T., & Gussenhoven, C. (1985). On the relation between pitch excursion size and prominence. Journal of Phonetics , 13, 299-308.
51.
Risberg, A., & Lubker, J. (1978). Prosody and speechreading. Speech Transmission Laboratory-Quarterly Progress Report, Status Report, 4, 1-16.
52.
Silverman, K., Beckman, M., Pitrelli, J., Ostendorf, M., Wightman, C., Price, P., ET AL. (1992). ToBI: A standard for labeling English prosody . Proceedings of the 1992 International Conference on Spoken Language Processing, 2: 867-870.
53.
Sluijter, A.M.C., van Heuven, V.J., & Pacilly, J.J. (1997). Spectral balance as a cue in the perception of linguistic stress. Journal of the Acoustical Society of America , 101, 503-513.
54.
Srinivasan, R.J., & Massaro, D.W. (2003). Perceiving prosody from the face and voice: Distinguishing statements from echoic questions in English. Language and Speech, 46, 1-22.
55.
Swerts, M., & Krahmer, E. (2008). Facial expression and prosodic prominence: Effects of modality and facial area. Journal of Phonetics, 36, 219-238.
56.
Thompson, D.M. (1934). On the detection of emphasis in spoken sentences by means of visual, tactual, and visual-tactual cues. Journal of General Psychology, 11, 160-172.
57.
van Kuijik, D., & Boves, L. (1999). Acoustic characteristics of lexical stress in continuous telephone speech. Speech Communication, 27, 95-111.
58.
Vatikiotis-Bateson, E., Eigsti, I.-M., Yano, S., & Munhall, K.G. (1998). Eye movement of perceivers during audiovisual speech perception. Perception and Psychophysics, 60, 926-940.
59.
Yehia, H.C., Kuratate, T., & Vatikiotis-Bateson, E. (2002). Linking facial animation, head motion and speech acoustics. Journal of Phonetics, 30, 555-568.
