Identifying whether an utterance is a statement, question, greeting, and so forth is integral to effective automatic understanding of natural dialog. Little is known, however, about how such dialog acts (DAs) can be automatically classified in truly natural conversation. This study asks whether current approaches, which use mainly word information, could be improved by adding prosodic information.
The study is based on more than 1000 conversations from the Switchboard corpus. DAs were hand-annotated, and prosodic features (duration, pause, F0, energy, and speaking rate) were automatically extracted for each DA. In training, decision trees based on these features were inferred; trees were then applied to unseen test data to evaluate performance. Performance was evaluated for prosody models alone, and after combining the prosody models with word information—either from true words or from the output of an automatic speech recognizer.
For an overall classification task, as well as three subtasks, prosody made significant contributions to classification. Feature-specific analyses further revealed that although canonical features (such as F0 for questions) were important, less obvious features could compensate if canonical features were removed. Finally, in each task, integrating the prosodic model with a DA-specific statistical language model improved performance over that of the language model alone, especially for the case of recognized words. Results suggest that DAs are redundantly marked in natural conversation, and that a variety of automatically extractable prosodic features could aid dialog processing in speech applications.
AYERS, G. M. (1994). Discourse functions of pitch range in spontaneous and read speech. In Working Papers in Linguistics No. 44 (pp. 1-49). Ohio State University.
2.
BENNACEF, S. K., NÉEL, F., & MAYNARD, H. B. (1995). An oral dialog model based on speech acts categorization. In P. Dalsgaard, L. Larsen, L. Boves, & I. Thomsen (Eds.), ESCA Workshop on Spoken Dialog Systems—Theories and applications (pp. 237–240). Vigsø, Denmark.
3.
BLAAUW, E. (1995). On the perceptual classification of spontaneous and read speech. Doctoral dissertation, Utrecht University. Published by LEd, Utrecht, The Netherlands.
4.
BREIMAN, L., FRIEDMAN, J. H., OLSHEN, R. A., & STONE, C. J. (1984). Classification and regression trees. Pacific Grove, CA: Wadsworth and Brooks.
5.
BRIDLE, J. (1990). Probabilistic interpretation of feedforward classification network outputs, with relationships to statistical pattern recognition. In F. Soulie & J. Herault (Eds.), Neurocomputing: Algorithms, architectures and applications (pp. 227–236). Berlin: Springer.
6.
BRUCE, G., GRANSTRÖM, B., GUSTAFSON, K., HORNE, M., HOUSE, D., & TOUATI, P. (1997). On the analysis of prosody in interaction. In Y. Sagisaka, N. Campbell, & N. Higuchi (Eds.), Computing prosody: Computational models for processing spontaneous speech (pp. 42–59). New York: Springer.
7.
CARLETTA, J. (1996). Assessing agreement on classification tasks: The Kappa statistic. Computational Linguistics, 22(2), 249–254.
8.
CARLETTA, J., ISARD, A., ISARD, S., KOWTKO, J., DOHERTY-SNEDDON, G., & ANDERSON, A. H. (1995). The coding of dialog structure in a corpus. In J. Andernach, S. van de Burgt, & G. van der Hoeven (Eds.), Proceedings of the Ninth Twente Workshop on Language Technology: Corpusbased approaches to dialog modeling. Universiteit Twente, Enschede.
9.
CHU-CARROLL, J. (1998). A statistical model for discourse act recognition in dialog interactions. In J. Chu-Carroll & N. Green (Eds.), Applying machine learning to discourse processing. Papers from the 1998 AAAI Spring Symposium. Technical Report SS-98-01 (pp. 12-17). Menlo Park, CA: AAAI Press.
10.
CLARK, H. (1996). Using language. Cambridge, U.K.: Cambridge University Press.
11.
COHEN, J. (1960). A coefficient of agreement for nominal scales. Educational and psychological measurement, 20, 37–46.
12.
CORE, M., & ALLEN, J. (1997). Coding dialogs with the DAMSL annotation scheme. In AAAI Fall Symposium on Communicative Action in Humans and Machines. Cambridge, MA. (http://www.cs.rochester.edu/u/trains/annotation/)
13.
DALY, N. A., & ZUE, V. W. (1992). Statistical and linguistic analyses of F0 in read and spontaneous speech. In J. J. Ohala, T. M. Nearey, B. L. Derwing, M. M. Hodge, & G. E. Wiebe (Eds.), Proceedings of the International Conference on Spoken Language Processing (Vol. 1, pp. 763-766). Banff, Canada.
14.
FINKE, M., LAPATA, M., LAVIE, A., LEVIN, L., TOMOKIYO, L. M., POLZIN, T., RIES, K., WAIBEL, A., & ZECHNER, K. (1998). Clarity: Inferring discourse structure from speech. In J. Chu-Carroll & N. Green (Eds.), Applying machine learning to discourse processing. Papers from the 1998 AAAI Spring Symposium. Technical Report SS-98-01 (pp 25-32). Menlo Park, CA: AAAI Press.
15.
GODFREY, J., HOLLIMAN, E., & McDANIEL, J. (1992). SWITCHBOARD: Telephone speech corpus for research and development. Proceedings of the IEEE Conference on Acoustics, Speech, and Signal Processing (pp. 517-520). San Francisco.
16.
GROSZ, B., & HIRSCHBERG, J. (1992). Some intonational characteristics of discourse structure. In J. J. Ohala, T. M. Nearey, B. L. Derwing, M. M. Hodge, & G. E. Wiebe (Eds.), Proceedings of the International Conference on Spoken Language Processing (Vol. 1, pp. 429-432). Banff, Canada.
17.
HAAN, J., HEUVEN, V. J. van, PACILLY, J. J. A., & BEZOOIJEN, R. van, (1997a). Intonational characteristics of declarativity and interrogativity in Dutch: A comparison. In A. Botonis, G. Kouroupetroglou, & G. Carayiannis (Eds.), ESCA, Workshop on Intonation: Theory, Models and Applications (pp. 173–176). Athens, Greece.
18.
HAAN, J., van HEUVEN, V. J., PACILLY, J. J. A., & van BEZOOIJEN, R. (1997b). An anatomy of Dutch question intonation. In H. de Hoop & M. den Dikken (Eds.), Linguistics in the Netherlands. Amsterdam: John Benjamins.
19.
HIRSCHBERG, J., & NAKATANI, C. (1996). A prosodic analysis of discourse segments in direction-giving monologs. Proceedings of the 34th Annual Meeting of The Association for Computational Linguistics (pp. 286-293). Santa Cruz, CA.
20.
HIRSCHBERG, J., & NAKATANI, C. H. (1998). Using machine learning to identify intonational segments. In J. Chu-Carroll & N. Green (Eds.), Applying machine learning to discourse processing. Papers from the 1998 AAAI Spring Symposium. Technical Report SS-98-01 (pp. 52-59). Menlo Park, CA: AAAI Press.
21.
JORDAN, M. I., & JACOBS, R. A. (1993). Hierarchical mixtures of experts and the EM algorithm. Neural Computation, 6(2), 181–214.
22.
JURAFSKY, D., BATES, R., COCCARO, N., MARTIN, R., METEER, M., RIES, K., SHRIBERG, E., STOLCKE, A., TAYLOR, P., & van ESS-DYKEMA, C. (1997a). Automatic detection of discourse structure for speech recognition and understanding. IEEE Workshop on Speech Recognition and Understanding. Santa Barbara, CA.
23.
JURAFSKY, D., SHRIBERG, E., & BIASCA, D. (1997b). Switchboard-DAMSL Labeling Project Coder's Manual (Tech. Rep. No. 97-02). University of Colorado Institute of Cognitive Science. (http://stripe.colorado.edu/∼jurafsky/manual.augustl.html)
24.
JURAFSKY, D., SHRIBERG, E., FOX, B., & CURL, T. (1998a). Lexical, prosodic, and syntactic cues for dialog acts. Proceedings of ACL-COLING Workshop on Discourse Relations and Discourse Markers (pp. 114-120). Montreal.
25.
JURAFSKY, D., BATES, R., COCCARO, N., MARTIN, R., METEER, M., RIES, K., SHRIBERG, E., STOLCKE, A., TAYLOR, R., & van ESS-DYKEMA, C. (1998b). Switchboard discourse language modeling project report (Research Note No. 30). Baltimore, MD: Center for Speech and Language Processing, Johns Hopkins University.
26.
KATZ, S. M. (1987). Estimation of probabilities from sparse data for the language model component of a speech recognizer. IEEE Transactions on Acoustics, Speech, and Signal Processing, 35(3), 400–401.
27.
KIESSLING, A., KOMPE, R., NIEMANN, H., NÖTH, E., & BATLINER, A. (1993). “Roger,” “Sorry,” “I'm still listening”: Dialog guiding signals in informational retrieval dialogs. In D. House & P. Touati (Eds.), ESCA, Workshop on Prosody (pp. 140–143). Lund, Sweden.
28.
KITA, K., FUKUI, Y., NAGATA, M., & MORIMOTO, T. (1996). Automatic acquisition of probabilistic dialog models. In H. T. Bunnell & W. Idsardi (Eds.), Proceedings of the International Conference on Spoken Language Processing (Vol. 1, pp. 196-199). Philadephia.
29.
KOMPE, R. (1997). Prosody in speech understanding systems. Berlin: Springer.
30.
KOMPE, R., KIESSLING, A., NIEMANN, H., NÖTH, E., SCHUKAT-TALAMAZZINI, E., ZOTTMANN, A., & BATLINER, A. (1995). Prosodic scoring of word hypotheses graphs. In J. M. Pardo, E. Enriquez, J. Ortega, J. Ferreiros, J. Macias, & F. J. Valverde (Eds.), Proceedings of the Fourth European Conference on Speech Communication and Technology (Vol. 2, pp. 1333-1336). Madrid.
31.
KOOPMANS-van BEINUM, F. J., & van DONZEL, M. E. (1996). Relationship between discourse structure and dynamic speech rate. In H. T. Bunnell & W. Idsardi (Eds.), Proceedings of the International Conference on Spoken Language Processing (Vol. 3, pp. 1724-1727). Philadephia.
32.
LEVELT, W. J. M. (1989). Speaking: From intention to articulation. Cambridge, MA: MIT Press.
33.
LITMAN, D. J., & PASSONNEAU, R. J. (1995). Combining multiple knowledge sources for discourse segmentation. Proceedings of the 33rd Annual Meeting of the Association for Computational Linguistics (pp. 108-115). Cambridge, MA.
34.
MAST, M., KOMPE, R., HARBECK, S., KIESSLING, A., NIEMANN, H., NÖTH, E., SCHUKAT-TALAMAZZINI, E. G., & WARNKE, V. (1996). Dialog act classification with the help of prosody. In H. T. Bunnell & W. Idsardi (Eds.), Proceedings of the International Conference on Spoken Language Processing (Vol. 3, pp. 1732-1735). Philadelphia.
35.
METEER, M., TAYLOR, A., MacINTYRE, R., & RUKMINI, I, (1995). Dysfluency Annotation Stylebook for the Switchboard Corpus. Linguistic Data Consortium. (Revised June 1995, by Ann Taylor.)
36.
MORGAN, N., & FOSSLER-LUSSIER, E. (1998). Combining multiple estimators of speaking rate. Proceedings of the IEEE Conference on Acoustics, Speech, and Signal Processing (Vol. 2, pp. 729-732). Seattle, Washington.
37.
MORGAN, N., FOSLER, E., & MIRGHAFORI, N. (1997). Speech recognition using on-line estimation of speaking rate. In G. Kokkinakis, N. Fakotakis, & E. Dermatas (Eds.), Proceedings of the Fifth European Conference on Speech Communication and Technology (Vol. 4, pp. 2079-2082). Rhodes, Greece.
38.
NAGATA, M., & MORIMOTO, T. (1994). First steps toward statistical modeling of dialog to predict the speech act type of the next utterance. Speech Communication, 15, 193–203.
39.
NAKAJIMA, S., & ALLEN, J. F. (1993). A study on prosody and discourse structure in cooperative dialogs. Phonetica, 50, 197–210.
40.
NAKAJIMA, S., & TSUKADA, H. (1997). Prosodic features of utterances in task-oriented dialogs. In Y. Sagisaka, N. Campbell, & N. Higuchi (Eds.), Computing prosody: Computational models for processing spontaneous speech (pp. 81–94). New York: Springer.
41.
NEUMEYER, L., & WEINTRAUB, M. (1994). Microphone-independent robust signal processing using probabilistic optimum filtering. In Proceedings of the ARPA Workshop on Human Language Technology (pp. 336-341). Plainsboro, NJ.
42.
NEUMEYER, L., & WEINTRAUB, M. (1995). Robust speech recognition in noise using adaptation and mapping techniques. Proceedings of the IEEE Conference on Acoustics, Speech, and Signal Processing. (Vol. 1, pp. 141-144). Detroit.
43.
NÜTH, E. (1991). Prosodische Information in der automatischen Spracherkenmung—Berechnung und Anwendung. Tübingen: Niemeyer.
44.
OSTENDORF, M., & ROSS, K. (1997). Multilevel recognition of intonation labels. In Y. Sagisaka, N. Campbell, & N. Higuchi (Eds.), Computing prosody: Computational models for processing spontaneous speech (pp. 291–308). New York: Springer.
45.
QUIRK, R., GREENBAUM, S., LEECH, G., & SVARTVIK, J. (1985). A comprehensive grammar of the English language. London: Longman.
46.
REITHINGER, N., ENGEL, R., KIPP, M., & KLESEN, M. (1996). Predicting dialog acts for a speech-to-speech translation system. In H. T. Bunnell & W. Idsardi (Eds.), Proceedings of the International Conference on Spoken Language Processing (Vol.2, pp. 654-657). Philadephia.
47.
REITHINGER, N., & KLESEN, M. (1997). Dialog act classification using language models. In G. Kokkinakis, N. Fakotakis, & E. Dermatas (Eds.), Proceedings of the Fifth European Conference on Speech Communication and Technology (Vol. 4, pp. 2235-2238). Rhodes, Greece.
48.
REITHINGER, N., & MAIER, E. (1995). Utilizing statistical dialogue act processing in Verbmobil. Proceedings of the 33rd Annual Meeting of the Association for Computational Linguistics (pp. 116-121). Cambridge, MA.
49.
SAMUEL, K., CARBERRY, S., & VIJAY-SHANKER, K. (1998). Computing dialog acts from features with transformation-based learning. In J. Chu-Carroll & N. Green (Eds.), Applying machine learning to discourse processing. Papers from the 1998 AAAI Spring Symposium. Technical Report SS-98-01 (pp. 90-97). Menlo Park, CA: AAAI Press.
50.
SHRIBERG, E., BATES, R., & STOLCKE, A. (1997). A prosody-only decision-tree model for disfluency detection. In G. Kokkinakis, N. Fakotakis, & E. Dermatas (Eds.), Proceedings of the Fifth European Conference on Speech Communication and Technology (Vol. 5, pp. 2383-2386). Rhodes, Greece.
51.
SIEGEL, S., & CASTELLAN, N. J., Jr. (1988). Nonparametric statistics for the behavioral sciences (Second ed.). New York: McGraw-Hill.
52.
STOLCKE, A., & SHRIBERG, E. (1996). Automatic linguistic segmentation of conversational speech. In H. T. Bunnell & W. Idsardi (Eds.), Proceedings of the International Conference on Spoken Language Processing (Vol. 2, pp. 1005-1008). Philadelphia.
53.
STOLCKE, A., SHRIBERG, E., BATES, R., COCCARO, N., JURAFSKY, D., MARTIN, R., METEER, M., RIES, K., TAYLOR, P., & van ESS-DYKEMA, C. (1998). Dialog act modeling for conversational speech. In J. Chu-Carroll & N. Green (Eds.), Applying Machine Learning to Discourse Processing. Papers from the 1998 AAAI Spring Symposium. Technical Report SS-98-01 (pp. 98-105). Menlo Park, CA: AAAI Press.
54.
SUHM, B., & WAIBEL, A. (1994). Toward better language models for spontaneous speech. Proceedings of the International Conference on Spoken Language Processing (Vol. 2, pp. 831-834). Yokohama.
55.
SWERTS, M. (1997). Prosodic features at discourse boundaries of different strength. Journal of the Acoustical Society of America, 101, 514–521.
56.
SWERTS, M., & OSTENDORF, M. (1997). Prosodic and lexical indications of discourse structure in human-machine interactions. Speech Communication, 22(1), 25–41.
57.
TAYLOR, P. A., & BLACK, A. W. (1994). Synthesizing conversational intonation from a linguistically rich input. Proceedings of the second ESCA/IEEE Workshop on Speech Synthesis (pp. 175-178). New York.
58.
TAYLOR, P. A., KING, S., ISARD, S. D., WRIGHT, H., & KOWTKO, J. (1997). Using intonation to constrain language models in speech recognition. In G. Kokkinakis, N. Fakotakis, & E. Dermatas (Eds.), Proceedings of the Fifth European Conference on Speech Communication and Technology (Vol. 5, pp. 2763-2766). Rhodes, Greece.
59.
TAYLOR, P. A., KING, S., ISARD, S. D., WRIGHT, H. (1998). Intonation and dialog context as constraints for speech recognition. Language and Speech, 41, 493–512.
60.
TERRY, M., SPARKS, R., & OBENCHAIN, P. (1994). Automated query identification in English dialogue. Proceedings of the International Conference on Spoken Language Processing (Vol. 2, pp. 891-894). Yokohama.
61.
TRAUM, D., & HEEMAN, P. (1996). Utterance units and grounding in spoken dialogue. In H. T. Bunnell & W. Idsardi (Eds.), Proceedings of the International Conference on Spoken Language Processing (Vol. 3, pp. 1884-1887). Philadephia.
62.
VAISSIÈRE, J. (1983). Language-independent prosodic features. In A. Cutler & D. R. Ladd (Eds.), Prosody: Models and measurements (pp. 53–66). Berlin: Springer.
63.
VEILLEUX, N. M., & OSTENDORF, M. (1993). Prosody/parse scoring and its application in ATIS. Proceedings of the ARPA Workshop on Human Language Technology (pp. 335-340). Princeton, NJ.
64.
WAIBEL, A. (1988). Prosody and speech recognition. San Mateo, CA: Morgan Kaufmann.
65.
WARNKE, V., KOMPE, R., NIEMANN, H., & NÖTH, E. (1997). Integrated dialog act segmentation and classification using prosodic features and language models. In G. Kokkinakis, N. Fakotakis, & E. Dermatas (Eds.), Proceedings of the Fifth European Conference on Speech Communication and Technology (Vol. 1, pp. 207-210). Rhodes, Greece.
66.
WEBER, E. (1993). Varieties of questions in English conversation. Amsterdam: John Benjamins.
67.
WIGHTMAN, C. W., & OSTENDORF, M. (1994). Automatic labeling of prosodic patterns. IEEE Transactions on Speech and Audio Processing, 2, 469–481.
68.
WOSZCZYNA, M., & WAIBEL, A. (1994). Inferring linguistic structure in spoken language. Proceedings of the International Conference on Spoken Language Processing (Vol. 2, pp. 847-850). Yokohama.
69.
YAMAOKA, T., & IIDA, H. (1991). Dialog interpretation model and its application to next utterance predication for spoken language processing. Proceedings of the Second European Conference on Speech Communication and Technology (Vol 2, pp. 849-852). Genova, Italy.
70.
YOSHIMURA, T., HAYAMIZU, S., OHMURA, H., & TANAKA, K. (1996). Pitch pattern clustering of user utterances in human-machine dialogue. In H. T. Bunnell & W. Idsardi (Eds.), Proceedings of the International Conference on Spoken Language Processing (pp. 837-840). Philadephia.
