Evaluation of methods and techniques for conversational and multimodal spoken dialogue systems is complex, as is gathering data for the modeling and tuning of such techniques. This article describes MushyPeek, an experiment framework that allows us to manipulate the audiovisual behavior of interlocutors in a setting similar to face-to-face human—human dialogue. The setup connects two subjects to each other over a Voice over Internet Protocol (VoIP) telephone connection and simultaneously provides each of them with an avatar representing the other. We present a first experiment which inaugurates, exemplifies, and validates the framework. The experiment corroborates earlier findings on the use of gaze and head pose gestures in turn-taking.
Anderson, A., Bader, M., Bard, E., Boyle, E., Doherty, G., Garrod, S., ET AL. (1991). The HCRC Map Task corpus. Language and Speech, 34(4), 351-366.
2.
Argyle, M., Lefebvre, L., & Cook, M. (1974). The meaning of five patterns of gaze. European Journal of Social Psychology, 4, 125-136.
3.
Bailenson, J.N., Beall, A.C., Loomis, J., Blascovich, J., & Turk, M. (2004). Transformed social interaction: Decoupling representation from behavior and form in collaborative virtual environments. Presence: Teleoperators & Virtual Environments, 13(4), 428-441.
4.
Bailenson, J.N., Yee, N., Merget, D., & Schroeder, R. (2006). The effect of behavioral realism and form realism of real-time avatar faces on verbal disclosure, nonverbal disclosure, emotion recognition, and copresence in dyadic interaction. Presence , 15(4), 359-372.
5.
Bechet, F., Riccardi, G., & Hakkani-Tur, D. (2004). Mining spoken dialogue corpora for system evaluation and modeling. In D. Lin & D. Wu (Eds.), Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing (pp.134-141). Barcelona: ACL .
6.
Beskow, J., Karlsson, I., Kewley, J., & Salvi, G. (2004). SYNFACE-A talking head telephone for the hearing-impaired . In K. Miesenberger, J. Klaus, W. Zagler, & D. Burger (Eds.), Computers helping people with special needs (pp.1178-1186). Berlin/ Heidelberg: Springer-Verlag.
7.
Cassell, J. (2007). Body language: Lessons from the near-human. In J. Riskin (Ed.), Genesis redux: Essays on the history and philosophy of artificial life (pp.346-374). Chicago: University of Chicago Press.
8.
Cassell, J., Bickmore, T., Campbell, L., Vilhjalmsson, H., & Yan, H. (2001). Human conversation as a system framework: designing embodied conversational agents. In J. Cassell , J. Sullivan, S. Prevost, & E. Churchill (Eds.), Embodied conversational agents (pp.29-63). Cambridge, MA: MIT Press.
9.
Dybkjær, L., Bernsen, N.O., & Minker, W. (2004). Evaluation and usability of multimodal spoken dialogue systems. Speech Communication, 43, 33-54.
10.
Edlund, J., Gustafson, J., Heldner, M., & Hjalmarsson, A. (2008). Towards human-like spoken dialogue systems. Speech Communication, 50(8-9), 630-645.
11.
Edlund, J., & Heldner, M. (2005). Exploring prosody in interaction control. Phonetica, 62(2-4), 215-226.
12.
Edlund, J., & Heldner, M. (2006). /nailon/-software for online analysis of prosody . In R. M. Stern (Ed.), Proceedings of Interspeech 2006 ICSLP (pp.2022-2025). Pittsburgh, PA.
13.
Edlund, J., & Nordstrand, M. (2002). Turn-taking gestures and hour-glasses in a multimodal dialogue system. In L. Dybkjær, E. André, W. Minker, & P. Heisterkamp (Eds.), Proceedings of ISCA Workshop Multi-modal Dialogue in Mobile Environments [CD-ROM]. Kloster Irsee: ISCA.
14.
Eklund, R. (2004). Disfluency in Swedish human-human and human-machine travel booking dialogues. Doctoral dissertation, Linköping University, Linköping, Sweden.
15.
Fisher, C. (1968). Confusions among visually perceived consonants . Journal of Speech and Hearing Research, 11, 796-804.
16.
Georgila, K., Henderson, J., & Lemon, O. (2006). User simulation for spoken dialogue systems: learning and evaluation. In ISCA Editorial Staff (Ed.), INTERSPEECH 2006 and 9th International Conference on Spoken Language Processing (Interspeech 2006-ICSLP) (pp.1065-1068. Pittsburgh, PA: ISCA/Curran Associates.
17.
Glass, J., Polifroni, J., Seneff, S., & Zue, V. (2000). Data collection and performance evaluation of spoken dialogue systems: The MIT experience. In Yuan, B., Huang, T., and Tang, X. (Eds.), Proceedings of the Sixth International Conference on Spoken Language Processing (ICSLP 2000) (pp.931-934), Beijing: China Military Friendship.
18.
Gratch, J., Okhmatovskaia, A., Lamothe, F., Marsella, S., Morales, M., Van Der Werf, R.J., ET AL. (2006). Virtual rapport. In J. G. Carbonell and J. Siekmann (Eds.), Proceedings of 6th International Conference on Intelligent Virtual Agents (pp.14-27). Berlin/ Heidelberg : Springer-Verlag.
19.
Healey, P.G.T., & Mills, G. (2006). Clarifying spatial descriptions: Local and global effects on semantic coordination. In D. Schlangen & R. Fernandez (Eds.), Proceedings of the 10th Workshop on the Semantics and Pragmatics of Dialogue (SemDial-10) (pp.122-129). Postdam: Potsdam Universitätsverlag.
20.
Hugot, V. (2007). Eye gaze analysis in human-human interactions . Unpublished master's thesis, CSC, KTH, Sweden.
21.
Jaffe, J., & Feldstein, S. (1970). Rythms of dialogue. New York : Academic Press.
22.
Kendon, A. (1967). Some functions of gaze direction in social interaction . Acta Psychologica, 26, 22-63.
23.
Kitawaki, N., & Itoh, K. (1991). Pure delay effects on speech quality in telecommunications . IEEE Journal on Selected Areas in Communications, 9(4), 586-593.
24.
Lemon, O., Cavedon, L., & Kelly, B. (2003). Managing dialogue interaction: A multilayered approach. In A. Kurematsu (Ed.), Proceedings of the 4th SIGdial Workshop on Discourse and Dialogue (pp.168-177). Morristown, NJ: Association for Computational Linguistics .
25.
McGurk, H., & MacDonald, J. (1976). Hearing lips and seeing voices. Nature, 264(5588), 746-748.
26.
Mori, M. (1970). The uncanny valley. Energy, 7(4), 33-35.
27.
Nakano, Y., Reinstein, G., Stocky, T., & Cassell, J. (2003). Towards a model of face-to-face grounding. In D. Yarowsky and S. Kurohashi (Eds.), Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL 2003) (pp.553-561). Morristown, NJ: Association for Computational Linguistics .
28.
Pelachaud, C., Badler, N., & Steedman, M. (1996). Generating facial expressions for speech. Cognitive Science, 20(1).
29.
Purver, M., Healey, P.G., King, J., Ginzburg, J., & Mills, G.J. (2003, July). Answering clarification questions. Paper presented at the 4th SIGdial Workshop on Discourse and Dialogue , Sapporo, Japan.
30.
Reeves, B., & Nass, C. (1996). The media equation. Stanford, CA: CSLI Publications.
31.
Schatzmann, J., Georgila, K., & Young, S. (2005). Quantitative evaluation of user simulation techniques for spoken dialogue systems. In L. Dybkjaer , & W. Minker (Eds.), SIGdial Workshop on Discourse and Dialogue. Lisbon: ACL.
32.
Schlangen, D., & Fernández, R. (2007). Speaking through a noisy channel: Experiments on inducing clarification behaviour in human-human dialogue. In H. van Hamme & R. van Son (Eds.), Proceedings of Interspeech 2007 (pp.1266-1269). Antwerp: ISCA.
33.
Thórisson, K.R. (1999). A mind model for multimodal communicative creatures and humanoids. International Journal of Applied Artificial Intelligence, 13(4-5), 449-486.
34.
Walker, M., Kamm, C., & Litman, D. (2000). Towards developing general models of usability with PARADISE. Natural Language Engineering, 6, 363-377.
35.
Wooffitt, R., Fraser, N.M., Gilber, N., & McGlashan, S. (1997). Humans, computers and wizards. London/New York: Routledge.
