This paper describes a novel approach to linguistic mutual inference, which enables robots not only to linguistically interpret the motion patterns in the form of sentences but also to generate the motions from the sentences. The inference can be established based on two modules, the motion language model and the natural language model. The motion language model stochastically represents an association structure between symbols of motion patterns and the words in sentences assigned to the motion. This is a statistical model with a three layered structure of motion symbols, latent states and words. The natural language model statistically represents a structure of sentences based on word bigrams. The motion language model and the natural language model correspond to semantics and syntax respectively. An approach to the integration of motion language model with the natural language model allows the linguistic mutual inference for the robots. The two kinds of inference can be made by solving search problems, search for a sequence of words corresponding to a motion and search for a symbol of motion pattern corresponding to a sentence. The proposed approach to interpretation of motion patterns as sentences and generation of motion patterns from the sentences through the integration of motion language model with the natural language model is validated by an experiment on the human behavioral data.
ArikanOForsythDAO’BrienJF (2003) Motion synthesis from annotations. ACM Transactions on Graphics22(3): 402–408.
2.
AsfourTGyarfasFAzadPDillmannR (2006) Imitation learning of dual-arm manipulation task in humanoid robots. In: Proceedings of the IEEE-RAS international conference on humanoid robots, Genova, Italy, 4–6 December, pp. 40–47.
3.
BillardACalinonSGuenterF (2006) Discriminative and adaptive imitation in uni-manual and bi-manual tasks. Robotics and Autonomous Systems54: 370–384.
4.
DeaconT (1997) Symbolic species: The co-evolution of language and the brain. W.W. Norton and Company Inc.
5.
GalleseVGoldmanA (1998) Mirror neuron and the simulation theory of mind-reading. Trends in Cognitive Sciences2(12): 493–501.
6.
IjspeertAJNakanishiJShaalS (2003) Learning control policies for movement imitation and movement recognition. Neural Information Processing System15: 1547–1554.
7.
InamuraTToshimaITanieHNakamuraY (2004) Embodied symbol emergence based on mimesis theory. International Journal of Robotics Research23(4): 363–377.
8.
KadabaMPRamakrishnanHKWoottenME (1990) Measurement of lower extremity kinematics during level walking. Journal of Orthopaedic Research8(3): 383–392.
9.
KadoneHNakamuraY (2005) Symbolic memory for humanoid robots using hierarchical bifurcations of attractors in nonmonotonic neural networks. In: Proceedings of the 2005 IEEE/RSJ international conference on intelligent robots and systems, Edmonton, Canada, 2–6 August, pp. 2900–2905.
10.
KudoTYamamotoKMatsumotoY (2004) Applying conditional random fields to Japanese morphological analysis. In: Proceedings of the 2004 conference on empirical methods in natural language processing, Barcelona, Spain, 25–26 July, pp. 230–237.
11.
KulicDTakanoWNakamuraY (2009) On-line segmentation and clustering from continuous observation of whole body motions. IEEE Transactions on Robotics25(5): 1158–1166.
12.
NakamuraTNagaiTIwahashiN (2007) Multimodal object categorization by a robot. In: Proceedings of the IEEE/RSJ international conference on intelligent robots and systems, San Diego, USA, 29 October–2 November, pp. 2415–2420.
13.
NakamuraTNagaiTIwahashiN (2011) Bag of multimodal lda models for concept formation. In: Proceedings of the IEEE international conference on robotics and automation, Shanghai, China, 9–13 May, pp. 6233–6238.
14.
NakaokaSNakazawaAKanehiroF. (2010) Learning from observation paradigm: Leg task models for enabling a biped humanoid robot to imitate human dances. International Journal of Robotics Research26(8): 829–844.
15.
OgataTMuraseMTaniJKomataniKOkunoHG (2007) Two-way translation of compound sentences and arm motions by recurrent neural networks. In: Proceedings of the IEEE/RSJ international conference on intelligent robots and systems, San Diego, USA, 29 October–2 November, pp. 1858–1863.
16.
OkadaMTataniKNakamuraY (2002) Polynomial design of the nonlinear dynamics for the brain-like information processing of whole body motion. In: Proceedings of the IEEE international conference on robotics and automation, Washington, USA, 11–15 May, pp. 1410–1415.
17.
OttCLeeDNakamuraY (2008) Motion capture based human motion recognition and imitation by direct marker control. In: Proceedings of the IEEE-RAS international conference on humanoid robots, Daejeon, Korea, 1–3 December, pp. 399–405.
18.
RabinerLJuangBH (1993) Fundamentals of speech recognition. In: Prentice Hall Signal Processing Series.
19.
RizzolattiGFogassiLGalleseV (2001) Neurophysiological mechanisms underlying the understanding and imitation of action. Nature Reviews: 661–670.
20.
RoseCBodenheimerBCohenMF (1998) Verbs and adverbs: Multidimensional motion interpolation. IEEE Computer Graphics and Application18(5): 32–40.
21.
SaussureFD (1966) Course in general linguistics. McGraw-Hill Book Company.
22.
SugitaYTaniJ (2005) Learning semantic combinatoriality from the interaction between linguistic and behavioral processes. Adaptive Behavior13: 33–52.
23.
SugiuraKIwahashiNKashiokaHNakamuraS (2010) Active learning of confidence measure function in robot language acquisition framework. In: Proceedings of the 2010 IEEE/RSJ international conference on intelligent robots and systems, Taipei, Taiwan, 18–22 October, pp. 1774–1779.
24.
TakanoWKulicDNakamuraY (2007a) Interactive topology formation of linguistic space and motion space. In: Proceedings of the IEEE/RSJ international conference on intelligent robots and systems, San Diego, USA, 29 October–2 November, pp. 1416–1422.
25.
TakanoWNakamuraY (2009) Statistically integrated semiotics that enables mutual inference between linguistic and behavioral symbols for humanoid robots. In: Proceedings of the IEEE international conference on robotics and automation, Kobe, Japan, 12–17 May, pp. 646–652.
26.
TakanoWNakamuraY (2010) Associative processes between behavioral symbols and a large scale language model. In: Proceedings of the IEEE international conference on robotics and automation, Anchorage, USA, 3–8 May, pp. 2404–2409.
27.
TakanoWNakamuraY (2012) Bigram-based natural language model and statistical motion symbol model for scalable language of humanoid robots. In: Proceedings of the IEEE International conference on robotics and automation, St. Paul, USA, 14–18 May, pp. 1232–1237.
28.
TakanoWYamaneKNakamuraY (2007b) Capture database through symbolization, recognition and generation of motion patterns. In: Proceedings of the IEEE international conference on robotics and automation, Rome, Italy, 10–14 April, pp. 3092–3097.
29.
TakeuchiKMatsumotoY (1995) Hmm parameter learning for japanese morphological analyzer. In: Proceedings of the 10th Pacific Asia conference on language, information and computation, vHong Kong, China, 27–28 December, pp. 163–172.
30.
TaniJItoM (2003) Self-organization of behavioral primitives as multiple attractor dynamics: A robot experiment. IEEE Transactions on Systems, Man and Cybernetics Part A: Systems and Humans33(4): 481–488.
31.
YamaneKAndersonSOHodginsJK (2010) Controlling humanoid robots with human motion data: Experimental validation. In: Proceedings of the IEEE-RAS international conference on humanoid robots, Nashville, USA, 6–8 December, pp. 504–510.
