This article addresses the problem of how modular robotics systems, i.e. systems composed of multiple modules that can be configured into different robotic structures, can learn to locomote. In particular, we tackle the problems of online learning, that is, learning while moving, and the problem of dealing with unknown arbitrary robotic structures. We propose a framework for learning locomotion controllers based on two components: a central pattern generator (CPG) and a gradient-free optimization algorithm referred to as Powell's method. The CPG is implemented as a system of coupled nonlinear oscillators in our YaMoR modular robotic system, with one oscillator per module. The nonlinear oscillators are coupled together across modules using Bluetooth communication to obtain specific gaits, i.e. synchronized patterns of oscillations among modules. Online learning involves running the Powell optimization algorithm in parallel with the CPG model, with the speed of locomotion being the criterion to be optimized. Interesting aspects of the optimization include the fact that it is carried out online, the robots do not require stopping or resetting and it is fast. We present results showing the interesting properties of this framework for a modular robotic system. In particular, our CPG model can readily be implemented in a distributed system, it is computationally cheap, it exhibits limit cycle behavior (temporary perturbations are rapidly forgotten), it produces smooth trajectories even when control parameters are abruptly changed and it is robust against imperfect communication among modules. We also present results of learning to move with three different robot structures. Interesting locomotion modes are obtained after running the optimization for less than 60 minutes.
Aoi, S. and Tsuchiya, K. (2006). Stability analysis of a simple walking model driven by an oscillator with a phase reset using sensory feedback. IEEE Transactions on Robotics, 22(2): 391—397.
2.
Arena, P., Fortuna, L., Frasca, M., and Sicurella, G. (2004). An adaptive, self-organizing dynamical system for hierarchical control of bio-inspired locomotion. IEEE Transactions on Systems, Man and Cybernetics, Part B, 34(4): 1823—1837.
3.
Beer, R.D. and Gallagher, J.C. (1992). Evolving dynamical neural networks for adaptive behavior. Adaptive Behavior, 1(1): 91—122.
4.
Bongard, J., Zykov, V., and Lipson, H. (2006). Resilient machines through continuous self-modeling . Science, 314(5802): 1118—1121.
5.
Brent, R. (1973). Algorithms for Minimization without Derivatives. NJPrentice-Hall.
6.
Buchli, J., Iida, F., and Ijspeert, A. (2006a). Finding resonance: Adaptive frequency oscillators for dynamic legged locomotion. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS2006) , pp. 3903—3909. IEEE.
7.
Buchli, J., Righetti, L., and Ijspeert, A. (2006b). Engineering entrainment and adaptation in limit cycle systems — from biological inspiration to applications in robotics . Biological Cybernetics, 95(6): 645—664.
8.
Cohen, A.H., Holmes, P.J., and Rand, R. (1982). The nature of coupling between segmented oscillations and the lamprey spinal generator for locomotion: a mathematical model. Journal of Mathematical Biology, 13: 345—369.
9.
Collins, S., Ruina, A., Tedrake, R., and Wisse, M. (2005). Efficient Bipedal Robots Based on Passive-Dynamic Walkers. Science, 307(5712): 1082—1085.
10.
Conradt, J. and Varshavskaya, P. (2003). Distributed central pattern generator control for a serpentine robot. In International Conference on Artificial Neural Networks (ICANN 2003).
11.
Delcomyn, F. (1980). Neural basis for rhythmic behaviour in animals . Science, 210: 492—498.
12.
Duff, D., Yim, M., and Roufas, K. (2001). Evolution of polybot: A modular reconfigurable robot. In Proceedings of the Harmonic Drive International Symposium, Nagano, Japan, and Proceedings of COE/Super-Mechano-Systems Workshop , Tokyo, Japan, November 2001.
13.
Endo, G., Nakanishi, J., Morimoto, J., and Cheng, G. (2005). Experimental studies of a neural oscillator for biped locomotion with Q RIO. In Proceedings of the 2005 IEEE International Conference on Robotics and Automation (ICRA2005) , Barcelona, Spain, pp. 598—604.
14.
Geng, T., Porr, B., and Wörgötter, F. (2006). Fast biped walking with a reflexive neuronal controller and real-time online learning. International Journal of Robotics Research, 3: 243—261.
15.
Grillner, S. (1985). Neural control of vertebrate locomotion — central mechanisms and reflex interaction with special reference to the cat . In W. J. P. Barnes and M. H. Gladden (Eds.), Feedback and motor control in invertebrates and vertebrates, pp. 35—56. Croom Helm .
16.
Gruau, F. and Quatramaran, K. (1997). Cellular encoding for interactive evolutionary robotics. In P. Husbands and I. Harvey (Eds.), Proceedings of the Fourth European Conference on Artificial Life, ECAL97, pp. 368—377. MIT Press.
17.
Ijspeert, A. (2001). A connectionist central pattern generator for the aquatic and terrestrial gaits of a simulated salamander. Biological Cybernetics, 84(5): 331—348.
18.
Ijspeert, A. (2008). Central pattern generators for locomotion control in animals and robots: a reviewNeural Networks, (In press).
19.
Ijspeert, A. and Kodjabachian, J. (1999). Evolution and development of a central pattern generator for the swimming of a lamprey. Artificial Life, 5(3): 247—269.
20.
Ijspeert, A.J. and Crespi, A. (2007). Online trajectory generation in an amphibious snake robot using a lamprey-like central pattern generator model. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA 2007).
21.
Ijspeert, A.J., Crespi, A., Ryczko, D., and Cabelguen, J.M. (2007). From swimming to walking with a salamander robot driven by a spinal cord model. Science, 315(5817): 1416 — 1420.
22.
Kamimura, A., Kurokawa, H., Toshida, E., Tomita, K., Murata, S., and Kokaji, S. (2003). Automatic locomotion pattern generation for modular robots. In IEEE International Conference on Robotics and Automation (ICRA2003).
23.
Kamimura, A., Kurokawa, H., Yoshida, E., Murata, S., Tomita, K., and Kokaji, S. (2004). Distributed adaptive locomotion by a modular robotic system, M-TRAN II. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS2004), pp. 2370—2377.
24.
Kimura, H., Akiyama, S., and Sakurama, K. (1999). Realization of dynamic walking and running of the quadruped using neural oscillators. Autonomous Robots , 7(3): 247— 258.
25.
Kohl, N. and Stone, P. (2004). Policy gradient reinforcement learning for fast quadrupedal locomotion. In Proceedings of the 2004 IEEE International Conference on Robotics & Automation, pp. 2619—2624.
26.
Kopell, N., Ermentrout, G.B., and Williams, T.L. (1991). On chains of oscillators forced at one end. SIAM, Journal of Applied Mathematics, 51(5): 1397—1417.
27.
Lachat, D., Crespi, A., and Ijspeert, A.J. (2006). Boxybot: a swimming and crawling fish robot controlled by a central pattern generator. In Proceedings of the first IEEE/RAS-EMBS International Conference on Biomedical Robotics and Biomechatronics (BioRob 2006).
28.
Lewis, M.A., Fagg, A.H., and Bekey, G.A. (1993). Genetic algorithms for gait synthesis in a hexapod robot. In Y. F. Zheng (Ed.), Recent trends in mobile robots. World Scientific.
29.
Marbach, D. and Ijspeert, A.J. (2005). Online optimization of modular robot locomotion . In Proceedings of the IEEE International Conference on Mechatronics and Automation (ICMA 2005), pp. 248—253.
30.
Matsubara, T., Morimoto, J., Nakanishi, J., Sato, M., and Doya, K. (2006). Learning CPG-based biped locomotion with a policy gradient method. Robotics and Autonomous Systems, 54: 911—920.
31.
Moeckel, R., Jaquier, C., Drapel, K., Dittrich, E., Upegui, A., and Ijspeert, A.J. (2006). Exploring adaptive locomotion with YaMoR, a novel autonomous modular robot with Bluetooth interface. Industrial Robot, 33(4): 285—290.
32.
Moeckel, R., Sproewitz, A., Maye, J., and Ijspeert, A.J. (2007). An easy to use bluetooth scatternet protocol for fast data exchange in wireless sensor networks and autonomous robots. In Proceedings of the 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 2801— 2806.
33.
Murata, S., Yoshida, E., Kamimura, A., Kurokawa, H., Tomita, K., and Kokaji, S. (2002). M-tran: Selfreconfigurable modular robotic system . IEEE/ASME Transactions on Mechatronics, 7(4): 431—441.
34.
Nakamura, Y., Mori, T., Sato, M., and Ishii, S. (2007). Reinforcement learning for a biped robot based on a cpg-actor-critic method. Neural Networks, 20(6): 723—735.
35.
Nishii, J., Uno, Y., and Suzuki, R. (1994). Mathematical models for the swimming pattern of a lamprey, i. analysis of collective oscillators with time-delayed interaction and multiple coupling. Biological Cybernetics, 72: 1—9.
36.
Ostergaard, E.H. and Lund H.H. (2003). Evolving control for modular robotic unit. In Proceedings of CIRA'03, IEEE International Symposium on Computational Intelligence in Robotics and Automation, pp. 886—892.
37.
Paul, C. and Bongard, J.C. (2001). The road less travelled: Morphology in the optimization of biped robot locomotion. In Proceedings of The IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS2001) .
38.
Press, W., Teukolsky, S.A., Vetterling, W.T., and Flannery, B.P. (1994). Numerical recipes in C: the art of scientific computing, 2nd edition. Cambridge University Press.
39.
Righetti, L. and Ijspeert, A. (2006). Programmable central pattern generators: an application to biped locomotion control. In Proceedings of the 2006 IEEE International Conference on Robotics and Automation (ICRA2006) , pp. 1585—1590.
40.
Righetti, L. and Ijspeert, A. (2008). Pattern generators with sensory feedback for the control of quadruped locomotion. In Proceedings of the 2008 IEEE International Conference on Robotics and Automation. Accepted for publication.
41.
Salemi, B. and Shen, W. (2004). Distributed behavior collaboration for self-reconfigurable robots. In Proceedings of the IEEE Conference on Robotics and Automation (ICRA 2004).
42.
Shen, W., Salemi, B., and Will, P. (2002). Hormone-inspired adaptive communication and distributed control for selfreconfigurable robots. IEEE Transactions on Robotics and Automation, 18(5): 1—12.
43.
Shen, W., Will, P., Galstyan, A., and Chuong, C. (2004). Hormone-inspired self-organization and distributed control of robotic swarms. Autonomous Robots, 17(4): 93—105.
44.
Sigvardt, K.A. and Williams, T.L. (1996). Effects of local oscillator frequency on intersegmental coordination in the lamprey locomotor CPG: theory and experiment. Journal of Neurophysiology, 76(6): 4094— 4103.
45.
Sims, K. (1994). Evolving 3d morphology and behavior by competition . In Proceedings, Artificial Life IV, pp. 28—39. MIT Press.
46.
Stoy, K., Shen, W., and Will, P. (2003). Implementing configuration dependent gaits in a self-reconfigurable robot. In Proceedings of the IEEE International conference on Robotics and Automation (ICRA2003).
47.
Sutton, R. and Barto, A.G. (1998). Reinforcement learning: an introduction. MIT Press.
48.
Upegui, A., Moeckel, R., Dittrich, E., Ijspeert, A.J., and Sanchez, E. (2005). An fpga dynamically reconfigurable framework for modular robotics. In U. Brinkschulte (Ed.), Workshop Procedings of the 18th International Conference on Architecture of Computing Systems 2005 (ARCS'05). VDE Verlag, Berlin.
49.
Vassilvitskii, S., Kubica, J., Rieffel, E., Suh, J., and Yim.M. (2002). On the general reconfiguration problem for expanding cube style modular robots. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA2002).
50.
Vona, M. and Rus, D. (2000). A physical implementation of the self-reconfigurable crystalline robot. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA2000), pp. 1726—1733.
51.
Weingarten, J.D., Lopes, G., Buehler, M., Groff, R.E., and Koditschek, D.E. (2004, April). Automated gait adaptation for legged robots. In Proceedings of the 2004 IEEE International Conference on Robotics & Automation, pp. 2153— 2158.
52.
Williams, T.L., Sigvardt, K.A., Kopell, N., Ermentrout, G.B., and Rempler, M.P. (1990). Forcing of coupled nonlinear oscillators: studies of intersegmental coordination in the lamprey locomotor central pattern generator . Journal of Neurophysiology, 64: 862—871.
53.
Yerly, M. (2007). Yamor lifelong learning. Master's thesis, EPFL.
54.
Yim, M. (1994). Locomotion with a Unit Modular Reconfigurable Robot. Ph. D. thesis, Stanford University Mechanical Engineering Dept.
55.
Zhang, Y., Fromherz, M., Crawford, L., and Shang, Y. (2002). A general constraint-based control framework with examples in modular self-reconfigurable robots. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS2002) .
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
