This paper introduces an integration of reinforcement learning and behavior-based control designed to produce real-time learning in situated agents. The model layers a distributed and asynchronous reinforcement learning algorithm over a learned topological map and standard behavioral substrate to create a reinforcement learning complex. The topological map creates a small and task-relevant state space that aims to make learning feasible, while the distributed and asynchronous aspects of the architecture make it compatible with behavior-based design principles.
We present the design, implementation and results of an experiment that requires a mobile robot to perform puck foraging in three artificial arenas using the new model, random decision making, and layered standard reinforcement learning. The results show that our model is able to learn rapidly on a real robot in a real environment, learning and adapting to change more quickly than both alternatives. We show that the robot is able to make the best choices it can given its drives and experiences using only local decisions and therefore displays planning behavior without the use of classical planning techniques.
Agre, P. , & Chapman, D. (1990). What are plans for? In P. Maes (Ed.), New architectures for autonomous agents: Task-level decomposition and emergent functionality. Cambridge, MA: MIT Press .
2.
Balch, T. (1997a). Clay: Integrating motor schemas and reinforcement learning (Technical Report GIT-CC-97-11). College of Computing, Georgia Institute of Technology .
3.
Balch, T. (1997b). Integrating RL and behavior-based control for soccer . RoboCup-97: Proceedings of the First Robot World Cup Soccer Games and Conferences. Berlin: Springer-Verlag.
4.
Balch, T. (1999). Reward and diversity in multirobot foraging . In S. Sen and J. M. Vidal (Eds.), Proceedings of the IJCAI Workshop on Agents Learning About, From and With Other Agents.
5.
Barto, A. , Bradtke, S. , & Singh, S. (1995). Learning to act using real-time dynamic programming . Artificial Intelligence, 72, 81-138 .
6.
Bertsekas, D. , & Tsitsiklis, J. (1989). Parallel and distributed computation: Numerical methods. Englewood Cliffs, NJ: Prentice Hall .
7.
Brooks, R. (1987). Planning is just a way of avoiding figuring out what to do next. In R. Brooks (Ed.), Cambrian intelligence: The early history of the new AI (pp. 103-110). Cambridge, MA: MIT Press .
8.
Brooks, R. (1991a). Intelligence without representation. In J. Haugeland (Ed.), Mind design II (pp. 395-420). Cambridge, MA: MIT Press .
9.
Brooks, R. (1991b). The role of learning in autonomous robots . In M. K. Warmuth and L. G. Valiant (Eds.), Proceedings of the Fourth Annual Workshop on Computational Learning Theory (COLT ’91) (pp. 5-10 ). San Francisco, CA: Morgan Kauffman.
10.
Bryson, J. (2002). Modularity and specialized learning: Reexamining behavior-based artificial intelligence . In M. Butz , P. Gérard , & O. Sigaud (Eds.), Proceedings of the Workshop on Adaptive Behavior in Anticipatory Learning Systems. Berlin: Springer.
11.
Choset, H. , & Nagatani, K. (2001). Topological simultaneous localization and mapping (SLAM): Towards exact localization without explicit localization . IEEE Transactions on Robotics and Automation, 17(2), 125-137 .
12.
Crook, P. , & Hayes, G. (2003). Learning in a state of confusion: Perceptual aliasing in grid world navigation . In U. Nehmzow and C. Melhush (Eds.), Proceedings of the 4th British Conference on (Mobile) Robotics: Towards Intelligent Mobile Robots (TIMR 2003). London, UK: IEE.
13.
Digney, B. (1998). Learning hierarchical control structures for multiple tasks and changing environments . In R. Pfeifer , B. Blumberg , J. Meyer , & S. Wilson (Eds.), From Animals to Animats 5: Proceedings of the Fifth International Conference on Simulation of Adaptive Behavior (pp. 321-330 ). Cambridge, MA: MIT Press.
14.
Dudek, G. , Freedman, P. , & Hadjres, S. (1993). Using local information in a non-local way for mapping graph-like worlds . In R. Bajcsy (Ed.) Proceedings of the International Joint Conference of Artificial Intelligence (pp. 1639-1647 ). San Francisco, CA: Morgan Kaufmann.
15.
Grefenstette, J. , & Ramsey, C. (1992). An approach to anytime learning . In D. H. Sleeman and P. Edwards (Eds.), Proceedings of the Ninth International Conference on Machine Learning (pp. 189-195 ). San Francisco, CA: Morgan Kaufmann.
16.
Harvey, I. (1995). The artificial evolution of adaptive behaviour. D. Phil. thesis, School of Cognitive and Computing Sciences, University of Sussex.
17.
Humphrys, M. (1996). Action selection methods using reinforcement learning . In P. Maes , M. Mataric , J.-A. Meyer , J. Pollack , & S. Wilson (Eds.), From Animals to Animats 4: The Fourth International Conference on the Simulation of Adaptive Behaviour (SAB-96) (pp. 135-144 ). Cambridge, MA: MIT Press.
18.
K-Team SA (1999a). Khepera K213 vision turret user manual. Lausanne, Switzerland .
19.
K-Team SA (1999b). Khepera user manual. Lausanne, Switzerland .
20.
Kohonen, T. (1989). Self-organization and associative memory (3rd ed.). Berlin: Springer-Verlag .
21.
Konidaris, G. (2003). Behaviour-based reinforcement learning. Master’s thesis, School of Informatics, University of Edinburgh.
22.
Maes, P. , & Brooks, R. (1990). Learning to coordinate behaviors . In T. Dietterich and W. Swartout (Eds.), Proceedings of the Eighth National Conference on Artificial Intelligence (pp. 796-802 ). Cambridge, MA.
23.
Mahadevan, S. , & Connell, J. (1992). Automatic programming of behavior-based robots using reinforcement learning . Artificial Intelligence, 55(2-3), 311-365 .
24.
Marsland, S. , Shapiro, J. , & Nehmzow, U. (2002). A self-organising network that grows when required . Neural Networks, 15(8-9), 1041-1058 .
25.
Mataric, M. (1994). Reward functions for accelerated learning. In W. W. Cohen and H. Hirsh (Eds.), Proceedings of the Eleventh International Conference on Machine Learning (pp. 181-189). San Francisco, CA: Morgan Kaufmann .
26.
Mataric, M. (1997). Reinforcement learning in the multi-robot domain . Autonomous Robots, 4(1), 73-83 .
27.
Mataric, M. , & Brooks, R. (1990). Learning a distributed map representation based on navigation behaviors. In R. Brooks (Ed.), Cambrian intelligence: The early history of the new AI. Cambridge, Massachusetts: The MIT Press .
28.
Mitchell, T. (1997). Machine learning. London, UK: McGraw-Hill . 42
29.
Moriarty, D. , Schultz, A. , & Grefenstette, J. (1999). Evolutionary algorithms for reinforcement learning . Journal of Artificial Intelligence Research, 11.
30.
Schmolke, A. , & Mallot, H. (2002). Polarization compass for robot navigation . In D. Polani , J. Kim , & T. Martinetz (Eds.), The Fifth German Workshop on Artificial Life (pp. 163-167 ). Berlin: Akademische Verlagsgesellschaft Aka.
31.
Smart, W. , & Kaelbling, L. (2000). Practical reinforcement learning in continuous spaces . In P. Langley (Ed.), Proceedings of the Seventeenth International Conference on Machine Learning (pp. 903-910 ). San Francisco, CA: Morgan Kaufmann.
32.
Smith, A. J. (2002). Applications of the self-organising map to reinforcement learning . Neural Networks, 15, 1107-1124 .
33.
Stone, P. , & Veloso, M. (2000). Layered learning . In R. Lopez de Mantaras and E. Plaza (Eds.), Proceedings of the 11th European Conference on Machine Learning (pp. 369-381 ). Berlin: Springer.
34.
Sutton, R. (1990). Reinforcement learning architectures for animats . In J. Meyer , & S. Wilson (Eds.), From animals to animats: Proceedings of the International Conference on Simulation of Adaptive Behavior (pp. 288-296 ). Cambridge, MA: MIT Press.
35.
Sutton, R. (1991). Planning by incremental dynamic programming . In L. Birnbaum and G. Collins (Eds.), Proceedings of the Ninth Conference on Machine Learning (pp. 353-357 ). San Francisco, CA: Morgan Kaufmann.
36.
Sutton, R. , & Barto, A. (1998). Reinforcement learning: An introduction. Cambridge, MA: MIT Press .
37.
Toombs, S. , Phillips, W. , & Smith, L. (1998). Reinforcement landmark learning . In R. Pfeifer , B. Blumberg , J. Meyer , & S. Wilson (Eds.), From animals to animats 5: Proceedings of the Fifth International Conference on Simulation of Adaptive Behavior (pp. 205-212 ). Cambridge, MA: MIT Press.
38.
Watkins, C. , & Dayan, P. (1992). Q-learning . Machine Learning, 8, 279-292 .
39.
Whiteson, S. , & Stone, P. (2003). Concurrent layered learning . In J. S. Rosenschein , M. Woolbridge , T. Sandholm and M. Yokoo (Eds.), Proceedings of the Second International Joint Conference on Autonomous Agents and Multi-Agent Systems (pp. 193-200 ). New York, NY: ACM Press.
