Biologically inspired scheme for continuous-time approximate dynamic programming

Abstract

Control system theory has been based on certain well understood and accepted techniques, such as transfer function-based methods, adaptive control, robust control, non-linear systems theory, state-space methods, etc. However, recently, the hypothesis that methods of modelling and analysis of life science systems hold out the power of significantly improving the performance of man-made control systems has gained increased interest, while becoming more and more a fact. For instance, in the last few decades many successful results were obtained by combining the potential of artificial neural networks (ANNs) with classical control structures. It has been shown that certain types of ANN can extend the capabilities of adaptive controllers by making them applicable for more complex non-linear systems, and at the same time greatly improving the system performance. In this paper, we will present a biologically inspired structure that will learn the optimal state feedback controller for a linear system, while at the same time performing continuous-time online control for the system at hand. Being based on a reinforcement learning technique, the optimal controller will be obtained while only making use of the input-to-state system dynamics. Mathematically speaking, the solution of the algebraic Riccati equation underlying the control problem will be obtained without making use of any knowledge of the system internal dynamics.

Keywords

Actor/Critic structures approximate dynamic programming linear quadratic regulation policy iteration.

Get full access to this article

View all access options for this article.

References

Al-Tamimi, A. , Abu-Khalaf, M. and Lewis, F.L. 2007a: Adaptive critic designs for discrete-time zero-sum games with application to H-Infinity control. IEEE Transactions on Systems, Man, and Cybernetics-Part B 37, 240-47.

Al-Tamimi, A. , Abu-Khalaf, M. and Lewis, F.L. 2007b: Model-free Q-learning designs for discrete-time zero-sum games with application to H-Infinity control. Automatica 43, 473-82.

Baird, L. 1994: Reinforcement learning in continuous-time: advantage updating . Proceedings of the International Conference on Neural Networks, Orlando, FL, June.

Barto, G. , Sutton, R.S. and Anderson, C.W. 1983: Neuronlike elements that can solve difficult learning control problems. IEEE Transactions on Systems, Man, and Cybernetics SMC- 13, 835-46.

Bertsekas, D. and Tsitsiklis, J. 1996: Neuro-dynamic programming. Athena Scientific .

Bertsekas, D.P. 2005: Dynamic programming and suboptimal control: a survey from ADP to MPC. Proceeding of CDC'05.

Bradtke, S.J. , Ydestie, B.E. and Barto, A.G. 1994: Adaptive linear quadratic control using policy iteration . Proceedings of the American Control Conference , Baltmore, MD, June, 3475-76.

Brewer, J.W. 1978: Kronecker products and matrix calculus in system theory . IEEE Transactions on Circuits and System CAS- 25, 772-81.

Doya, K. 1996: Temporal difference learning in continuous-time and space . Advances in Neural Information Processing Systems 8, 1073-79.

10.

Doya, K. 2000: Reinforcement learning in continuous-time and space. Neural Computation 12, 219-45.

11.

Ferrari, S. and Stengel, R. 2002: An adaptive critic global controller. Proceedings of the American Control Conference, Anchorage, AK, 2665-70.

12.

Galambos, R. , Makeig, S. and Talmachoff, J. 1981: A 40-Hz auditory potential recorded from the human scalp . Proceedings of the National Academy of Sciences, 78, 2643-47.

13.

Howard, R.A. 1960: Dynamic programming and Markov processes. MIT Press.

14.

Kleinman, D. 1968: On an iterative technique for Riccati equation computations . IEEE Transactions on Automatic Control 13, 114-15.

15.

Landelius, T. 1997: Reinforcement learning and distributed local model synthesis. PhD dissertation, Linköping University.

16.

Levine, D.S. and Elsberry, W.R., editors. 1997: Optimality in biological and artificial networks? Lawrence Erlbaum Associates.

17.

Levine, D.S., Brown, V.R. and Shirey, V.T., editors. 2000: Oscillations in neural systems. Lawrence Erlbaum Associates.

18.

Liu, X. and Balakrishnan, S.N. 2001: Adaptive critic based neuro-observer . Proceedings of the American Control Conference , Arlington, VA, 1616-21.

19.

Malsburg, C. von der and Schneider, W. 1986: A neural cocktail party processor. Biological Cybernetics 54, 29-40.

20.

Murray, J. , Cox, C. , Lendaris, G. and Saeks, R. 2002: Adaptive dynamic programming. IEEE Transaction on Systems , Man, and Cybernetics 32, 140-53.

21.

Prokhorov, D. and Wunsch, D. 1997: Adaptive critic designs. IEEE Transactions on Neural Networks 8, 997-1007.

22.

Singer, W. and Gray, C.M. 1995: Visual feature integration and the temporal correlation hypothesis. Annual Review of Neuroscience 18, 555-86.

23.

Soltesz, I. 2000: The interplay of intrinsic and synaptic membrane currents in delta, theta and 40-Hz oscillations. In Levine, D.S., Brown, V.R. & Shirey, V.T., editors. Oscillations in neural systems. Lawrence Erlbaum Associates.

24.

Sutton, R.S. 1988: Learning to predict by the method of temporal differences . Machine Learning 3, 9-44.

25.

Sutton, R.S. and Barto, A.G. 1998: Reinforcement learning: an introduction. MIT Press.

26.

Vrabie, D. , Abu-Khalaf, M. , Lewis, F.L. and Wang, Youyi. 2007: Continuous-time ADP for linear systems with partially unknown dynamics. Proceedings of the IEEE International Symposium on Approximate Dynamic Programming and Reinforcement Learning, 247-53.

27.

Watkins, C. 1989: Learning from delayed rewards. Ph.D. thesis, Cambridge University.

28.

Werbos, P.J. 1992: Approximate dynamic programming for real-time control and neural modeling. In White, D.A. and Sofge, D.A., editors. Handbook of intelligent control. Van Nostrand.

29.

Werbos, P.J. 1997: Optimization: a foundation for understanding consciousness . In Levine, D.S. and Elsberry, W.R., editors, Optimality in biological and artificial networks? Lawrence Erlbaum Associates.