Adaptive estimator design for unstable output error systems: A test problem and traditional system identification based analysis

Abstract

A key open question in adaptive estimator design is how to assure that the parameters of the proposed algorithms are converging to their almost correct solutions; hence, the learning algorithm is unbiased. Moreover, determining the speed of parameter convergence is important as it provides insight about the performance of the learning algorithms. The main contributions of the article are fourfold: the first one is that the article, initially, introduces an adaptive estimator to learn the discounted Q-function and approximate optimal control policy without requiring linear, discrete time, unstable output error system dynamics, but using only the noisy system measurements. The simulation results show that the adaptive estimator minimizes the stochastic cost function and temporal difference error and also learns the approximate Q-function together with the control policy. The second one is consideration of a different approach by taking a simple test problem to investigate issues associated with the Q-function’s representation and parametric convergence. In particular, the terminal convergence problem is analyzed with a known optimal control policy where the aim is to accurately learn only the Q-function. It is parameterized by terms which are functions of the unknown plant’s parameters and the Q-function’s discount factor, and their convergence properties are analyzed and compared with the adaptive estimator. The third one is to show that even though the adaptive estimator with a large Q-function discount factor yields larger control feedback gains, so that faster state converges upright, the learning problem is badly conditioned; hence, the parameter convergence is sluggish, as the Q-function discount factor approaches the inverse of the dominant pole of the unstable system. Finally, the fourth one is comparison of the state output learned by the adaptive estimator with the ones obtained from traditional system identification algorithms. Simulation result for a higher order unstable output error system shows that the adaptive estimator closely follows the real system output whereas the system identification algorithms do not.

Keywords

Adaptive estimator badly conditioned learning closed-loop identification discounted Q-function parameter convergence analysis unknown and unstable linear system with random output error-type noise

Get full access to this article

View all access options for this article.

References

Forssell

Ljung

. Closed-loop identification revisited. Automatica 1999; 35(7): 1215–1241.

Pannocchia

De Luca

Bottai

. Prediction error based performance monitoring, degradation diagnosis and remedies in offset-free MPC: theory and applications. Asian J Control 2014; 16(4): 995–1005.

Huang

Ding

Qin

. Closed-loop subspace identification: an orthogonal projection approach. J Process Contr 2005; 15(1): 53–66.

Tóth

Laurain

Gilson

. Instrumental variable scheme for closed-loop LPV model identification. Automatica 2012; 48(9): 2314–2320.

Tufa

Ramasamy

Shuhaimi

. Closed-loop system identification using OBF-ARMAX model. J Appl Sci 2010; 10: 3175–3182.

Tufa

Ramasamy

. Closed-loop identification of systems with uncertain time delays using ARX–OBF structure. J Process Contr 2011; 21(8): 1148–1154.

Abderrahim

Mathlouthi

Msahli

. New approaches to finite impulse response systems identification using higher-order statistics. IET Signal Process 2010; 4(5): 488–501.

Van den Hof

. Closed-loop issues in system identification. Annu Rev Control 1998; 22: 173–186.

Tufa

Ramasamy

Shuhaimi

. Improved method for development of parsimonious orthonormal basis filter models. J Process Contr 2011; 21(1): 36–45.

10.

Hashimoto

Narita

Kasahara

. Humanoid robots in Waseda University—Hadaly-2 and WABIAN. Auton Robot 2002; 12(1): 25–38.

11.

Malik

Salman

. State-space least mean square. Digit Signal Process 2008; 18(3): 334–345.

12.

Ding

Liu

. Gradient based and least-squares based iterative identification methods for OE and OEMA systems. Digit Signal Process 2010; 20(3): 664–677.

13.

Forssell

Ljung

. Identification of unstable systems using output error and Box-Jenkins model structures. IEEE T Automat Contr 2000; 45(1): 137–141.

14.

Lewis

Syrmos

. Optimal control. 2nd ed. New York: Wiley, 1995, p.xi (541 pp.).

15.

Khan

Herrmann

Lewis

. A novel Q-learning based adaptive optimal controller implementation for a humanoid robotic arm. In: Proceedings of the 18th IFAC world congress, Università Cattolica del Sacro Cuore, Milano, 28 August–2 September 2011, pp.13528–13533. Laxenburg: International Federation of Automatic Control.

16.

Zheng

Jagannathan

. Generalized Hamilton–Jacobi–Bellman formulation-based neural network control of affine nonlinear discrete-time systems. IEEE T Neural Networ 2008; 19(1): 90–106.

17.

Tassa

Erez

. Least squares solutions of the HJB equation with neural network value-function approximators. IEEE T Neural Networ 2007; 18(4): 1031–1041.

18.

Becerikli

Oysal

Konar

. Trajectory priming with dynamic fuzzy networks in nonlinear optimal control. IEEE T Neural Networ 2004; 15(2): 383–394.

19.

Modares

Lewis

Naghibi-Sistani

. Adaptive optimal control of unknown constrained-input systems using policy iteration and neural networks. IEEE T Neural Networ Learn Syst 2013; 24(10): 1513–1525.

20.

Busoniu

Ernst

De Schutter

. Approximate reinforcement learning: an overview. In: 2011 IEEE symposium on adaptive dynamic programming and reinforcement learning (ADPRL), Paris, 11–15 April 2011. New York: IEEE.

21.

Sutton

Barto

. Reinforcement learning: an introduction. Cambridge, MA: MIT Press, 1998, 322 pp.

22.

Vamvoudakis

Vrabie

Lewis

. Online adaptive algorithm for optimal control with integral reinforcement learning. Int J Robust Nonlin 2014; 24: 2686–2710.

23.

Vrabie

Pastravanu

Abu-Khalaf

. Adaptive optimal control for continuous-time linear systems based on policy iteration. Automatica 2009; 45(2): 477–484.

24.

Vrabie

Pastravanu

Lewis

. Policy iteration for continuous-time systems with unknown internal dynamics. In: Mediterranean conference on control and automation 2007 (MED’07), Athens, 27–29 June 2007. New York: IEEE.

25.

Busoniu

Babuska

Schutter

. Reinforcement learning and dynamic programming using function approximators (ed Ge

Lewis

). Boca Raton, FL: CRC Press, 2010, 270 pp.

26.

Watkins

. Learning from delayed rewards. Cambridge: University of Cambridge, 1989.

27.

Werbos

. Approximate dynamic programming for real-time control and neural modeling. In: White

Sofge

(eds) Handbook of intelligent control. New York: Van Nostrand Reinhold, 1992, pp.493–526.

28.

Vamvoudakis

Lewis

. Online actor–critic algorithm to solve the continuous-time infinite horizon optimal control problem. Automatica 2010; 46(5): 878–888.

29.

Al-Tamimi

Lewis

Abu-Khalaf

. Model-free Q-learning designs for linear discrete-time zero-sum games with application to H-infinity control. Automatica 2007; 43(3): 473–481.

30.

Jagannathan

Lewis

. Stochastic optimal control of unknown linear networked control system in the presence of random delays and packet losses. Automatica 2012; 48(6): 1017–1030.

31.

Tutsoy

. Design and comparison base analysis of adaptive estimator for completely unknown linear systems in the presence of OE noise and constant input time delay. Asian J Control. Epub ahead of print 15 July 2015. DOI: 10.1002/asjc.1184.

32.

Hongbin

Kai-Yew

. Adaptive control for a discrete-time first-order nonlinear system with both parametric and non-parametric uncertainties. In: 46th IEEE conference on decision and control, New Orleans, LA, 12–14 December 2007. New York: IEEE.

33.

Ljung

Gunnarsson

. Adaptation and tracking in system identification—a survey. Automatica 1990; 26(1): 7–21.

34.

Guo

. On critical stability of discrete-time adaptive nonlinear control. IEEE T Automat Contr 1997; 42(11): 1488–1499.

35.

Boskovic

. Stable adaptive control of a class of first-order nonlinearly parameterized plants. IEEE T Automat Contr 1995; 40(2): 347–350.

36.

Loh

Fong

. Adaptive control of discrete time systems with concave/convex parametrizations. IEEE T Automat Contr 2003; 48(6): 1069–1072.

37.

. Finite-model adaptive control using WLS-like algorithm. Automatica 2007; 43(4): 677–684.

38.

Salgado

Goodwin

Middleton

. Modified least squares algorithm incorporating exponential resetting and forgetting. Int J Control 1988; 47(2): 477–491.

39.

Lum

K-Y

. Adaptive estimation and control for systems with parametric and nonparametric uncertainties (ed K

You

). Vienna: I-Tech Education and Publishing, 2009.

40.

. Several algorithms for finite-model adaptive control. Math Control Signal 2008; 20(3): 271–303.

41.

Iida

Kuwayama

Kanoh

. A dynamic allocation method of basis functions in reinforcement learning. In: Webb

(eds) AI 2004: advances in artificial intelligence. Berlin, Heidelberg: Springer, 2005, pp.272–283.

42.

Tutsoy

Brown

. An analysis of value function learning with piecewise linear control. J Exp Theor Artif In. Epub ahead of print 19 March 2015. DOI: 10.1080/0952813X.2015.1020517.

43.

Tutsoy

Brown

. Chaotic dynamics and convergence analysis of temporal difference algorithms with bang-bang control. Optim Contr Appl Met. Epub ahead of print 20 January 2015. DOI: 10.1002/oca.2156.

44.

Tutsoy

Brown

. Reinforcement learning analysis for a minimum time balance problem. T I Meas Control. Epub ahead of print 27 April 2015. DOI: 10.1177/0142331215581638.

45.

Cai

Manteuffel

McCormick

. First-order system least squares for second-order partial differential equations: part II. SIAM J Numer Anal 1997; 34(2): 425–454.

46.

Morimoto

Doya

. Reinforcement learning of dynamic motor sequence: learning to stand up. In: 1998 IEEE/RSJ international conference on intelligent robots and systems, Victoria, BC, Canada, 13–17 October 1998. New York: IEEE.

47.

Kenji

. Reinforcement learning in continuous time and space. Cambridge, MA: MIT Press, 2000, pp.219–245.

48.

Ding

. Performance analysis of the auxiliary model-based least-squares identification algorithm for one-step state-delay systems. Int J Comput Math 2012; 89(15): 2019–2028.

49.

Dugard

Landau

. Recursive output error identification algorithms theory and evaluation. Automatica 1980; 16(5): 443–462.

50.

Ljung

McKelvey

. Subspace identification from closed loop data. Signal Process 1996; 52(2): 209–215.

51.

Ljung

(ed.). System identification: theory for the user. 2nd ed. Upper Saddle River, NJ: Prentice Hall PTR, 1999, 609 pp.

52.

Michaud

Fontaine

Lecomte

. ARMAX model and recursive least-squares identification for DOI measurement in PET. In: IEEE nuclear science symposium conference record, Portland, OR, 19–25 October 2003. New York: IEEE.