This article develops a model-free adaptive optimal control policy for discrete-time Markov jump systems. First, a two-player zero-sum game is formulated to obtain an optimal control policy that minimizes a cost function against the worst-case disturbance. Second, an action and mode-dependent value function is set up for zero-sum game to search such a policy with convergence guarantee rather than solving an optimization problem satisfying coupled algebraic Riccati equations. To be specific, motivated by the Bellman optimal principle, we develop an online value iterations algorithm to solve the zero-sum game, which is learning while controlling without any initial stabilizing policy. By this algorithm, we can achieve disturbance attenuation for Markov jump systems without knowledge of the system matrices. The adaptivity to slowly changing uncertainties can also be achieved due to the model-free feature and policy convergence. Finally, the effectiveness and practical potential of the algorithm are demonstrated by considering two numerical examples and a solar boiler system.
ZhuJYuXZhangT, et al. The mean-square stability probability of H∞ control of continuous Markovian jump systems. IEEE Trans Autom Control2015; 61: 1918–1924.
3.
ChenGSunJChenJ. Mean square exponential stabilization of sampled-data Markovian jump systems. Int J Rob Nonlin Control2018; 28: 5876–5894.
4.
MoonJ. A sufficient condition for linear-quadratic stochastic zero-sum differential games for Markov jump systems. IEEE Trans Autom Control2018; 64: 1619–1626.
5.
CostaOLde OliveiraA. Optimal mean-variance control for discrete-time linear systems with Markovian jumps and multiplicative noises. Automatica2012; 48: 304–315.
6.
LongYYangGH. Fault detection for a class of nonhomogeneous Markov jump systems based on delta operator approach. Proc IMechE, Part I: J Systems and Control Engineering2013; 227: 129–141.
7.
DongLLiWMWangCH, et al. Gyro motor fault classification model based on a coupled hidden Markov model with a minimum intra-class distance algorithm. Proc IMechE, Part I: J Systems and Control Engineering2020; 234: 646–661.
8.
WangAMLiJN. Event-triggered asynchronous H∞ fault-tolerant control for discrete-time Markov jump system with actuator faults. Proc IMechE, Part I: J Systems and Control Engineering2021; 235: 781–794.
9.
YangTZhangLLamHK. H∞ fuzzy control of semi-Markov jump nonlinear systems under σ-error mean square stability. Int J Syst Sci2017; 48: 2291–2299.
10.
MaKZhuangGXiaJ, et al. Observer-based non-fragile mixed passivity and H∞ feedback control for fuzzy stochastic jump system subject to quantized measurements. Int J Control Autom Syst2021; 19: 3136–3149.
11.
FengYGaoX. Constrained model predictive control for nonlinear Markov jump system with persistent disturbance via quadratic boundedness. IEEE Access2020; 8: 168273–168281.
12.
ZhangYLimCCLiuF. Robust mixed H2/H∞ model predictive control for Markov jump systems with partially uncertain transition probabilities. J Frankl Inst2018; 355: 3423–3437.
13.
ChengPHeSLuanX, et al. Finite-region asynchronous H∞ control for 2D Markov jump systems. Automatica2021; 129: 109590.
14.
ZhaiDAnLYeD, et al. Adaptive reliable H∞ static output feedback control against Markovian jumping sensor failures. IEEE Trans Neur Netw Learn Syst2017; 29: 631–644.
15.
ShakeriNRahmaniZRanjbar NoeiA, et al. Direct methanol fuel cell modeling based on the norm optimal iterative learning control. Proc IMechE, Part I: J Systems and Control Engineering2021; 235: 68–79.
16.
DuYJiangBMaY. Policy iteration based online adaptive optimal fault compensation control for spacecraft. Int J Control Autom Syst2021; 19: 1607–1617.
17.
LiangYZhangHXiaoG, et al. Reinforcement learning-based online adaptive controller design for a class of unknown nonlinear discrete-time systems with time delays. Neur Comput Appl2018; 30: 1733–1745.
18.
GuoLRizviSAALinZ. Optimal control of a two-wheeled self-balancing robot by reinforcement learning. Int J Robust Nonlin Control2021; 31: 1885–1904.
19.
HeSFangHZhangM, et al. Adaptive optimal control for a class of nonlinear systems: the online policy iteration approach. IEEE Trans Neur Netw Learn Syst2019; 31: 549–558.
20.
HeSSongJDingZ, et al. Online adaptive optimal control for continuous-time Markov jump linear systems using a novel policy iteration algorithm. IET Control Theo Appl2015; 9: 1536–1543.
21.
SongJHeSLiuF, et al. Data-driven policy iteration algorithm for optimal control of continuous-time Itô stochastic systems with Markovian jumps. IET Control Theor Appl2016; 10: 1431–1439.
22.
SongJHeSDingZ, et al. A new iterative algorithm for solving H∞ control problem of continuous-time Markovian jumping linear systems based on online implementation. Int J Robust Nonlin Control2016; 26: 3737–3754.
23.
HeSZhangMFangH, et al. Reinforcement learning and adaptive optimization of a class of Markov jump systems with completely unknown dynamic information. Neur Comput Appl2020; 32: 14311–14320.
24.
FangHZhuGStojanovicV, et al. Adaptive optimization algorithm for nonlinear Markov jump systems with partial unknown dynamics. Int J Robust Nonlin Control2021; 31: 2126–2140.
25.
XuYChenX. Reinforcement learning-based direct adaptive optimal control of JLQ model. Control Decis2008; 23: 1359–1362.
26.
CostaOLAyaJC. Monte Carlo TD (λ)-methods for the optimal control of discrete-time Markovian jump linear systems. Automatica2002; 38: 217–225.
27.
BeirigoRLTodorovMGBarretoADMS. Online TD(A) for discrete-time Markov jump linear systems. In: Proceedings of the 2018 IEEE conference on decision and control (CDC), Miami, FL, 17–19 December 2018, pp.2229–2234. New York: IEEE.
28.
ChenYWenJLuanX, et al. H∞ optimal control for semi-Markov jump linear systems via TP-free temporal difference (λ) learning. Int J Robust Nonlin Control2021; 31: 6905–6916.
29.
Al-TamimiALewisFLAbu-KhalafM. Model-free Q-learning designs for linear discrete-time zero-sum games with application to H-infinity control. Automatica2007; 43: 473–481.
30.
GoodwinGCSinKS. Adaptive filtering prediction and control. North Chelmsford, MA: Courier Corporation, 2014.
31.
BrewerJ. Kronecker products and matrix calculus in system theory. IEEE Trans Circ Syst1978; 25: 772–781.
32.
ZhuJZhangGXieW. Jump linear quadratic Gaussian problem for a class of nonhomogeneous Markov jump linear systems. Optim Control Appl Meth2017; 38: 668–680.
