This paper presents an optimal trajectory tracking control algorithm for autonomous surface vessels (ASVs) using data-driven reinforcement learning (RL) to address challenges arising from model uncertainties and time-varying external disturbances in complex marine environments. To ensure robust performance under these conditions, we first employ the H∞ control method. Then, we design a model-based RL algorithm to achieve the optimal trajectory tracking control law for the ASV despite uncertainties and disturbances. In addition, we extend the model-based RL algorithm to a model-free data-driven RL algorithm, removing the requirement for model of the ASV. The model-free algorithm directly learns the optimal control law from real-time data, providing a more flexible solution when the model of the ASV is unknown. Simulations are conducted to verify the proposed algorithms.
BuşoniuLDe SchutterBTolićD, et al. (2018) Reinforcement learning for control: Performance, stability, and deep approximators. Annual Reviews in Control46: 8–28.
2.
CaoXPengJLiuWZ, et al. (2024) AUV collision avoidance strategy based on fuzzy reinforcement learning. IEEE Transactions on Intelligent Vehicles99: 1–13.
3.
CaoXRenLSunCY (2022) Research on obstacle detection and avoidance of autonomous underwater vehicle based on forward-looking sonar. IEEE Transactions on Neural Networks and Learning Systems34(11): 9198–9208.
4.
ChenBM (2013) Robust and H∞ control. London: Springer.
5.
ChenLDaiSLDongC (2024) Adaptive optimal tracking control of an underactuated surface vessel using actor–critic reinforcement learning. IEEE Transactions on Neural Networks and Learning Systems35(6): 7520–7533.
6.
ChengMLiuHWenGH, et al. (2023) Data-driven time-varying formation-containment control for a heterogeneous air-ground vehicle team subject to active leaders and switching topologies. Automatica153: 111029.
7.
GuNWangDPengZH, et al. (2023) Safety-critical containment maneuvering of underactuated autonomous surface vehicles based on neurodynamic optimization with control barrier functions. IEEE Transactions on Neural Networks and Learning Systems34(6): 2882–2895.
8.
HasanvandASeifMS (2024) Adaptive path-following control for displacement vessels at any loading conditions under ocean disturbances. Journal of Marine Science and Application23: 900–913.
9.
HouYQYanYJiangT, et al. (2024) Distributed finite-time sliding mode containment control for multi-unmanned surface vehicles with input and output quantization. Transactions of the Institute of Measurement and Control. Epub ahead of print October 22. DOI: 10.1177/01423312241273768.
10.
HuangBPengHZhangC, et al. (2024) Distributed optimal coordinated control for unmanned surface vehicles with interleaved periodic event-based mechanism. IEEE Transactions on Vehicular Technology73(12): 18073–18086.
11.
HuangYWLiuZYHuangWC, et al. (2020) Robust H∞ control for nonlinear course system of unmanned surface vessel with polytopic uncertainty based on sum of squares. Transactions of the Institute of Measurement and Control43(2): 390–399.
12.
HuangZSLiTSLongY, et al. (2025) Observer-based human-in-the-loop optimal output cluster synchronization control for multiagent systems: A model-free reinforcement learning method. IEEE Transactions on Cybernetics55(2): 649–660.
13.
JiangTYanYGeSZ (2025) Adaptive predefined-time bounded consensus tracking control of multiagent systems under input/output quantization. IEEE Transactions on Systems, Man, and Cybernetics: Systems99: 1–12.
14.
JiangTYanYWuDF, et al. (2022) Neural network based adaptive sliding mode tracking control of autonomous surface vehicles with input quantization and saturation. Ocean Engineering265: 112505.
15.
KhalajiAKBahramiS (2022) Finite-time sliding mode control of underwater vehicles in 3D space. Transactions of the Institute of Measurement and Control44(16): 3215–3228.
16.
LewisFLVrabieDL (2009) Reinforcement learning and adaptive dynamic programming for feedback control. IEEE Circuits and Systems Magazine9(3): 32–50.
LiZChenHCheL (2024) Antiswing control of offshore cranes under ship rolling disturbances: An active disturbance rejection control based approach. Nonlinear Dynamics112(23): 21097–21116.
19.
LiuKZYaoY (2016) Robust Control: Theory and Applications. Hoboken, NJ: Wiley.
20.
LiuXYanHCZhouWX, et al. (2024) Event-triggered optimal tracking control for underactuated surface vessels via neural reinforcement learning. IEEE Transactions on Industrial Informatics20(11): 12837–12847.
21.
MengXFZhangGCHanB, et al. (2024) Fault-tolerant control of underactuated unmanned surface vessels in presence of deception and injection attacks. Transactions of the Institute of Measurement and Control47: 01423312241263386.
22.
ModaresHLewisFL (2014) Linear quadratic tracking control of partially-unknown continuous-time systems using reinforcement learning. IEEE Transactions on Automatic Control59(11): 3051–3056.
23.
ModaresHLewisFLJiangZP (2015) H∞ tracking control of completely unknown continuous-time systems via off-policy reinforcement learning. IEEE Transactions on Neural Networks and Learning Systems26(10): 2550–2562.
24.
MuDLangZFanY, et al. (2024) Trajectory tracking control of unmanned surface vehicle based on optimized barrier Lyapunov function under real ocean wave modeling. Journal of the Franklin Institute361(16): 107182.
25.
NguyenKDangVTPhamDD, et al. (2023) Formation control scheme with reinforcement learning strategy for a group of multiple surface vehicles. International Journal of Robust and Nonlinear Control34(3): 2252–2279.
26.
QiaoYYYinJXWangW, et al. (2023) Survey of deep learning for autonomous surface vehicles in marine environments. IEEE Transactions on Intelligent Transportation Systems24(4): 3678–3701.
27.
RigatosG (2023) A nonlinear optimal control approach for unmanned surface vessels. Marine Systems and Ocean Technology18: 89–110.
28.
SkjetneRFossenTKokotovićP (2005) Adaptive maneuvering, with experiments, for a model ship in a marine control laboratory. Automatica41(2): 289–298.
29.
VuVTPhamTLDaoPN (2022a) Disturbance observer-based adaptive reinforcement learning for perturbed uncertain surface vessels. ISA Transactions130: 277–292.
30.
VuVTTranQHPhamTL, et al. (2022b) Online actor-critic reinforcement learning control for uncertain surface vessel systems with external disturbances. International Journal of Control, Automation and Systems20: 1029–1040.
31.
WangHDongZQiS, et al. (2022a) Trajectory-tracking control of an underactuated unmanned surface vehicle based on quasi-infinite horizon model predictive control algorithm. Transactions of the Institute of Measurement and Control44(14): 2709–2718.
32.
WangTQWangCYanSY, et al. (2024a) Disturbance observer–based fixed-time sliding mode trajectory tracking control for marine surface vehicles with uncertain dynamics. Transactions of the Institute of Measurement and Control46(8): 1452–1466.
33.
WangXHXiaoZHRenZM, et al. (2024b) Formation cooperative trajectory tracking control for unmanned aerial vehicles via differential game and reinforcement learning. Transactions of the Institute of Measurement and Control47: 1762–1770.
34.
WangYDLiuWZLiuJ, et al. (2023) Cooperative USV–UAV marine search and rescue with visual navigation and reinforcement learning-based control. ISA Transactions137: 222–235.
35.
WangYDSunJHeHB, et al. (2019) Deterministic policy gradient with integral compensator for robust quadrotor control. IEEE Transactions on Systems, Man, and Cybernetics: Systems50(10): 3713–3725.
36.
WangYHWangHBFuMY (2022b) Prescribed performance trajectory tracking control of dynamic positioning ship under input saturation. Transactions of the Institute of Measurement and Control44(1): 30–39.
37.
WangYLYuSHLiCP, et al. (2021) Sampled-data consensus of networked Euler-Lagrange systems: A discrete small-gain approach. IEEE Access9: 156548–156555.
38.
WuHNLuoB (2012) Neural network based online simultaneous policy update algorithm for solving the HJI equation in nonlinear H∞ control. IEEE Transactions on Neural Networks and Learning Systems23(12): 1884–1899.
39.
YanYYinCTHouYQ, et al. (2025) Adaptive fixed-time containment control of underactuated surface vessels with input quantization. Ocean Engineering318: 120157.
40.
YuanXWangYDLiuJ, et al. (2022) Action mapping: A reinforcement learning method for constrained-input systems. IEEE Transactions on Neural Networks and Learning Systems34(10): 7145–7157.
41.
ZhangRChenH (2022) An adaptive tracking control method for offshore cranes with unknown gravity parameters. Ocean Engineering260: 111809.
42.
ZhaoWBLiuHLewisFL (2020) Robust formation control for cooperative underactuated quadrotors via reinforcement learning. IEEE Transactions on Neural Networks and Learning Systems32(10): 4577–4587.
43.
ZhouLShenZPNieY, et al. (2023a) Event-triggered adaptive dynamic programming for optimal tracking control of unmanned surface vessel with input constraints. Transactions of the Institute of Measurement and Control45(15): 2835–2847.
44.
ZhouWXFuJYanHC, et al. (2023b) Event-triggered approximate optimal path-following control for unmanned surface vehicles with state constraints. IEEE Transactions on Neural Networks and Learning Systems34(1): 104–118.
