Automatic tuning of controller parameters based on reinforcement learning with applications in intelligent vehicle path tracking

Abstract

Controller parameters tuning is a tedious work for engineers since it always requires a number of repetitive trials. In this paper, we focus on the automatic tuning of controller parameters, aimed at eliminating the requirement of expertise and reducing the trials required. We extend the original PILCO, a model-based reinforcement learning framework known for data efficiency, to automatically tune complex nonlinear controllers directly from real-world trial data. The key improvement of our method is that we calculate the probability distribution of control through Monte Carlo sampling rather than an analytical method, which avoids the problem of the expression of the control function relative to the state in the integrand function being too complex to analytically calculate the distribution of control in nonlinear controllers, greatly improving the flexibility of the original PILCO in nonlinear controllers. To verify its effectiveness, the proposed method has been applied to tune three path tracking controllers for an intelligent vehicle, including a PI, pure pursuit (PP) and enhanced Stanley controllers. Experiment results show that the proposed method can achieve automatic parameter tuning in a few trials, and controllers tuning by improved-PILCO exhibit high tracking accuracy in lane change of 40 km/h and intricate path. The proposed method is hopeful to accelerate the development of control module for intelligent vehicles.

Keywords

intelligent vehicle path tracking controller automatic parameters tuning model-based reinforcement learning

Get full access to this article

View all access options for this article.

References

Attia

Orjuela

Basset

Combined longitudinal and lateral control for automated vehicle guidance. Veh Syst Dyn 2014; 52(2): 261–279.

Kumar

Raaja

Jerome

Adaptive PSO for optimal LQR tracking control of 2 DoF laboratory helicopter. Appl Soft Comput 2016; 41: 77–90.

Wang

Jiang

Lin

, et al. A learning-based automatic parameters tuning framework for autonomous vehicle control in large scale system deployment. In: 2021 American control conference (ACC), New Orleans, LA, USA, 25–28 May 2021, pp.2919–2926. New York: IEEE.

Chen

Yang

Long

, et al. A moderate online servo controller parameter self-tuning method via variable-period inertia identification. IEEE Trans Power Electron 2019; 34(12): 12165–12180.

Wang

Zhou

Wang

Extremum-seeking-based adaptive model-free control and its application to automated vehicle path tracking. IEEE ASME Trans Mechatron 2022; 27(5): 3874–3884.

Sutton

Barto

AG.

Reinforcement learning: an introduction. MIT Press, 2018, p.1.

Tang

Chen

Qin

, et al. Reinforcement learning-based energy management for hybrid power systems: state-of-the-art survey, review, and perspectives. Chin J Mech Eng 2024; 37(1): 43.

Atkeson

Santamaria

. A comparison of direct and model-based reinforcement learning. In: Proceedings of international conference on robotics and automation, Albuquerque, NM, USA, 25 April 1997, vol. 4, pp.3557–3564. New York: IEEE.

Polydoros

Nalpantidis

Survey of model-based reinforcement learning: applications on robotics. J Intell Robot Syst 2017; 86(2): 153–173.

10.

Williams

CKI

Rasmussen

. Gaussian processes for machine learning. MIT Press, 2006.

11.

Deisenroth

Rasmussen

CE.

PILCO: a model-based and data-efficient approach to policy search. In: Proceedings of the 28th international conference on machine learning (ICML-11), Bellevue, WA, USA, 28 June–2 July 2011, pp.465–472. Madison, WI: Omnipress.

12.

Deisenroth

Rasmussen

Fox

Learning to control a low-cost manipulator using data-efficient reinforcement learning. In: Durrant-Whyte

Roy

Abbeel

(eds) Robotics: science and systems VII. MIT Press, 2011, pp.57–64.

13.

Bischoff

Nguyen-Tuong

Koller

, et al. Learning throttle valve control using policy search. In: Machine learning and knowledge discovery in databases: European conference, Prague, Czech Republic, 23–27 September 2013, pp.49–64. Berlin, Heidelberg: Springer-Verlag.

14.

Deisenroth

Calandra

Seyfarth

, et al. Toward fast policy search for learning legged locomotion. In: 2012 IEEE/RSJ international conference on intelligent robots and systems, Vilamoura-Algarve, Portugal, 7–12 October 2012, pp.1787–1792. New York: IEEE.

15.

Hesse

Timmermann

Hüllermeier

, et al. A reinforcement learning strategy for the swing-up of the double pendulum on a cart. Procedia Manuf 2018; 24: 15–20.

16.

Doerr

Nguyen-Tuong

Marco

, et al. Model-based policy search for automatic tuning of multivariate PID controllers. In: 2017 IEEE international conference on robotics and automation (ICRA), Singapore, 29 May–3 June 2017, pp.5295–5301. New York: IEEE.

17.

Beaudoin

Boulet

Improving gearshift controllers for electric vehicles with reinforcement learning. Mech Mach Theory 2022; 169: 104654.

18.

Gal

McAllister

Rasmussen

CE.

Improving PILCO with Bayesian neural network dynamics models. In: Data-efficient machine learning workshop ICML, New York, NY, USA, 2016, vol. 4, no. 34, p.25. Massachusetts: JMLR.org.

19.

Cutler

How

. Efficient reinforcement learning for robots using informative simulated priors. In: 2015 IEEE international conference on robotics and automation (ICRA), Seattle, WA, USA, 26–30 May 2015, pp.2605–2612. New York: IEEE.

20.

Amadio

Dalla Libera

Antonello

, et al. Model-based policy search using Monte Carlo gradient estimation with real systems application. IEEE Trans Robot 2022; 38(6): 3879–3898.

21.

Amer

Zamzuri

Hudha

, et al. Path tracking controller of an autonomous armoured vehicle using modified Stanley controller optimized with particle swarm optimization. J Braz Soc Mech Sci Eng 2018; 40: 1–17.

22.

Mohamed

Rosca

Figurnov

, et al. Monte Carlo gradient estimation in machine learning. J Mach Learn Res 2020; 21(1): 5183–5244.

23.

Lim

Lee

Sunwoo

, et al. Hierarchical trajectory planning of an autonomous car based on the integration of a sampling and an optimization method. IEEE Trans Intell Transp Syst 2018; 19(2): 613–626.

24.

Rajamani

Vehicle dynamics and control. 2nd ed. Springer Science & Business Media, 2011, p.40.

25.

Domina

Tihanyi

. Path following controller for autonomous vehicles. In: 2019 IEEE international conference on connected vehicles and expo (ICCVE), Graz, Austria, 4–8 November 2019, pp.1–5. New York: IEEE.

26.

Liu

Yang

Huang

, et al. Simulation performance evaluation of pure pursuit, Stanley, LQR, MPC controller for autonomous vehicles. In: 2021 IEEE international conference on real-time computing and robotics (RCAR), Xining, China, 15–19 July 2021, pp.1444–1449. New York: IEEE.