The fuel economy of hybrid electric vehicles is inextricably linked to the energy management strategy (EMS). In this study, a practicality-oriented learning-based EMS for a power-split hybrid electric bus (HEB) is presented, which combines the generative adversarial imitation learning (GAIL) and deep reinforcement learning (DRL). Considering the regular and fixed route of the HEB, optimal control samples that are not affected by the cost function can be obtained by the boundary-line dynamic programing (B-DP) method. On this basis, the samples are dynamically fitted using the GAIL method to inverse derive the reward function that can explain the B-DP control behavior. Concurrently, the proximal policy optimization DRL algorithm will regulate the energy distribution of the vehicle in real time and continuously optimize the energy management capability based on the reward feedback from GAIL. Finally, the feasibility and effectiveness of the proposed EMS is verified by simulation. The results show that the proposed strategy exhibits near-optimal control performance under both the China heavy-duty commercial vehicle cycle-bus and China city bus cycle.
HuangYWangHKhajepourA, et al. A review of power management strategies and component sizing methods for hybrid vehicles. Renew Sustain Energ Rev2018; 96: 132–144.
2.
LiXEvangelouSA. Torque-leveling threshold-changing rule-based control for parallel hybrid electric vehicles. IEEE Trans Vehicular Technol2019; 68: 6509–6523.
3.
ChenJXuCWuC, et al. Adaptive fuzzy logic control of fuel-cell-battery hybrid systems for electric vehicles. IEEE Trans Ind Inform2018; 14: 292–300.
4.
LiSGSharkhSMWalshFC, et al. Energy and battery management of a plug-in series hybrid electric vehicle using fuzzy logic. IEEE Trans Vehicular Technol2011; 60: 3571–3585.
5.
EnangWBannisterC. Modelling and control of hybrid electric vehicles (A comprehensive review). Renew Sustain Energ Rev2017; 74: 1210–1239.
6.
ZhangYLiuHGuoQ. Varying-domain optimal management strategy for parallel hybrid electric vehicles. IEEE Trans Vehicular Technol2014; 63: 603–616.
7.
ChenZXiongRWangK, et al. Optimal energy management strategy of a plug-in hybrid electric vehicle based on a particle swarm optimization algorithm. Energies2015; 8: 3661–3678.
8.
LiLYangCZhangY, et al. Correctional DP-based energy management strategy of plug-in hybrid electric bus for city-bus route. IEEE Trans Vehicular Technol2015; 64: 2792–2803.
9.
YangRYangXHuangW, et al. Energy management of the power-split hybrid electric city bus based on the stochastic model predictive control. IEEE Access2021; 9: 2055–2071.
10.
SunCHuXSMouraSJ, et al. Velocity predictors for predictive energy management in hybrid electric vehicles. IEEE Trans Control Syst Technol2015; 23: 1197–1204.
11.
MusardoCRizzoniGGuezennecY, et al. A-ECMS: an adaptive algorithm for hybrid electric vehicle energy management. Eur J Control2005; 11: 509–524.
12.
KimNChaSWPengH. Optimal equivalent fuel consumption for hybrid electric vehicles. IEEE Trans Control Syst Technol2012; 20: 817–825.
13.
XieSHuXXinZ, et al. Time-efficient stochastic model predictive energy management for a plug-in hybrid electric bus with an adaptive reference state-of-charge advisory. IEEE Trans Vehicular Technol2018; 67: 5671–5682.
14.
ZhangHPengJTanH, et al. A deep reinforcement learning-based energy management framework with lagrangian relaxation for plug-in hybrid electric vehicle. IEEE Trans Transp Electrification2021; 7: 1146–1160.
15.
PeiJSuYZhangD, et al. Velocity forecasts using a combined deep learning model in hybrid electric vehicles with V2V and V2I communication. Sci China Technol Sci2020; 63: 55–64.
16.
LiuTZouYLiuD, et al. Reinforcement learning of adaptive energy management with transition probability for a hybrid electric tracked vehicle. IEEE Trans Ind Electron2015; 62: 7837–7846.
17.
LiuTHuX. A bi-level control for energy efficiency improvement of a hybrid tracked vehicle. IEEE Trans Ind Inform2018; 14: 1616–1625.
18.
XuBRathodDZhangD, et al. Parametric study on reinforcement learning optimized energy management strategy for a hybrid electric vehicle. Appl Energy2020; 259: 114200.
19.
LiuTTanWTangX, et al. Driving conditions-driven energy management strategies for hybrid electric vehicles: a review. Renew Sustain Energ Rev2021; 151: 111521.
20.
WuJHeHPengJ, et al. Continuous reinforcement learning of energy management with deep Q network for a power split hybrid electric bus. Appl Energy2018; 222: 799–811.
21.
WangPYLiYShekharS, et al. A deep reinforcement learning framework for energy management of extended range electric delivery vehicle. In: 2019 IEEE intelligent vehicles symposium (IV), Paris, France, 9–12 June 2019, pp.1837–1842. New York, NY: IEEE.
22.
YangCZZhouCC. An energy management strategy of hybrid electric vehicles based on deep reinforcement learning. Int J Eng Adv Res Technol2019; 5: 1–4.
23.
HanXHeHWuJ, et al. Energy management based on reinforcement learning with double deep Q-learning for a hybrid electric tracked vehicle. Appl Energy2019; 254: 113708.
24.
QiXLuoYWuG, et al. Deep reinforcement learning enabled self-learning control for energy efficient driving. Transp Res Part C Emerg Technol2019; 99: 67–81.
25.
SchroerCLießnerRBäkerB. Adaptive operating strategies for the energy management of hybrid electric vehicles with deep reinforcement learning. In: Electric & Electronic Systems in Hybrid and Electrical Vehicles and Electrical Energy Management (EEHE), Würzburg, Gemany, 12–13 June 2018, pp.1–12. Essen: Haus der Technik e. V.
26.
LiessnerRSchmittJDietermannA, et al. Hyperparameter optimization for deep reinforcement learning in vehicle energy management. In: Proceedings of the 11th International conference on agents and artificial intelligence, Prague, Czech Republic, 19–21 February 2019, pp.134–144. Setubal: SciTePress.
27.
LiYHeHPengJ, et al. Deep reinforcement learning-based energy management for a series hybrid electric vehicle enabled by history cumulative trip information. IEEE Trans Vehicular Technol2019; 68: 7416–7430.
28.
HeHWangYLiJ, et al. An improved energy management strategy for hybrid electric vehicles integrating multi-states of vehicle-traffic information. IEEE Trans Transp Electrification2021; 7: 1161–1172.
29.
WangYTanHWuY, et al. Hybrid electric vehicle energy management with computer vision and deep reinforcement learning. IEEE Trans Ind Inform2021; 17: 3857–3868.
30.
ZhouJXueSXueY, et al. A novel energy management strategy of hybrid electric vehicle via an improved TD3 deep reinforcement learning. Energy2021; 224: 120118.
31.
WuJWeiZLiW, et al. Battery thermal- and health-constrained energy management for hybrid electric bus based on soft actor-critic DRL algorithm. IEEE Trans Ind Inform2021; 17: 3751–3761.
32.
LianRPengJWuY, et al. Rule-interposing deep reinforcement learning based energy management strategy for power-split hybrid electric vehicle. Energy2020; 197: 117297.
33.
TianXHeRSunX, et al. An ANFIS-based ECMS for energy optimization of parallel hybrid electric bus. IEEE Trans Vehicular Technol2020; 69: 1473–1483.
34.
WangYZengXSongD, et al. Optimal rule design methodology for energy management strategy of a power-split hybrid electric bus. Energy2019; 185: 1086–1099.
35.
YangXYangRTanS, et al. MPGA-based-ECMS for energy optimization of a hybrid electric city bus with dual planetary gear. Proc IMechE, Part D: J Automobile Engineering. Epub ahead of print 23August2021. DOI: 10.1177/09544070211041074
36.
PanLCaiQPHuangLB. Softmax deep double deterministic policy gradients. In: 34th Conference on neural information processing systems (NeurIPS 2020), Virtual, Online, 6–12 December 2020, pp.11767–11777. New York: Curran Associates Inc.
HoJErmonS. Generative adversarial imitation learning. In: 30th Conference on neural information processing systems (NeurIPS 2016), Barcelona, Spain, 5–10 December 2016, pp.4565–4573. New York: Curran Associates Inc.
39.
ZiebartBDMaasABagnellJA, et al. Maximum entropy inverse reinforcement learning. In: Proceedings of the twenty-third AAAI conference on artificial intelligence, Chicago, Illinois, USA, 13–17 July 2008, pp.1433–1438. Menlo Park: AAAI Press.
40.
CreswellAWhiteTDumoulinV, et al. Generative adversarial networks: an overview. IEEE Signal Process Mag2018; 35: 53–65.
41.
ZhouWYangLCaiY, et al. Dynamic programming for new energy vehicles based on their work modes part I: Electric Vehicles and Hybrid Electric Vehicles. J Power Sources2018; 406: 151–166.
42.
SundströmOAmbühlDGuzzellaL. On implementation of dynamic programming for optimal control problems with final state constraints. Oil Gas Sci Technol Rev IFP2010; 65: 91–102.
