Aiming at poor effects and single consideration factors of traditional driving decision-making algorithm in high-speed and complex environment, a method based on improved deep reinforcement learning (DRL) is proposed in this paper. We innovatively design and optimize the reward function of the Dueling Deep Q network (Dueling DQN), and the factors such as safety, comfort, traffic efficiency and altruism are taken into account. The weight of each influencing factor is determined by the Analytic Hierarchy Process (AHP), which makes the influence of each factor on driving behavior decision-making more acceptable. Subsequently, a decision-making model of autonomous vehicles (AVs) is built by using improved Dueling DQN. Furthermore, the action space is enriched and combined with the trajectory planner, so that AVs can take appropriate behaviors in the longitudinal and lateral directions according to the environment. The output of the decision model can be combined with the underlying controller with a view to make the AVs maneuver reasonably. The driving decision-making method in two different traffic scenarios is simulated. Moreover, the improved method compares with other methods. The results illustrate that the improved Dueling DQN can make the AVs take safe, comfortable, efficient, and altruistic behavior.
Ibañez-GuzmánJLaugierCYoderJD, et al. Autonomous driving: context and state-of-the-art. In: EskandarianA (ed.) Handbook of intelligent vehicles. London: Springer, 2012.
2.
UrmsonCWhittakerW.Self-driving cars and the urban challenge. IEEE Intell Syst2008; 23(2): 66–68.
3.
WangXWangSCaoJ, et al. Data-driven based tiny-YOLOv3 method for front vehicle detection inducing SPP-net. IEEE Access2020; 8: 110227–110236.
4.
GippsPG.A model for the structure of lane-changing decisions. Transp Res Part B Methodol1986; 20(5): 403–414.
5.
KestingAMartin, et al. General lane-changing model MOBIL for car-following models. Transp Res Rec2018; 1999: 86–94.
6.
VanholmeBGruyerDLusettiB, et al. Highly automated driving on highways based on legal safety. IEEE Trans Intell Transp Syst2013; 14(1): 333–347.
7.
UrmsonCAnhaltJBagnellD, et al. Autonomous driving in urban environments: boss and the urban challenge. J Field Robot2008; 25(8): 425–466.
8.
MontemerloMBeckerJBhatS, et al. Junior: the stanford entry in the urban challenge. J Field Robot2008; 25(9): 569–597.
9.
ZieglerJBenderPSchreiberM, et al. Making bertha drive—an autonomous journey on a historic route. IEEE Intell Transp Syst Mag2014; 6(2): 8–20.
10.
FurdaAVlacicL.Enabling safe autonomous driving in real-world city traffic using multiple criteria decision making. IEEE Intell Transp Syst Mag2011; 3(1): 4–17.
11.
LiuYWangXLiL, et al. A novel lane change decision-making model of autonomous vehicle based on support vector machine. IEEE Access2019; 7: 26543–26550.
12.
BojarskiMDel TestaDDworakowskiD, et al. End to end learning for self-driving cars, arXiv preprint arXiv:1604.07316, 2016.
13.
GuoCWangXSuL, et al. Safety distance model for longitudinal collision avoidance of logistics vehicles considering slope and road adhesion coefficient. Proc IMechE, Part D: J Automobile Engineering2021; 235(2-3): 498–512.
14.
WeiJDolanJMSniderJM, et al. A point-based MDP for robust single-lane autonomous driving behavior under uncertainties. In: 2011 IEEE international conference on robotics and automation, Shanghai, China, 9–13May2011, pp.2586–2592. New York: IEEE.
15.
BrechtelSGindeleTDillmanR.Probabilistic decision-making under uncertainty for autonomous driving using continuous POMDPs. In: International IEEE conference on intelligent transportation systems, Qingdao, China, 8–11October2014. New York: IEEE.
16.
UlbrichSMaurerM.Situation assessment in tactical lane change behavior planning for automated vehicles. In: 2015 IEEE 18th international conference on intelligent transportation systems, Las Palmas, 2015, pp.975–981. DOI: 10.1109/ITSC.2015.163.
17.
KoberJBagnellJAPetersJ.Reinforcement learning in robotics: a survey. Int J Rob Res2013; 32: 1238–1274.
18.
LiY.Deep reinforcement learning: an overview. arXiv:1701.07274, 2017
19.
Vincent FranOFonteneauRDamienE. Playing atari with deep reinforcement learning. arXiv:1312.5602, 2013.
20.
MnihVKavukcuogluKSilverD, et al. Human-level control through deep reinforcement learning. Nature2015; 518: 529–533.
21.
SallabAEAbdouMPerotE, et al. Deep reinforcement learning framework for autonomous driving. Electron Imaging2017; 29: 70–76.
22.
WangPChanCYArnaudDLF.A reinforcement learning based approach for automated lane change maneuvers. In: 2018 IEEE AVs symposium (IV), Changshu, 2018, pp.1379–1384. DOI: 10.1109/IVS.2018.8500556.
23.
ChaeHKangCMKimB, et al. Autonomous braking system via deep reinforcement learning. arXiv preprint arXiv: 1702.02302, 2017.
24.
SallabAEAbdouMPerotE, et al. End-to-end deep reinforcement learning for lane keeping assist. arXiv preprint arXiv:1612.04340, 2016.
25.
LinYMcpheeJAzadNL.Anti-jerk on-ramp merging using deep reinforcement learning. arXiv preprint arXiv:1909.12967, 2019.
26.
WolfPKurzerKWingertT, et al. Adaptive behavior generation for autonomous driving using deep reinforcement learning with compact semantic states. In: 2018 IEEE AVs symposium (IV), Changshu, 2018, pp.993–1000. DOI: 10.1109/IVS.2018.8500427.
27.
WangJZhangQZhaoD, et al. Lane change decision-making through deep reinforcement learning with rule-based constraints. In: International joint conference on neural networks (IJCNN), Budapest, Hungary, 2019, pp.1–6. New York, NY: IEEE.
28.
HoelCWolffKLaineL.Automated speed and lane change decision making using deep reinforcement learning. In: 2018 21st international conference on intelligent transportation systems (ITSC), Maui, HI, 2018, pp.2148–2155. New York, NY: IEEE.
29.
WangPLiHChanCY.Continuous control for automated lane change behavior based on deep deterministic policy gradient algorithm. In: 2019 IEEE AVs symposium (IV), Paris, France, 2019, pp.1454–1460. New York, NY: IEEE.
30.
XieJShaoZLiY, et al. Deep reinforcement learning with optimized reward functions for robotic trajectory planning. IEEE Access2019; 7: 105669–105679.
31.
YeFChengXWangP, et al. Automated lane change strategy using proximal policy optimization-based deep reinforcement learning. arXiv:2002.02667v2, 2020.
32.
YangWZhengLLiY, et al. Automated highway driving decision considering driver characteristics. IEEE Trans Intell Transp Syst2020; 21(6): 2350–2359.
33.
KohSZhouBFangH, et al. Real-time deep reinforcement learning based vehicle navigation. Appl Soft Comput2020; 96: 106694.
34.
XiCShiTWuY, et al. Efficient motion planning for automated lane change based on imitation learning and mixed-integer optimization. In: 2020 IEEE 23rd International conference on intelligent transportation systems (ITSC), Rhodes, 2020, pp.1–6. DOI: 10.1109/ITSC45102.2020.9294679.
35.
RoneckerMPZhuY.Deep Q-network based decision making for autonomous driving. In: 2019 3rd international conference on robotics and automation sciences (ICRAS), Wuhan, China, 2019, pp.154–160. DOI: 10.1109/ICRAS.2019.8808950.
36.
WangZSchaulTHesselM, et al. Dueling network architectures for deep reinforcement learning. arXiv preprint arXiv: 1511.06581, 2016.
37.
SchaulTQuanJAntonoglouI, et al. Prioritized experience replay. arXiv preprint arXiv:1511.05952, 2015.
38.
KingmaDPBaJ.Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
39.
LiuTMuXTangX, et al. Dueling Deep Q network for highway decision making in autonomous vehicles: a case study. arXiv preprint arXiv:2007.08343, 2020.
40.
WangXWangSCaoJ, et al. Data-Driven Based Tiny-YOLOv3 Method for Front Vehicle Detection Inducing SPP-Net. IEEE Access, 2020; 8:110227–110236.
41.
VaidyaOSKumarS.Analytic hierarchy process: an overview of applications. Eur J Oper Res2006; 169(1): 1–29.
42.
JiJKhajepourAMelekWW, et al. Path planning and tracking for vehicle collision avoidance based on model predictive control with multiconstraints. IEEE Trans Vehicular Technol2017; 66(2): 952–964.
43.
GonzalezDSGarzonMDibangoyeJS, et al. Human-like decision-making for automated driving in highways. In: 2019 IEEE intelligent transportation systems conference (ITSC), Auckland, New Zealand, 2019, pp.2087–2094. DOI: 10.1109/ITSC.2019.8917449.
