We present an end-to-end imitation learning system for agile, off-road autonomous driving using only low-cost on-board sensors. By imitating a model predictive controller equipped with advanced sensors, we train a deep neural network control policy to map raw, high-dimensional observations to continuous steering and throttle commands. Compared with recent approaches to similar tasks, our method requires neither state estimation nor on-the-fly planning to navigate the vehicle. Our approach relies on, and experimentally validates, recent imitation learning theory. Empirically, we show that policies trained with online imitation learning overcome well-known challenges related to covariate shift and generalize better than policies trained with batch imitation learning. Built on these insights, our autonomous driving system demonstrates successful high-speed off-road driving, matching the state-of-the-art performance.
BojarskiMDel TestaDDworakowskiD, et al. (2016) End to end learning for self-driving cars. arXiv preprint arXiv:1604.07316.
2.
BojarskiMYeresPChoromanskaA, et al. (2017) Explaining how a deep neural network trained with end-to-end learning steers a car. arXiv preprint arXiv:1704.07911.
3.
BorrelliFFalconePKeviczkyTAsgariJHrovatD (2005) MPC-based approach to active steering for autonomous vehicle systems. International Journal of Vehicle Autonomous Systems3(2): 265–291.
4.
ChengCABootsB (2017) Variational inference for Gaussian process models with linear complexity. In: Advances in Neural Information Processing Systems.
5.
ChengCABootsB (2018) Convergence of value aggregation for imitation learning. In: International Conference on Artificial Intelligence and Statistics, pp. 1801–1809.
6.
ChengCALeeJGoldbergKBootsB (2019a) Online learning with continuous variations: Dynamic regret and reductions. arXiv preprint arXiv:1902.07286.
7.
ChengCAYanXRatliffNBootsB (2019b) Predictor–corrector policy optimization. In: International Conference on Machine Learning.
8.
ChengCAYanXTheodorouEABootsB (2019c) Accelerating imitation learning with predictive models. In: International Conference on Artificial Intelligence and Statistics.
9.
ChengCAYanXWagenerNBootsB (2018) Fast policy learning through imitation and reinforcement. In: Conference on Uncertainty in Artificial Intelligence.
10.
DrewsPWilliamsGGoldfainBTheodorouEARehgJM (2017) Aggressive deep driving: Model predictive control with a CNN cost model. arXiv preprint arXiv:1707.05303.
11.
GeistARHansenASolowjowEYangSKreuzerE (2017) Data collection for robust end-to-end lateral vehicle control. In: ASME 2017 Dynamic Systems and Control Conference. American Society of Mechanical Engineers, V001T45A007.
12.
GibbsALSuFE (2002) On choosing and bounding probability metrics. International Statistical Review70(3): 419–435.
13.
GijsbertsAMettaG (2013) Real-time model learning using incremental sparse spectrum Gaussian process regression. Neural Networks41: 59–69.
14.
HensmanJFusiNLawrenceND (2013) Gaussian processes for big data. arXiv preprint arXiv:1309.6835.
15.
JacobsonDHMayneDQ (1970) Differential dynamic programming (Modern Analytic and Computational Methods in Science and Mathematics, No. 24). New York: Elsevier.
16.
KaessMJohannssonHRobertsRIlaVLeonardJJDellaertF (2012) isam2: Incremental smoothing and mapping using the Bayes tree. The International Journal of Robotics Research31(2): 216–235.
17.
KahnGZhangTLevineSAbbeelP (2017) Plato: Policy learning using adaptive trajectory optimization. In: IEEE International Conference on Robotics and Automation, pp. 3342–3349.
18.
KakadeSLangfordJ (2002) Approximately optimal approximate reinforcement learning. In: International Conference on Machine Learning, Vol. 2, pp. 267–274.
19.
KingmaDBaJ (2014) ADAM: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
20.
KoenigNHowardA (2004) Design and use paradigms for Gazebo, an open-source multi-robot simulator. In: IEEE/RSJ International Conference onIntelligent Robots and Systems, 2004 (IROS 2004), Vol. 3. IEEE, pp. 2149–2154.
21.
KongJPfeifferMSchildbachGBorrelliF (2015) Kinematic and dynamic vehicle models for autonomous driving control design. In: 2015 IEEE Intelligent Vehicles Symposium (IV). IEEE, pp. 1094–1099.
22.
LaskeyMChuckCLeeJ, et al. (2016) Comparing human-centric and robot-centric sampling for robot deep learning from demonstrations. arXiv preprint arXiv:1610.00850.
23.
LaurenseVAGohJYGerdesJC (2017) Path-tracking for autonomous vehicles at the limit of friction. In: 2017 American Control Conference (ACC), pp. 5586–5591.
24.
Lázaro-GredillaMQuiñonero CandelaJRasmussenCEFigueiras-VidalAR (2010) Sparse spectrum Gaussian process regression. Journal of Machine Learning Research11: 1865–1881.
25.
LeeJLaskeyMTanwaniAKAswaniAGoldbergK (2018) A dynamic regret analysis and adaptive regularization algorithm for on-policy robot imitation learning. arXiv preprint arXiv:1811.02184.
26.
LevineSFinnCDarrellTAbbeelP (2016) End-to-end training of deep visuomotor policies. Journal of Machine Learning Research17(1): 1334–1373.
27.
LjungL (1998) System identification. In: Signal Analysis and Prediction. New York: Springer, pp. 163–173.
28.
MaatenLvdHintonG (2008) Visualizing data using t-SNE. Journal of Machine Learning Research9: 2579–2605.
29.
MichelsJSaxenaANgAY (2005) High speed obstacle avoidance using monocular vision and reinforcement learning. In: International Conference on Machine learning, pp. 593–600.
30.
MordatchITodorovE (2014) Combining the benefits of function approximation and trajectory optimization. In: Robotics: Science and Systems, pp. 5–32.
31.
MullerUBenJCosattoEFleppBCunYL (2006) Off-road obstacle avoidance through end-to-end learning. In: Advances in Neural Information Processing Systems, pp. 739–746.
32.
PadenBČápMYongSZYershovDFrazzoliE (2016) A survey of motion planning and control techniques for self-driving urban vehicles. IEEE Transactions on Intelligent Vehicles1(1): 33–55.
33.
PanYChengCASaigolK, et al. (2018) Agile autonomous driving via end-to-end deep imitation learning. In: Proceedings of Robotics: Science and Systems (RSS).
34.
PanYYanXTheodorouEABootsB (2017) Prediction under uncertainty in sparse spectrum Gaussian processes with applications to filtering and control. In: International Conference on Machine Learning, pp. 2760–2768.
35.
PomerleauDA (1989) ALVINN: An autonomous land vehicle in a neural network. In: Advances in Neural Information Processing Systems, pp. 305–313.
36.
RajamaniR (2011) Vehicle dynamics and control. New York: Springer Science & Business Media.
37.
RauschVHansenASolowjowELiuCKreuzerEHedrickJK (2017) Learning a deep neural net policy for end-to-end control of autonomous vehicles. In: IEEE American Control Conference.
38.
RossSBagnellJA (2014) Reinforcement and imitation learning via interactive no-regret learning. arXiv preprint arXiv: 1406.5979.
39.
RossSGordonGJBagnellD (2011) A reduction of imitation learning and structured prediction to no-regret online learning. In: International Conference on Artificial Intelligence and Statistics, Vol. 1, p. 6.
40.
RossSMelik-BarkhudarovNShankarKS, et al. (2013) Learning monocular reactive UAV control in cluttered natural environments. In: IEEE International Conference on Robotics and Automation, pp. 1765–1772.
41.
RutherfordSJColeDJ (2010) Modelling nonlinear vehicle dynamics with neural networks. International Journal of Vehicle Design53(4): 260–287.
Shalev-ShwartzS (2012) Online learning and online convex optimization. Foundations and Trends in Machine Learning4(2): 107–194.
44.
ShimodairaH (2000) Improving predictive inference under covariate shift by weighting the log-likelihood function. Journal of Statistical Planning and Inference90(2): 227–244.
45.
SnelsonEGhahramaniZ (2006) Sparse Gaussian processes using pseudo-inputs. In: Advances in Neural Information Processing Systems.
46.
TassaYErezTSmartWD (2008) Receding horizon differential dynamic programming. In: Advances in Neural Information Processing Systems, pp. 1465–1472.
47.
TitsiasM (2009) Variational learning of inducing variables in sparse Gaussian processes. In: International Conference on Artificial Intelligence and Statistics.
48.
VolodymyrMKorayKDavidSAndreiARJoelV (2015) Human-level control through deep reinforcement learning. Nature518(7540): 529–533.
49.
WagenerNChengCASacksJBootsB (2019) An online learning approach to model predictive control. In: Robotics: Science and Systems.
50.
WilliamsCRasmussenC (2006) Gaussian Processes for Machine Learning. Cambridge, MA: MIT Press.
51.
WilliamsGDrewsPGoldfainBRehgJMTheodorouEA (2016) Aggressive driving with model predictive path integral control. In: IEEE International Conference on Robotics and Automation, pp. 1433–1440.
52.
WilliamsGWagenerNGoldfainB, et al. (2017) Information theoretic MPC for model-based reinforcement learning. In: 2017 IEEE International Conference on Robotics and Automation, pp. 1714–1721.
53.
YangZZhangYYuJCaiJLuoJ (2018) End-to-end multi-modal multi-task vehicle control for self-driving cars with visual perception. arXiv preprint arXiv:1801.06734.
54.
YuHYangSGuWZhangS (2017) Baidu driving dataset and end-to-end reactive control model. In: 2017 IEEE Intelligent Vehicles Symposium (IV), pp. 341–346.
