Autonomous helicopter flight is widely regarded to be a highly challenging control problem. Despite this fact, human experts can reliably fly helicopters through a wide range of maneuvers, including aerobatic maneuvers at the edge of the helicopter’s capabilities. We present apprenticeship learning algorithms, which leverage expert demonstrations to efficiently learn good controllers for tasks being demonstrated by an expert. These apprenticeship learning algorithms have enabled us to significantly extend the state of the art in autonomous helicopter aerobatics. Our experimental results include the first autonomous execution of a wide range of maneuvers, including but not limited to in-place flips, in-place rolls, loops and hurricanes, and even auto-rotation landings, chaos and tic-tocs, which only exceptional human pilots can perform. Our results also include complete airshows, which require autonomous transitions between many of these maneuvers. Our controllers perform as well as, and often even better than, our expert pilot.
Abbeel, P., Coates, A., Hunter, T. and Ng, A.Y. ( 2008). Autonomous auto-rotation of an RC helicopter. Proceedings of ISER.
2.
Abbeel, P., Coates, A., Quigley, M. and Ng, A.Y. ( 2007). An application of reinforcement learning to aerobatic helicopter flight. Proceedings of NIPS19.
3.
Abbeel, P., Ganapathi, V. and Ng, A.Y. ( 2006a). Learning vehicular dynamics with application to modeling helicopters. Proceedings of NIPS18.
4.
Abbeel, P. and Ng, A.Y. ( 2004). Apprenticeship learning via inverse reinforcement learning . Proceedings of ICML.
5.
Abbeel, P. and Ng, A.Y. ( 2005a). Exploration and apprenticeship learning in reinforcement learning. Proceedings of ICML.
6.
Abbeel, P. and Ng, A.Y. ( 2005b). Learning first order Markov models for control. Proceedings of NIPS 18.
7.
Abbeel, P., Quigley, M., and Ng, A.Y. ( 2006b). Using inaccurate models in reinforcement learning. Proceedings of ICML.
8.
An, C.H., Atkeson, C.G. and Hollerbach, J.M. ( 1988). Model-Based Control of a Robot Manipulator. Cambridge, MA, MIT Press.
9.
Anderson, B. and Moore, J. ( 1989). Optimal Control: Linear Quadratic Methods. Englewood Cliffs, NJ, Prentice-Hall.
10.
Atkeson, C.G., Moore, A.W. and Schaal, S. ( 1997). Locally weighted learning for control. Artificial Intelligence Review, 11(1-5).
11.
Atkeson, C.G. and Schaal, S. ( 1997). Robot learning from demonstration. Proceedings of the 14th International Conference on Machine Learning. San Mateo, CA, Morgan Kaufmann, pp. 12-20.
12.
Bagnell, J. and Schneider, J. ( 2001). Autonomous helicopter control using reinforcement learning policy search methods. International Conference on Robotics and Automation. Piscataway, NJ, IEEE Press.
13.
Bar-Itzhack, I.Y. and Oshman, Y. ( 1985). Attitude determination from vector observations: Quaternion estimation. IEEE Transaction on Aerospace and Electronic Systems .
14.
Bertsekas, D.P. ( 2001). Dynamic Programming and Optimal Control, volume 1, 2nd edn. New York, Athena Scientific.
15.
Beygelzimer, A., Kakade, S. and Langford, J. ( 2006). Cover trees for nearest neighbor. ICML ‘06: Proceedings of the 23rd International Conference on Machine Learning.
16.
Boutilier, C., Friedman, N., Goldszmidt, M. and Koller, D. ( 1996). Context-specific independence in Bayesian networks. Proceedings of UAI.
17.
Calinon, S., Guenter, F. and Billard, A. ( 2007). On learning, representing and generalizing a task in a humanoid robot. IEEE Transaction on Systems, Man, and Cybernetics, Part B, 37.
18.
Coates, A., Abbeel, P. and Ng, A.Y. ( 2008). Learning for control from multiple demonstrations. Proceedings of ICML.
19.
Dempster, A.P., Laird, N.M. and Rubin, D.B. ( 1977). Maximum likelihood from incomplete data via the EM algorithm . Journal of the Royal Statistical Society.
20.
Frazzoli, E., Dahleh, M. and Feron, E. ( 2002). Real-time motion planning for agile autonomous vehicles . AIAA Journal on Guidance, Control and Dynamics.
21.
Frazzoli, E., Dahleh, M. and Feron, E. ( 2005). Maneuver-based motion planning for nonlinear systems with symmetries. IEEE Transactions on Robotics.
22.
Gavrilets, V., Martinos, I., Mettler, B. and Feron, E. ( 2002a). Control logic for automated aerobatic flight of miniature helicopter. AIAA Guidance, Navigation and Control Conference.
23.
Gavrilets, V., Martinos, I., Mettler, B. and Feron, E. ( 2002b). Flight test and simulation results for an autonomous aerobatic helicopter. AIAA/IEEE Digital Avionics Systems Conference.
24.
Gavrilets, V., Martinos, M., Mettler, B. and Feron, E. ( 2002c). Control logic for automated aerobatic flight of miniature helicopter. Proceedings of the AIAA Guidance, Navigation, and Control Conference.
25.
Gelb, A. (ed.) (1974). Applied Optimal Estimation. Cambrdige, MA, MIT Press.
Johnson, W. ( 1977). Helicopter Optimal Descent and Landing After Power Loss . Report NASA TM-73244.
28.
Kalman, R.E. ( 1960). A new approach to linear filtering and prediction problems . Transactions of the ASME-Journal of Basic Engineering (Series D), 82: 35-45.
29.
La Civita, M., Messner, W.C. and Kanade, T. ( 2002). Modeling of small-scale helicopters with integrated first-principles and system-identification techniques. Proceedings of the 58th Forum of the American Helicopter Society.
30.
La Civita, M., Papageorgiou, G., Messner, W.C. and Kanade, T. ( 2003). Design and flight testing of a gain-scheduled H-infinity loop shaping controller for wide-envelope flight of a robotic helicopter. Proceedings of the American Control Conference.
31.
La Civita, M., Papageorgiou, G., Messner, W.C. and Kanade, T. ( 2006). Design and flight testing of a high-bandwidth ∞ loop shaping controller for a robotic helicopter. Journal of Guidance, Control, and Dynamics, 29(2): 485-494.
32.
Lee, A. ( 1985). Optimal Landing of a Helicopter in Autorotation. PhD Thesis, Stanford University.
33.
Lefferts, E., Markley, F.L. and Shuster, M.D. ( 1982). Kalman filtering for spacecraft attitude estimation. Journal of Guidance, Control, and Dynamics.
34.
Leishman, J. ( 2000). Principles of Helicopter Aerodynamics. Cambridge, Cambridge University Press.
35.
Listgarten, J. ( 2006). Analysis of Sibling Time Series Data: Alignment and Difference Detection. PhD Thesis, University of Toronto.
36.
Listgarten, J., Neal, R.M., Roweis, S.T. and Emili, A. ( 2005). Multiple alignment of continuous time series. Proceedings of NIPS17.
37.
Mettler, B., Tischler, M. and Kanade, T. ( 1999). System identification of small-size unmanned helicopter dynamics. American Helicopter Society, 55th Forum.
38.
Moore, A., Schneider, J. and Deng, K. ( 1997). Efficient locally weighted polynomial regression predictions . Proceedings of ICML.
39.
Neal, R. and Hinton, G. ( 1999). A view of the EM algorithm that justifies incremental, sparse, and other variants. Learning in Graphical Models. Cambrdige, MA, MIT Press, pp. 355-368.
40.
Needleman, S. and Wunsch, C. ( 1970). A general method applicable to the search for similarities in the amino acid sequence of two proteins. Journal of Molecular Biology.
41.
Neu, G. and Szepesvari, C. ( 2007). Apprenticeship learning using inverse reinforcement learning and gradient methods. Proceedings of UAI.
42.
Ng, A.Y., Coates, A., Diel, M., Ganapathi, V., Schulte, J., Tse, B., Berger, E. and Liang, E. ( 2004a). Autonomous inverted helicopter flight via reinforcement learning. Proceedings of ISER.
43.
Ng, A.Y., Kim, H.J., Jordan, M., and Sastry, S. ( 2004b). Autnonomous helicopter flight via reinforcement learning . Proceedings of NIPS16.
44.
Ng, A.Y. and Russell, S. ( 2000). Algorithms for inverse reinforcement learning. Proceedings of ICML.
45.
Preparata, F.P. and Shamos, M. ( 1985). Computational Geometry. Berlin, Springer-Verlag.
46.
Ramachandran, D. and Amir, E. ( 2007). Bayesian inverse reinforcement learning. Proceedings of IJCAI.
47.
Randlov, J. and Alstrom, P. ( 1998). Learning to drive a bicycle using reinforcement learning and shaping. Proceedings of ICML.
48.
Ratliff, N., Bagnell, J. and Zinkevich, M. ( 2006). Maximum margin planning. Proceedings of ICML.
49.
Roberts, J.M., Corke, P.I. and Buskey, G. ( 2003). Low-cost flight control system for a small autonomous helicopter. IEEE International Conference on Robotics and Automation.
50.
Sakoe, H. and Chiba, S. ( 1978). Dynamic programming algorithm optimization for spoken word recognition. IEEE Transactions on Acoustics, Speech, and Signal Processing.
51.
Saripalli, S., Montgomery, J. and Sukhatme, G. ( 2003). Visually-guided landing of an unmanned aerial vehicle .
52.
Seddon, J. ( 1990). Basic Helicopter Aerodynamics (AIAA Education Series) . America Institute of Aeronautics and Astronautics.
53.
Shuster, M.D. ( 2003). Constraint in attitude estimation: Part II unconstrained estimation. Journal of the Astronautical Sciences.
54.
Syed, U. and Schapire, R.E. ( 2008). A game-theoretic approach to apprenticeship learning . Proceedings of NIPS20.
55.
Tedrake, R., Zhang, T.W. and Seung, H.S. ( 2004). Stochastic policy gradient reinforcement learning on a simple 3D biped. Proceedings of the IEEE International Conference on Intelligent Robots and Systems (IROS).
56.
Tischler, M. and Cauffman, M. ( 1992). Frequency response method for rotorcraft system identification: flight application to BO-105 couple rotor/fuselage dynamics. Journal of theAmerican Helicopter Society.
57.
Zanetti, R. and Bishop, R.H. ( 2006). Quaternion estimation and norm constrained Kalman filtering . AIAA/AAS Astrodynamics Specialist Conference .
