The application of deep learning in robotics leads to very specific problems and research questions that are typically not addressed by the computer vision and machine learning communities. In this paper we discuss a number of robotics-specific learning, reasoning, and embodiment challenges for deep learning. We explain the need for better evaluation metrics, highlight the importance and unique challenges for deep robotic learning in simulation, and explore the spectrum between purely data-driven and model-driven approaches. We hope this paper provides a motivating overview of important research directions to overcome the current limitations, and helps to fulfill the promising potentials of deep learning in robotics.
AgrawalPNairAAbbeelPMalikJLevineS (2016) Learning to poke by poking: Experiential learning of intuitive physics. In: Advances in Neural Information Processing Systems (NIPS).
AtanasovNSankaranBLe NyJPappasGJDaniilidisK (2014) Nonmyopic view planning for active object classification and pose estimation. IEEE Transactions on Robotics30(5): 1078–1090.
5.
BaillargeonRLiJGertnerYWuD (20111) How do infants reason about physical events? In: The Wiley-Blackwell Handbook of Childhood Cognitive Development (2nd edn). Oxford: Blackwell.
6.
BaillargeonRStavansMWuDet al. (2012) Object individuation and physical reasoning in infancy: An integrative account. Language Learning and Development8(1): 4–46.
7.
BarrettSTaylorMEStoneP (2010) Transfer learning for reinforcement learning on a physical robot. In: Ninth International Conference on Autonomous Agents and Multiagent Systems - Adaptive Learning Agents Workshop (AAMAS - ALA).
8.
BattagliaPPascanuRLaiMRezendeDJet al. (2016) Interaction networks for learning about objects, relations and physics. In: Advances in Neural Information Processing Systems (NIPS), pp. 4502–4510.
9.
BattagliaPWHamrickJBTenenbaumJB (2013) Simulation as an engine of physical scene understanding. Proceedings of the National Academy of Sciences110(45): 18327–18332.
10.
BendaleABoultTE (2015) Towards open world recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1893–1902.
11.
BendaleABoultTE (2016) Towards open set deep networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1563–1572.
12.
BertinettoLHenriquesJFValmadreJTorrPVedaldiA (2016) Learning feed-forward one-shot learners. In: Advances in Neural Information Processing Systems, pp. 523–531.
13.
BircherAKamelMAlexisKOleynikovaHSiegwartR (2016) Receding horizon “next-best-view” planner for 3D exploration. In: IEEE International Conference on Robotics and Automation (ICRA). IEEE, pp. 1462–1468.
14.
BlanzVTarrMJBülthoffHH (1999) What object attributes determine canonical views?Perception28(5): 575–599.
15.
ByravanAFoxD (2017) SE3-nets: Learning rigid body motion using deep neural networks. In: Proceedings of the IEEE International Conference on Robotics and Automation (ICRA).
16.
ByravanALeebFMeierFFoxD (2018) SE3-Pose-Nets: Structured deep dynamics models for visuomotor planning and control. In: Proceedings of the IEEE International Conference on Robotics and Automation (ICRA).
17.
C FinnIGLevineS (2016) Unsupervised learning for physical interaction through video prediction. In: Advances in Neural Information Processing Systems (NIPS).
18.
CadenaCCarloneLCarrilloHet al. Past, present, and future of simultaneous localization and mapping: Toward the robust-perception age. IEEE Transactions on Robotics32(6): 1309–1332.
19.
ChoyCBXuDGwakJChenKSavareseS (2016) 3D-R2N2: A unified approach for single and multi-view 3D object reconstruction. In: European Conference on Computer Vision (ECCV). New York: Springer, pp. 628–644.
20.
CohnDAGhahramaniZJordanMI (1996) Active learning with statistical models. Journal of Artificial Intelligence Research4(1): 129–145.
21.
CoxDDDeanT (2014) Neural networks and neuroscience-inspired computer vision. Current Biology24(18): R921–R929.
22.
CsurkaG (2017) Domain adaptation for visual applications: A comprehensive survey. arXiv preprint arXiv:1702.05374.
23.
DayoubFSünderhaufNCorkeP (2017) Episode-based active learning with Bayesian neural networks. In: CVPR Workshop on Deep Learning for Robotic Vision. arXiv preprint arXiv:1703.07473.
24.
DoumanoglouAKouskouridasRMalassiotisSKimTK (2016) Recovering 6D object pose and predicting next-best-view in the crowd. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3583–3592.
25.
EmbretsonSEReiseSP (2000) Item Response Theory for Psychologists. Lawrence Erlbaum Associates, Inc.
26.
FinnCAbbeelPLevineS (2017) Model-agnostic meta-learning for fast adaptation of deep networks. arXiv preprint arXiv:1703.03400.
27.
GalYGhahramaniZ (2016) Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In: International Conference on Machine Learning (ICML), pp. 1050–1059.
28.
GalYIslamRGhahramaniZ (2017) Deep Bayesian active learning with image data. arXiv preprint arXiv:1703.02910.
29.
GaninYUstinovaEAjakanHet al. (2015) Domain-adversarial training of neural networks. CoRR abs/1505.07818.
30.
GargRCarneiroGReidI (2016) Unsupervised CNN for single view depth estimation: Geometry to the rescue. In: European Conference on Computer Vision. New York: Springer, pp. 740–756.
31.
GodardCMac AodhaOBrostowGJ (2017) Unsupervised monocular depth estimation with left–right consistency. In: Computer Vision and Pattern Recognition (CVPR).
32.
GoldsteinEBBrockmoleJ (2016) Sensation and perception. Cengage Learning.
33.
GoodfellowIJMirzaMXiaoDCourvilleABengioY (2013) An empirical investigation of catastrophic forgetting in gradient-based neural networks. arXiv preprint arXiv:1312.6211.
34.
GoodfellowIJShlensJSzegedyC (2014) Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572.
35.
GreffKvan SteenkisteSSchmidhuberJ (2017) Neural expectation maximization. In: Advances in Neural Information Processing Systems (NIPS), pp. 6694–6704.
36.
GuSLillicrapTPSutskeverILevineS (2016) Continuous deep Q-learning with model-based acceleration. In: ICML 2016.
37.
GuoCPleissGSunYWeinbergerKQ (2017) On calibration of modern neural networks. arXiv preprint arXiv:1706.04599.
HäneCTulsianiSMalikJ (2017) Hierarchical surface prediction for 3D object reconstruction. arXiv preprint arXiv:1704.00710.
40.
HariharanBGirshickR (2016) Low-shot visual recognition by shrinking and hallucinating features. arXiv preprint arXiv:1606.02819.
41.
HeKGkioxariGDollárPGirshickR (2017) Mask R-CNN. In: IEEE International Conference on Computer Vision (ICCV).
42.
HeessNWayneGSilverDLillicrapTPErezTTassaY (2015) Learning continuous control policies by stochastic value gradients. In: Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, 7–12 December 2015, Montreal, Quebec, Canada, pp. 2944–2952. Available at: http://papers.nips.cc/paper/5796-learning-continuous-control-policies-by-stochastic-value-gradients (accessed 31 March 2018).
43.
HendrycksDGimpelK (2017) A baseline for detecting misclassified and out-of-distribution examples in neural networks. In: International Conference on Machine Learning (ICML).
44.
HesposSGredebaGvon HofstenCSpelkeE (2009) Occlusion is hard: Comparing predictive reaching for visible and hidden objects in infants and adults. Cognitive Science33(8): 1483–1502.
45.
HintonGVinyalsODeanJ (2015) Distilling the knowledge in a neural network. CoRR abs/1503.02531.
46.
HöferSRaffinAJonschkowskiRBrockOStulpF (2016) Unsupervised learning of state representations for multiple tasks. In: Deep Learning Workshop at the Conference on Neural Information Processing Systems (NIPS).
47.
JamesSJohnsE (2016) 3D simulation for robot arm control with deep Q-learning. ArXiv e-prints.
48.
JonschkowskiRBrockO (2015) Learning state representations with robotic priors. Autonomous Robots39(3): 407–428.
49.
JonschkowskiRBrockO (2016) End-to-end learnable histogram filters. In: Workshop on Deep Learning for Action and Interaction at the Conference on Neural Information Processing Systems (NIPS).
50.
JonschkowskiRHafnerRScholzJRiedmillerM (2017) Pves: Position-velocity encoders for unsupervised learning of structured state representations. arXiv preprint arXiv:1705.09805.
KaessMJohannssonHRobertsRIlaVLeonardJDellaertF (2012) iSAM2: Incremental smoothing and mapping using the Bayes tree. The International Journal of Robotics Research31(2): 216–235.
53.
KendallAGalY (2017) What uncertainties do we need in bayesian deep learning for computer vision?arXiv preprint arXiv:1703.04977.
54.
KerstenDMamassianPYuilleA (2004) Object perception as Bayesian inference. Annual Review of Psychology55: 271–304.
55.
KingmaDPMohamedSRezendeDJWellingM (2014) Semi-supervised learning with deep generative models. In: Advances in Neural Information Processing Systems (NIPS), pp. 3581–3589.
56.
KuindersmaSDeitsRFallonMet al. (2016) Optimization-based locomotion planning, estimation, and control design for the Atlas humanoid robot. Autonomous Robots40(3): 429–455.
57.
KümmerleRGrisettiGStrasdatHKonoligeKBurgardW (2011) g2o: A general framework for graph optimization. In: Proceedings of International Conference on Robotics and Automation (ICRA), pp. 3607–3613.
58.
KunzeLBeetzM (2015) Envisioning the qualitative effects of robot manipulation actions using simulation-based projections. Artificial Intelligence247: 352–380.
59.
LakeBMSalakhutdinovRTenenbaumJB (2015) Human-level concept learning through probabilistic program induction. Science350(6266): 1332–1338.
60.
LakshminarayananBPritzelABlundellC (2017) Simple and scalable predictive uncertainty estimation using deep ensembles. In: Advances in Neural Information Processing Systems (NIPS), pp. 6393–6395.
61.
LevineSAbbeelP (2014) Learning neural network policies with guided policy search under unknown dynamics. In: GhahramaniZWellingMCortesCLawrenceNDWeinbergerKQ (eds.) Advances in Neural Information Processing Systems 27. Curran Associates, Inc., pp. 1071–1079.
62.
LevineSFinnCDarrellTAbbeelP (2015) End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research17(1): 1334–1373.
63.
LillicrapTPHuntJJPritzelAet al. (2015) Continuous control with deep reinforcement learning. CoRR abs/1509.02971.
64.
LinDFidlerSUrtasunR (2013) Holistic scene understanding for 3D object detection with RGBD cameras. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1417–1424.
65.
LiuFShenCLinGReidI (2016a) Learning depth from single monocular images using deep convolutional neural fields. IEEE Transactions on Pattern Analysis and Machine Intelligence38(10): 2024–2039.
66.
LiuWAnguelovDErhanDet al. (2016b) SSD: Single shot multibox detector. In: European Conference on Computer Vision. New York: Springer, pp. 21–37.
LomonacoVMaltoniD (2017) Core50: a new dataset and benchmark for continuous object recognition. arXiv preprint arXiv:1705.03550.
69.
LongMCaoYWangJJordanMI (2015) Learning transferable features with deep adaptation networks. In: Proceedings of the 32nd International Conference on Machine Learning (ICML 2015), Lille, France, 6–11 July 2015, pp. 97–105.
70.
Lopez-PazDBottouLSchölkopfBVapnikV (2016) Unifying distillation and privileged information. CoRR abs/1511.03643.
71.
MacKayDJ (1992) A practical Bayesian framework for backpropagation networks. Neural Computation4(3): 448–472.
72.
MalmirMSikkaKForsterDFaselIMovellanJRCottrellGW (2017) Deep active object recognition by joint label and action prediction. Computer Vision and Image Understanding156: 128–137.
MensinkTVerbeekJPerronninFCsurkaG (2012) Metric learning for large scale image classification: Generalizing to new classes at near-zero cost. Computer Vision (ECCV 2012), pp. 488–501.
75.
MillerDNicholsonLDayoubFSünderhaufN (2017) Dropout sampling for robust object detection in open-set conditions. In: International Conference on Robotics and Automation (ICRA).
76.
MnihVBadiaAPMirzaMet al. (2016) Asynchronous methods for deep reinforcement learning. In: International Conference on Machine Learning (ICML).
77.
NealRM (1995) Bayesian learning for neural networks. PhD Thesis, University of Toronto.
78.
NguyenAYosinskiJCluneJ (2015a) Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 427–436.
79.
NguyenAYosinskiJCluneJ (2015b) Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 427–436.
80.
OlivaATorralbaA (2007) The role of context in object recognition. Trends in Cognitive Sciences11(12): 520–527.
81.
PapandreouGChenLCMurphyKPYuilleAL (2015) Weakly-and semi-supervised learning of a deep convolutional network for semantic image segmentation. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1742–1750.
82.
PatelVMGopalanRLiRChellappaR (2015) Visual domain adaptation: A survey of recent advances. IEEE Signal Processing Magazine32(3): 53–69.
83.
PengXSunBAliKSaenkoK (2015) Learning deep object detectors from 3D models. In: 2015 IEEE International Conference on Computer Vision (ICCV 2015), Santiago, Chile, 7–13 December 2015, pp. 1278–1286.
84.
PiagetJ (2013) The Construction of Reality in the Child. London: Routledge.
85.
PillaiSLeonardJ (2015) Monocular slam supported object recognition. In: Robotics: Science and Systems.
86.
PosnerIHadsellRRiedmillerMWulfmeierMPaulR (2016) Neural Information Processing Systems (NIPS) Workshop on Acting and Interacting in the Real World: Challenges in Robot Learning. Available at: http://sites.google.com/view/nips17robotlearning/home (accessed 31 March 2018).
87.
PovinelliDJ (2000) Folk Physics for Apes: The Chimpanzee’s Theory of How the World Works. Oxford: Oxford University Press.
88.
RasmusABerglundMHonkalaMValpolaHRaikoT (2015) Semi-supervised learning with ladder networks. In: Advances in Neural Information Processing Systems (NIPS), pp. 3546–3554.
RedmonJFarhadiA (2016) Yolo9000: better, faster, stronger. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
91.
RezendeDDanihelkaIGregorKet al. (2016) One-shot generalization in deep generative models. In: International Conference on Machine Learning, pp. 1521–1529.
92.
RichardWebsterBAnthonySEScheirerWJ (2016) Psyphy: A psychophysics driven evaluation framework for visual recognition. CoRR abs/1611.06448.
93.
RockI (1983) The Logic of Perception. Cambridge: MIT Press.
SadeghiFLevineS (2016) Cad2rl: Real single-image flight without a single real image. arXiv preprint arXiv:1611.04201.
97.
Salas-MorenoRFNewcombeRAStrasdatHKellyPHDavisonAJ (2013) SLAM++: Simultaneous localisation and mapping at the level of objects. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1352–1359.
98.
SantoroABartunovSBotvinickMWierstraDLillicrapT (2016) Meta-learning with memory-augmented neural networks. In: International Conference on Machine Learning, pp. 1842–1850.
99.
ScheirerWJAnthonySENakayamaKCoxDD (2014a) Perceptual annotation: Measuring human vision to improve computer vision. IEEE Transactions on Pattern Analysis and Machine Intelligence36(8): 1679–1686.
100.
ScheirerWJde Rezende RochaASapkotaABoultTE (2013a) Toward open set recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence35(7): 1757–1772.
101.
ScheirerWJde Rezende RochaASapkotaABoultTE (2013b) Toward open set recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence35(7): 1757–1772.
102.
ScheirerWJJainLPBoultTE (2014b) Probability models for open set recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence36(11): 2317–2324.
103.
SchierwagenA (2012) On reverse engineering in the brain and cognitive sciences. Natural Computing11(1): 141–150.
104.
SchmidtTHertkornKNewcombeRMartonZSuppaSFoxD (2015) Robust real-time tracking with visual and physical constraints for robot manipulation. In: Proceedings of the IEEE International Conference on Robotics and Automation (ICRA).
105.
SchulmanJLevineSMoritzPJordanMIAbbeelP (2015) Trust region policy optimization. In: Proceedings of the 32nd International Conference on Machine Learning (ICML).
106.
SchulmanJMoritzPLevineSJordanMAbbeelP (2016) High-dimensional continuous control using generalized advantage estimation. In: Proceedings of the International Conference on Learning Representations (ICLR).
107.
ShiLGriffithsTL (2009) Neural implementation of hierarchical bayesian inference by importance sampling. In: Proceedings of the Neural Information Processing Systems Conference (NIPS).
108.
SimonHA (1996) The Sciences of the Artificial. Cambridge, MA: MIT Press.
109.
SuHQiCRLiYGuibasLJ (2015) Render for CNN: viewpoint estimation in images using cnns trained with rendered 3d model views. In: 2015 IEEE International Conference on Computer Vision (ICCV 2015), Santiago, Chile, 7–13 December 2015, pp. 2686–2694.
110.
SünderhaufNDayoubFMcMahonSet al. (2016) Place categorization and semantic mapping on a mobile robot. In: 2016 IEEE International Conference on Robotics and Automation (ICRA). IEEE, pp. 5729–5736.
111.
SünderhaufNLeitnerJAbbeelPMilfordMCorkeP (2017) Robotics: Science and Systems (RSS) Workshop on New Frontiers for Deep Learning in Robotics. Available at: http://juxi.net/workshop/deep-learning-rss-2017/ (accessed 31 March 2018).
112.
SünderhaufNLeitnerJMilfordMet al. (2016) Robotics: Science and Systems (RSS) Workshop Are the Sceptics Right? Limits and Potentials of Deep Learning in Robotics. Available at: http://juxi.net/workshop/deep-learning-rss-2016/ (accessed 31 March 2018).
113.
SünderhaufNPhamTTLatifYMilfordMReidI (2017) Meaningful maps - object-oriented semantic mapping. In: International Conference on Intelligent Robots and Systems (IROS).
TamarALevineSAbbeelP (2016) Value iteration networks. CoRR abs/1602.02867.
116.
ThrunSBurgardWFoxD (2005) Probabilistic Robotics. Cambridge, MA: The MIT Press.
117.
TobinJFongRRayASchneiderJZarembaWAbbeelP (2017) Domain randomization for transferring deep neural networks from simulation to the real world. In: 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, pp. 23–30.
118.
TodorovEErezTTassaY (2012) Mujoco: A physics engine for model-based control. In: International Conference on Intelligent Robots and Systems IROS.
119.
TorralbaAEfrosAA (2011) Unbiased look at dataset bias. In: Computer Vision and Pattern Recognition (CVPR). IEEE, pp. 1521–1528.
120.
TzengEDevinCHoffmanJet al. (2015a) Towards adapting deep visuomotor representations from simulated to real environments. CoRR abs/1511.07111.
121.
TzengEHoffmanJDarrellTSaenkoK (2015b) Simultaneous deep transfer across domains and tasks. In: 2015 IEEE International Conference on Computer Vision (ICCV 2015), Santiago, Chile, 7–13 December 2015, pp. 4068–4076.
122.
TzengEHoffmanJZhangNSaenkoKDarrellT (2014) Deep domain confusion: Maximizing for domain invariance. CoRR abs/1412.3474.
123.
VinyalsOBlundellCLillicrapTet al. (2016) Matching networks for one shot learning. In: Advances in Neural Information Processing Systems, pp. 3630–3638.
124.
Von HelmholtzH (1867) Handbuch der physiologischen Optik, volume 9. Voss.
125.
WangYXHebertM (2016) Learning to learn: Model regression networks for easy small sample learning. In: European Conference on Computer Vision. Berlin: Springer, pp. 616–634.
126.
WatterMSpringenbergJBoedeckerJRiedmillerM (2015) Embed to control: A locally linear latent dynamics model for control from raw images. In: Advances in Neural Information Processing Systems (NIPS), pp. 2728–2736.
127.
WilsonRCFinkelLH (2009) A neural implementation of the Kalman Filter. In: Proceedings of the Neural Information Processing Systems Conference (NIPS).
128.
WuJYildirimILimJJFreemanBTenenbaumJ (2015a) Galileo: Perceiving physical object properties by integrating a physics engine with deep learning. In: Advances in Neural Information Processing Systems (NIPS), pp. 127–135.
129.
WuZSongSKhoslaAYuFZhangLTangXXiaoJ (2015b) 3d shapenets: A deep representation for volumetric shapes. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
130.
YanXYangJYumerEGuoYLeeH (2016) Perspective transformer nets: Learning single-view 3D object reconstruction without 3D supervision. In: Advances in Neural Information Processing Systems (NIPS), pp. 1696–1704.
131.
YildirimIGerstenbergTSaeedBToussaintMTenenbaumJ (2017) Physical problem solving: Joint planning with symbolic, geometric, and dynamic constraints. arXiv preprint arXiv:1707.08212.
132.
ZhangCBengioSHardtMRechtBVinyalsO (2016) Understanding deep learning requires rethinking generalization. CoRR abs/1611.03530.
133.
ZhangFLeitnerJMilfordMUpcroftBCorkeP (2015) Towards vision-based deep reinforcement learning for robotic motion control. In: Australasian Conference on Robotics and Automation.
134.
ZhangHPatelV (2016) Sparse representation-based open set recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence PP(99): 1–1.
135.
ZhangYBaiMKohliPIzadiSXiaoJ (2017) Deepcontext: context-encoding neural pathways for 3D holistic scene understanding. In: IEEE International Conference on Computer Vision (ICCV).
136.
ZhuRGaloogahiHKWangCLuceyS (2017) Rethinking reprojection: Closing the loop for pose-aware shape reconstruction from a single image. In: IEEE International Conference on Computer Vision (ICCV). IEEE, pp. 57–65.
137.
ZhuYMottaghiRKolveEet al. (2016) Target-driven visual navigation in indoor scenes using deep reinforcement learning. CoRR abs/1609.05143.
