Tool manipulation is vital for facilitating robots to complete challenging task goals. It requires reasoning about the desired effect of the task and, thus, properly grasping and manipulating the tool to achieve the task. Most work in robotics has focused on task-agnostic grasping, which optimizes for only grasp robustness without considering the subsequent manipulation tasks. In this article, we propose the Task-Oriented Grasping Network (TOG-Net) to jointly optimize both task-oriented grasping of a tool and the manipulation policy for that tool. The training process of the model is based on large-scale simulated self-supervision with procedurally generated tool objects. We perform both simulated and real-world experiments on two tool-based manipulation tasks: sweeping and hammering. Our model achieves overall 71.1% task success rate for sweeping and 80.0% task success rate for hammering.
BaberC (2003) Cognition and Tool Use: Forms of Engagement in Human and Animal Use of Tools. Boca Raton, FL: CRC Press.
2.
BohgJMoralesAAsfourTKragicD (2014) Data-driven grasp synthesis – a survey. IEEE Transactions on Robotics30(2): 289–309.
3.
BousmalisKIrpanAWohlhartP, et al. (2017) Using simulation and domain adaptation to improve efficiency of deep robotic grasping. arXiv preprint arXiv:1709.07857.
4.
BousmalisKIrpanAWohlhartP, et al. (2018) Using simulation and domain adaptation to improve efficiency of deep robotic grasping. In: 2018 IEEE International Conference on Robotics and Automation. IEEE, pp. 4243–4250.
5.
BrownSSammutC (2012) Tool use and learning in robots. In: Encyclopedia of the Sciences of Learning. Berlin: Springer, pp. 3327–3330.
6.
CiocarlieMTAllenPK (2009) Hand posture subspaces for dexterous robotic grasping. The International Journal of Robotics Research28(7): 851–867.
7.
CoumansEBaiY (2016) PyBullet, a Python module for physics simulation, games, robotics and machine learning. Available at: http://pybullet.org/ (accessed August 2019).
8.
DangHAllenPK (2012) Semantic grasping: Planning robotic grasps functionally suitable for an object manipulation task. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 1311–1317.
9.
DetryRPaponJMatthiesLH (2017) Task-oriented grasping with semantic and geometric scene understanding. In: 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 3266–3273.
10.
DoTTNguyenAReidICaldwellDGTsagarakisNG (2017) Affordancenet: An end-to-end deep learning approach for object affordance detection. arXiv preprint arXiv:1709.07326.
11.
DogarMSrinivasaS (2011) A framework for push-grasping in clutter. In: Robotics: Science and Systems.
12.
EitelAHauffNBurgardW (2017) Learning to singulate objects using a push proposal network. arXiv preprint arXiv:1707.08101.
13.
FangKBaiYHinterstoisserSSavareseSKalakrishnanM (2018a) Multi-task domain adaptation for deep learning of instance grasping from simulation. In: IEEE International Conference on Robotics and Automation.
14.
FangKZhuYGargA, et al. (2018b) Learning task-oriented grasping for tool manipulation from simulated self-supervision. In: Robotics: Science and Systems.
15.
FerrariCCannyJ (1992) Planning optimal grasps. In: IEEE International Conference on Robotics and Automation, pp. 2290–2295.
16.
FitzpatrickPMettaGNataleLRaoSSandiniG (2003) Learning about objects through action-initial steps towards artificial cognition. In: IEEE International Conference on Robotics and Automation, Vol. 3, pp. 3140–3145.
17.
GibsonJJ (1979) The Ecological Approach to Visual Perception. Houghton Mifflin.
18.
GoldfederCCiocarlieMDangHAllenPK (2009) The Columbia grasp database. In: IEEE International Conference on Robotics and Automation, pp. 1710–1716.
19.
HartleyRZissermanA (2003) Multiple View Geometry in Computer Vision. Cambridge: Cambridge University Press.
20.
HaschkeRSteilJJSteuwerIRitterH (2005) Task-oriented quality measures for dextrous grasping. In: Proceedings 2005 IEEE International Symposium on Computational Intelligence in Robotics and Automation (CIRA 2005), pp. 689–694.
21.
HeKZhangXRenSSunJ (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778.
22.
HerzogAPastorPKalakrishnanMRighettiLAsfourTSchaalS (2012) Template-based learning of grasp selection. In: 2012 IEEE International Conference on Robotics and Automation, pp. 2379–2384.
23.
IoffeSSzegedyC (2015) Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: International Conference on Machine Learning, pp. 448–456.
24.
JainRInamuraT (2011) Learning of tool affordances for autonomous tool manipulation. In: 2011 IEEE/SICE International Symposium on System Integration (SII), pp. 814–819.
25.
JangEVijaynarasimhanSPastorPIbarzJLevineS (2017) End-to-end learning of semantic grasping. arXiv preprint arXiv:1707.01932.
26.
JiangYLimMSaxenaA (2012) Learning object arrangements in 3D scenes using human context. arXiv preprint arXiv:1206.6462.
27.
KalashnikovDIrpanAPastorP, et al. (2018) QT-OPT: Scalable deep reinforcement learning for vision-based robotic manipulation. arXiv preprint arXiv:1806.10293.
28.
KapplerDBohgJSchaalS (2015) Leveraging big data for grasp planning. In: IEEE International Conference on Robotics and Automation. IEEE, pp. 4304–4311.
29.
KatzDVenkatramanAKazemiMBagnellJAStentzA (2014) Perceiving, learning, and exploiting object affordances for autonomous pile manipulation. Autonomous Robots37(4): 369–382.
30.
KokicMStorkJAHausteinJAKragicD (2017) Affordance detection for task-specific grasping using big ol’ neural nets. In: 2017 IEEE-RAS 17th International Conference on Humanoid Robotics (Humanoids). IEEE, pp. 91–98.
31.
KomizunaiSTeppeiTFumiyaNNomuraYOwaT (2008) Experiments on hammering a nail by a humanoid robot HRP-2. In: Proceedings of the 17th CISM-IFToMM Symposium on Robot Design, Dynamics, and Control.
32.
KoppulaHSGuptaRSaxenaA (2013) Learning human activities and object affordances from RGB-D videos. The International Journal of Robotics Research32(8): 951–970.
33.
KroemerOUgurEÖztopEPetersJ (2012) A kernel-based approach to direct action perception. In: 2012 IEEE International Conference on Robotics and Automation, pp. 2605–2610.
34.
LaskeyMChuckCLeeJ, et al. (2017) Comparing human-centric and robot-centric sampling for robot deep learning from demonstrations. In: IEEE International Conference on Robotics and Automation, pp. 358–365.
35.
LenzILeeHSaxenaA (2015) Deep learning for detecting robotic grasps. The International Journal of Robotics Research34(4–5): 705–724.
36.
LevineSPastorPKrizhevskyAIbarzJQuillenD (2016) Learning hand–eye coordination for robotic grasping with deep learning and large-scale data collection. The International Journal of Robotics Research37(4–5): 421–436.
37.
LiZSastrySS (1988) Task-oriented optimal grasping by multifingered robot hands. IEEE Journal on Robotics and Automation4(1): 32–44.
38.
LynchKMMasonMT (1996) Stable pushing: Mechanics, controllability, and planning. The International Journal of Robotics Research15(6): 533–556.
39.
MahlerJLiangJNiyazS, et al. (2017) Dex-Net 2.0: Deep learning to plan robust grasps with synthetic point clouds and analytic grasp metrics. arXiv preprint arXiv:1703.09312.
40.
MahlerJPokornyFTHouB, et al. (2016) Dex-Net 1.0: A cloud-based network of 3D objects for robust grasp planning using a multi-armed bandit model with correlated rewards. In: IEEE International Conference on Robotics and Automation, pp. 1957–1964.
41.
MamouKGhorbelF (2009) A simple and efficient approach for 3D mesh approximate convex decomposition. In: 2009 16th IEEE International Conference on Image Processing.
42.
MarTTikhanoffVMettaGNataleL (2015) Self-supervised learning of grasp dependent tool affordances on the iCub humanoid robot. In: IEEE International Conference on Robotics and Automation, pp. 3200–3206.
43.
MarTTikhanoffVMettaGNataleL (2017) Self-supervised learning of tool affordances from 3D tool representation through parallel som mapping. In: IEEE International Conference on Robotics and Automation. IEEE, pp. 894–901.
44.
MeriçliTVelosoMAknHL (2015) Push-manipulation of complex passive mobile objects using experimentally acquired motion models. Autonomous Robots38(3): 317–329.
45.
MillerATKnoopSChristensenHIAllenPK (2003) Automatic grasp planning using shape primitives. In: IEEE International Conference on Robotics and Automation, Vol. 2, pp. 1824–1829.
46.
MorrisonDTowAWMcTaggartM, et al. (2018) Cartman: The low-cost Cartesian manipulator that won the Amazon robotics challenge. In: 2018 IEEE International Conference on Robotics and Automation, pp. 7757–7764.
47.
OsiurakFJarryCLe GallD (2010) Grasping the affordances, understanding the reasoning: toward a dialectical theory of human tool use. Psychological Review117(2): 517.
48.
PintoLGuptaA (2016) Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours. In: IEEE International Conference on Robotics and Automation, pp. 3406–3413.
49.
PratsMSanzPJDel PobilAP (2007) Task-oriented grasping using hand preshapes and task frames. In: IEEE International Conference on Robotics and Automation. IEEE, pp. 1794–1799.
50.
RodriguezAMasonMTFerryS (2012) From caging to grasping. The International Journal of Robotics Research31(7): 886–900.
51.
RubinsteinRYKroeseDP (2004) The cross-entropy method: A unified approach to monte carlo simulation, randomized optimization and machine learning. In: Information Science & Statistics. New York: Springer Verlag.
52.
SaxenaADriemeyerJNgAY (2008) Robotic grasping of novel objects using vision. The International Journal of Robotics Research27(2): 157–173.
53.
SongDHuebnerKKyrkiVKragicD (2010) Learning task constraints for robot grasping using graphical models. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 1579–1585.
54.
StoytchevA (2005) Behavior-grounded representation of tool affordances. In: IEEE International Conference on Robotics and Automation, pp. 3060–3065.
55.
ten PasAGualtieriMSaenkoKPlattR (2017) Grasp pose detection in point clouds. The International Journal of Robotics Research36: 1455–1473.
56.
TobinJZarembaWAbbeelP (2017) Domain randomization and generative models for robotic grasping. CoRR abs/1710.06425.
57.
VaradarajanKMVinczeM (2012) Afrob: The affordance network ontology for robots. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 1343–1350.
58.
ViereckUPasAtSaenkoKPlattR (2017) Learning a visuomotor controller for real world robotic grasping using easily simulated depth images. arXiv preprint arXiv:1706.04652.
59.
WeiszJAllenPK (2012) Pose error robust grasping from contact wrench space metrics. In: IEEE International Conference on Robotics and Automation, pp. 557–562.
60.
WilliamsonMM (1999) Robot arm control exploiting natural dynamics. PhD Thesis, Massachusetts Institute of Technology.
61.
YanXHsuJKhansariM, et al. (2018) Learning 6-DOF grasping interaction via deep geometry-aware 3D representations. In: 2018 IEEE International Conference on Robotics and Automation. IEEE, pp. 1–9.
62.
ZhuYFathiAFei-FeiL (2014) Reasoning about object affordances in a knowledge base representation. In: European Conference on Computer Vision.
63.
ZhuYZhaoYChun ZhuS (2015) Understanding tools: Task-oriented object modeling, learning and recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2855–2864.
