We present online multi-modal perception systems for extracting kinematic and dynamic models of articulated objects from physical interactions with the environment. The systems rely on a RGB-D stream, contact wrenches, and proprioception. The proposed systems share an algorithmic foundation: they are based on an architecture of coupled recursive estimation processes. We present and advocate this architecture as a general, versatile, and robust solution for online interactive perception problems. We validate the architecture in extensive experiments to extract kinematic models interactively, varying the appearance, size, structure, and dynamic properties of objects for different tasks and under different environmental conditions. In addition, we experimentally show that the information acquired by the online perception systems enables robot manipulation of articulated objects. Furthermore, we discuss the relationship between the proposed architecture for robot perception and insights about biological perception systems.
AhissarMHochsteinS (2004) The reverse hierarchy theory of visual perceptual learning. Trends in Cognitive Science8(10): 457–464.
2.
AtkesonCGAnCHHollerbachJM (1986) Estimation of inertial parameters of manipulator loads and links. The International Journal of Robotics Research5(3): 101–119.
3.
Bar-ShalomYLiXKirubarajanT (2001) Estimation with Applications to Tracking and Navigation. New York: John Wiley & Sons, Inc.
4.
BarfootTDFurgalePT (2014) Associating uncertainty with three-dimensional poses for use in estimation problems. IEEE Transactions on Robotics30(3): 679–693.
5.
BohgJHausmanKSankaranB, et al. (2017) Interactive perception: Leveraging action in perception and perception in action. IEEE Transactions on Robotics33(6): 1273–1291.
6.
BruyninckxHSchutterJD (1996) Specification of force-controlled actions in the “task frame formalism” - a synthesis. IEEE Transactions on Robotics12(4): 581–589.
7.
CannyJ (1986) A computational approach to edge detection. IEEE Transactions on Pattern Analysis and Machine Intelligence8(6): 679–698.
8.
ClarkJJYuilleAL (1990) Data Fusion for Sensory Information Processing Systems. Norwell, MA: Kluwer.
9.
CosteiraJPKanadeT (1998) A multibody factorization method for independently moving objects. International Journal of Computer Vision29(3): 159–179.
10.
DagumPGalperAHorvitzE (1992) Dynamic network models for forecasting. In: Proceedings of the Eighth International Conference on Uncertainty in Artificial Intelligence (UAI’92), San Francisco, CA, USA, pp. 41–48.
11.
DeimelRBrockO (2016) A novel type of compliant and underactuated robotic hand for dexterous grasping. The International Journal of Robotics Research35(1–3): 161–185.
12.
DupontPE (1990) Friction modeling in dynamic robot simulation. In: Proceedings of the IEEE International Conference on Robotics and Automotion (ICRA), pp. 1370–1376.
13.
EndresFTrinkleJBurgardW (2013) Learning the dynamics of doors for robotic manipulation. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 3543–3549.
14.
EppnerCMartín-MartínRBrockO (2018) Physics-based selection of informative actions for interactive perception. In: Proceedings of the IEEE International Conference on Robotics and Automotion (ICRA).
15.
FangFKerstenDMurraySO (2008) Perceptual grouping and inverse fMRI activity patterns in human visual cortex. Journal of Vision8(7): 2.
16.
FenziMDragonRLeal-TaixéLRosenhahnBOstermannJ (2012) 3D object recognition and pose estimation for multiple objects using multi-prioritized ransac and model updating. In: PinzAPockTBischofHLeberlF (eds.) Pattern Recognition. Berlin: Springer, pp. 123–133.
17.
ForsterCPizzoliMScaramuzzaD (2014) SVO: Fast semi-direct monocular visual odometry. In: Proceedings of the IEEE International Conference on Robotics and Automotion (ICRA), pp. 15–22.
18.
FortmannTEBar-ShalomYScheffeM (1980) Multi-target tracking using joint probabilistic data association. In: Proceedings of the IEEE International Conference on Decision and Control. IEEE, pp. 807–812.
19.
Garcia CifuentesCIssacJWüthrichMSchaalSBohgJ (2017) Probabilistic articulated real-time tracking for robot manipulation. IEEE Robotics and Automation Letters2(2): 577–584.
20.
GibsonJJ (1966) The Senses Considered as Perceptual Systems. 1st ed.Greenwood Press Reprint.
21.
GibsonJJ (1979) The Ecological Approach to Visual Perception. London: Routledge.
22.
GollischTMeisterM (2010) Eye smarter than scientists believed: Neural computations in circuits of the retina. Neuron65(2): 150–164.
HaarnojaTAjayALevineSAbbeelP (2016) Backprop KF: Learning discriminative deterministic state estimators. In: Advances in Neural Information Processing Systems (NIPS), pp. 4376–4384.
25.
HausmanKNiekumSOsentoskiSSukhatmeGS (2015) Active articulation model estimation through interactive perception. In: Proceedings of the IEEE International Conference on Robotics and Automotion (ICRA), pp. 3305–3312.
26.
HeDKerstenDFangF (2012) Opposite modulation of high-and low-level visual aftereffects by perceptual grouping. Current Biology22(11): 1040–1045.
27.
HebertPHudsonNMaJ, et al. (2012) Combined shape, appearance and silhouette for simultaneous manipulator and object tracking. In: Proceedings of the IEEE International Conference on Robotics and Automotion (ICRA), pp. 2405–2412.
28.
HeldRHeinA (1963) Movement-produced stimulation in the development of visually guided behavior. Journal of Comparative and Physiological Psychology56(5): 872–876.
29.
HerzogMH (2018) Perceptual grouping. Current Biology28(12): R687–R688.
30.
HesposSJvanMarleK (2012) Physics for infants: Characterizing the origins of knowledge about objects, substances, and number. Wiley Interdisciplinary Reviews: Cognitive Science3(1): 19–27.
31.
HochsteinSAhissarM (2002) View from the top: Hierarchies and reverse hierarchies in the visual system. Neuron36: 791–804.
32.
HuangXWalkerIBirchfieldS (2012) Occlusion-aware reconstruction and manipulation of 3D articulated objects. Proceedings of the IEEE International Conference on Robotics and Automotion (ICRA), pp. 1365–1371.
33.
IlonenJBohgJKyrkiV (2014) Three-dimensional object reconstruction of symmetric objects by fusing visual and tactile sensing. The International Journal of Robotics Research33(2): 321–341.
34.
JainANguyenHRathMOkermanJKempCC (2010) The complex structure of simple devices: A survey of trajectories and forces that open doors and drawers. In: Proceedings of the IEEE, RAS and EMBS International Conference on Biomedical Robotics and Biomechatronics (BioRob), pp. 184–190.
35.
JonasJBSchmidtAMMüller-BerghJSchlötzer-SchrehardtUNaumannG (1992) Human optic nerve fiber count and optic disc size. Investigative Ophthalmology and Visual Science33(6): 2012–2018.
36.
JonschkowskiRRastogiDBrockO (2018) Differentiable particle filters: End-to-end learning with algorithmic priors. In: Proceedings of Robotics: Science and Systems, Pittsburgh, PA. DOI: 10.15607/RSS.2018.XIV.001.
37.
KarayiannidisYSmithCBarrientosFEVÖgrenPKragicD (2016) An adaptive control approach for opening doors and drawers under uncertainties. IEEE Transactions on Robotics32: 161–175.
38.
KatzDKazemiMBagnellJAStentzA (2013) Interactive segmentation, tracking, and kinematic modeling of unknown articulated objects. In: Proceedings of the IEEE International Conference on Robotics and Automotion (ICRA), pp. 5003–5010.
39.
KatzDOrtheyABrockO (2014) Interactive perception of articulated objects. In: KhatibOKumarVSukhatmeG (eds.) Experimental Robotics (Springer Tracts in Advanced Robotics, Vol. 79). Berlin: Springer, pp. 301–315.
40.
KerstenDYuilleA (2014) Inferential models of the visual cortical hierarchy. The Cognitive Neurosciences. Cambridge, MA: MIT Press, pp. 389–404.
41.
KhoshelhamKElberinkSO (2012) Accuracy and resolution of kinect depth data for indoor mapping applications. Sensors12(2): 1437–1454.
42.
KristjánssonÁEyjólfsdóttirKÓJónsdóttirAArnkelssonG (2010) Temporal consistency is currency in shifts of transient visual attention. PloS one5(10): e13660.
43.
KwokCCTFoxD (2004) Map-based multiple model tracking of a moving object. In: RoboCup 2004: Robot Soccer World Cup VIII, pp. 18–33.
44.
LeeTSMumfordD (2003) Hierarchical Bayesian inference in the visual cortex. Journal of the Optical Society of America20(7): 1434–1448.
45.
LiaoLPattersonDJFoxDKautzH (2007) Learning and inferring transportation routines. Artificial Intelligence171(5): 311–331.
46.
LivingstoneMHubelD (1988) Segregation of form, color, movement, and depth - anatomy, physiology, and perception. Science240(4853): 740–749.
47.
MaDHollerbachJM (1996) Identifying mass parameters for gravity compensation and automatic torque sensor calibration. In: Proceedings of the IEEE International Conference on Robotics and Automotion (ICRA), Vol. 1, pp. 661–666.
48.
MaljkovicVMartiniP (2005) Implicit short-term memory and event frequency effects in visual search. Vision Research45(21): 2831–2846.
49.
MaljkovicVNakayamaK (1994) Priming of pop-out: I. Role of features. Memory and Cognition22(6): 657–672.
50.
Martín-MartínRBrockO (2014) Online interactive perception of articulated objects with multi-level recursive estimation based on task-specific priors. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 2494–2501.
51.
Martín-MartínRBrockO (2017) Cross-modal interpretation of multi-modal sensor streams in interactive perception based on coupled recursion. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 3289–3295.
52.
McGurkHMacDonaldJ (1976) Hearing lips and seeing voices. Nature264(5588): 746–748.
53.
MumfordD (1992) On the computational architecture of the neocortex. Biological Cybernetics66(3): 241–251.
54.
NiemiPNäätänenR (1981) Foreperiod and simple reaction time. Psychological Bulletin89(1): 133.
55.
NistérDNaroditskyOBergenJR (2004) Visual odometry. In: Proceedings of the IEEE Computer Vision and Pattern Recognition (CVPR), Vol. 1.
56.
PfluegerMSukhatmeGS (2015) Multi-step planning for robotic manipulation. In: Proceedings of the IEEE International Conference on Robotics and Automotion (ICRA), pp. 2496–2501.
57.
QiuCBurtonPCKerstenDOlmanCA (2016) Responses in early visual areas to contour integration are context dependent. Journal of Vision16(8): 19–19.
58.
RaoRPBallardDH (1999) Predictive coding in the visual cortex: A functional interpretation of some extra-classical receptive-field effects. Nature Neuroscience2(1): 79–87.
59.
RaoRPNBallardDH (1997) Dynamic model of visual recognition predicts neural response properties in the visual cortex. Neural Computation9(4): 721–763.
60.
ReidD (1979) An algorithm for tracking multiple targets. IEEE Transactions on Automatic Control24(6): 843–854.
61.
RojasR (1996) Neural Networks: A Systematic Introduction. New York: Springer-Verlag.
62.
RossDTarlowDZemelR (2008) Unsupervised learning of skeletons from motion. In: Proceedings of the European Conference on Computer Vision, pp. 560–573.
63.
SchiebenerDUdeAAsfourT (2014) Physical interaction for segmentation of unknown textured and non-textured rigid objects. In: Proceedings of the IEEE International Conference on Robotics and Automotion (ICRA). pp. 4959–4966.
64.
ShiJTomasiC (1994) Good features to track. In: Proceedings of the IEEE Computer Vision and Pattern Recognition (CVPR), pp. 593–600.
65.
SimonHA (1996) The Sciences of the Artificial. Cambridge, MA: MIT Press.
66.
SinapovJBergquistTSchenckCOhiriUGriffithSStoytchevA (2011) Interactive object recognition using proprioceptive and auditory feedback. The International Journal of Robotics Research30(10): 1250–1262.
67.
SmoljkicGBorghesanGReynaertsDSchutterJDSlotenJVPoortenEV (2015) Constraint-based interaction control of robots featuring large compliance and deformation. IEEE Transactions on Robotics31(5): 1252–1260.
68.
SpelkeESVishtonPVon HofstenC (1995) Object perception, object-directed action, and physical knowledge in infancy. In: The Cognitive Neurosciences. Cambridge, MA: MIT Press, pp. 165–179.
69.
StilmanM (2007) Task constrained motion planning in robot joint space. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 3074–3081.
70.
SturmJJainAStachnissCKempCCBurgardW (2010a) Operating articulated objects based on experience. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 2739–2744.
71.
SturmJKonoligeKStachnissCBurgardW (2010b) Vision-based detection for learning articulation models of cabinet doors and drawers in household environments. In: Proceedings of the IEEE International Conference on Robotics and Automotion (ICRA), pp. 362–368.
72.
SturmJStachnissCBurgardW (2011) A probabilistic framework for learning kinematic models of articulated objects. Journal of Artificial Intelligence Research41(2): 477–526.
73.
SturmJStachnissCPradeepVPlagemannCKonoligeKBurgardW (2009) Learning kinematic models for articulated objects. In: Proceedings of the International Joint Conference on Artificial Intelligence, pp. 1851–1856.
74.
TatavartyGBhatnagarRYoungB (2007) Discovery of temporal dependencies between frequent patterns in multivariate time series. In: IEEE Symposium on Computational Intelligence and Data Mining, pp. 688–696.
75.
TomasiCKanadeT (1991) Detection and tracking of point features. Technical Report, Carnegie Mellon University.
76.
TomasiCKanadeT (1992) Shape and motion from image streams under orthography: A factorization method. International Journal of Computer Vision9(2): 137–154.
77.
TresadernPReidI (2005) Articulated structure from motion by factorization. In: Proceedings of the IEEE Computer Vision and Pattern Recognition (CVPR), Vol. 2, pp. 1110–1115.
78.
TriggsBMcLauchlanPFHartleyRIFitzgibbonAW (2000) Bundle adjustment — a modern synthesis. In: TriggsBZissermanASzeliskiR (eds.) Vision Algorithms: Theory and Practice. Berlin: Springer, pp. 298–372.
79.
van HoofHKroemerOBen AmorHPetersJ (2012) Maximally informative interaction learning for scene exploration. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 5152–5158.
80.
WangJOlsonE (2016) Apriltag 2: Efficient and robust fiducial detection. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 4193–4198.
81.
WattRJPhillipsWA (2000) The function of dynamic grouping in vision. Trends in Cognitive Science4(12): 447–454.
82.
XinjilefuXFengSHuangWAtkesonCG (2014) Decoupled state estimation for humanoids using full-body dynamics. In: Proceedings of the IEEE International Conference on Robotics and Automotion (ICRA), pp. 195–201.
83.
XuKHuangHShiY, et al. (2015) Autoscanning for coupled scene reconstruction and proactive object analysis. ACM Transactions on Graphics34(6): 177.
84.
YanJPollefeysM (2006) Automatic kinematic chain building from feature trajectories of articulated objects. In: Proceedings of the IEEE Computer Vision and Pattern Recognition (CVPR), Vol. 1, pp. 712–719.
