RGB-D cameras, which give an RGB image together with depths, are becoming increasingly popular for robotic perception. In this paper, we address the task of detecting commonly found objects in the three-dimensional (3D) point cloud of indoor scenes obtained from such cameras. Our method uses a graphical model that captures various features and contextual relations, including the local visual appearance and shape cues, object co-occurrence relationships and geometric relationships. With a large number of object classes and relations, the model’s parsimony becomes important and we address that by using multiple types of edge potentials. We train the model using a maximum-margin learning approach. In our experiments concerning a total of 52 3D scenes of homes and offices (composed from about 550 views), we get a performance of 84.06% and 73.38% in labeling office and home scenes respectively for 17 object classes each. We also present a method for a robot to search for an object using the learned model and the contextual information available from the current labelings of the scene. We applied this algorithm successfully on a mobile robot for the task of finding 12 object classes in 10 different offices and achieved a precision of 97.56% with 78.43% recall.1
AnguelovDTaskarBChatalbashevVKollerDGuptaDHeitzGNgA (2005) Discriminative learning of markov random fields for segmentation of 3d scan data. In Computer Vision and Pattern Recognition (CVPR), San Diego, USA, 20–26 June 2005.
2.
BatraDSaxenaA (2012) Learning the right model: Efficient max-margin learning in laplacian crfs. In Computer Vision and Pattern Recognition (CVPR), Providence, USA, 16–21 June 2012.
Collet RomeaASrinivasaSHebertM (2011) Structure discovery in multi-modal data: a region-based approach. In IEEE International Conference on Robotics and Automation, Shanghai, China, 9–13 May 2011, pp. 665–671.
6.
CsurkaGDanceCFanLWillamowskiJBrayC (2004) Visual categorization with bags of keypoints. In Workshop on Statistical Learning in Computer Vision, European Conference on Computer Vision (ECCV), Prague11–14 May 2004. Czech Technical University, Prague, 12 May 2004, pp. 22.
7.
DalalNTriggsB (2005) Histograms of oriented gradients for human detection. In Computer Vision and Pattern Recognition (CVPR), San Diego, USA, 20–26 June 2005.
8.
DenzlerJBrownC (2002) Information theoretic sensor data selection for active object recognition and state estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence24(2): 145–157.
9.
DimaCVandapelNHebertM (2004) Classifier fusion for outdoor obstacle detection. In IEEE International Conference on Robotics and Automation, Barcelona, Spain, 18–22 April 2004.
10.
DivvalaSHoiemDHaysJEfrosAHebertM (2009) An empirical study of context in object detection. In Computer Vision and Pattern Recognition (CVPR), Miami, USA, 20–26 June, 2009.
11.
DouillardBFoxDRamosFDurrant-WhyteH (2011) Classification and semantic mapping of urban environments. International Journal of Robotics Research30(1): 5–32.
12.
FelzenszwalbPMcAllesterDRamananD (2008) A discriminatively trained, multiscale, deformable part model. In Computer Vision and Pattern Recognition (CVPR), Anchorage, USA, 24–26 June 2008.
13.
FinleyTJoachimsT (2008) Training structural svms when exact inference is intractable. In International Conference on Machine Learning (ICML), Helsinki, Finland, 5–9 July 2008.
14.
GolovinskiyAKimVGFunkhouserT (2009) Shape-based recognition of 3d point clouds in urban environments. In International Conference on Computer Vision (ICCV), Kyoto, Japan, 29 September– 2 October 2009.
15.
GouldSBaumstarckPQuigleyMNgAYKollerD (2008) Integrating visual and range data for robotic object detection. In European Conference on Computer Vision Workshop Multi-camera Multi-modal (M2SFA2).
16.
HammerPHansenPSimeoneB (1984) Roof duality, complementation and persistency in quadratic 0–1 optimization. Mathematical Programming28(2): 121–155.
17.
HedauVHoiemDForsythD (2010) Thinking inside the box: Using appearance models and context based on room geometry. In European Conference on Computer Vision, Crete, Greece, 5–11 September 2010.
18.
HeitzGGouldSSaxenaAKollerD (2008) Cascaded classification models: Combining models for holistic scene understanding. In Neural Information Processing Systems (NIPS), Vancouver, Canada, 8–13 December 2008.
19.
HeitzGKollerD (2008) Learning spatial context: Using stuff to find things. In European Conference on Computer Vision, Marseille, France, 12–18 October 2008.
20.
HoiemDEfrosAAHebertM (2006) Putting objects in perspective. In Computer Vision and Pattern Recognition.
21.
JiaZSaxenaAChenT (2011a) Robotic object detection: Learning to improve the classifiers using sparse graphs for path planning. In International Joint Conference on Artificial Intelligence (IJCAI), Barcelona, Spain16–22 July.
22.
JiaZSaxenaAChenT (2011b) Robotic object detection: Learning to improve the classifiers using sparse graphs for path planning. In International Joint Conference on Artificial Intelligence (IJCAI), Barcelona, Spain16–22 July 2011.
23.
JiangYLimMZhengCSaxenaA (2012) Learning to place new objects in a scene. Internationnal Journal of Robotics Research31(9): 1021–1043.
24.
JoachimsTFinleyTYuC (2009) Cutting-plane training of structural SVMs. Machine Learning77(1): 27–59.
25.
KoppulaHAnandAJoachimsTSaxenaA (2011a) Semantic labeling of 3d point clouds for indoor scenes. In Neural Information Processing Systems (NIPS), Granada, Spain, 12–17 December 2011.
26.
KoppulaH. SAnandAJoachimsTSaxenaA (2011b) Labeling 3d scenes for personal assistant robots. In RSS workshop RGB-D: Advanced Reasoning with Depth Cameras.
27.
LaffertyJ. DMcCallumAPereiraF.C. N (2001) Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In International Conference on Machine Learning (ICML), Williamstown, USA, 28 June–1 July 2001.
28.
LaiKBoLRenXFoxD (2011a) A large-scale hierarchical multi-view rgb-d object dataset. In IEEE International Conference on Robotics and Automation, Shanghai, China, 9–13 May 2011.
29.
LaiKBoLRenXFoxD (2011b) Sparse distance learning for object recognition combining rgb and depth information. In IEEE International Conference on Robotics and Automation, Shanghai, China, 9–13 May 2011.
30.
LalondeJFVandapelNHuberDFHebertM (2006) Natural terrain classification using three-dimensional ladar data for ground robot mobility. Journal of Field Robotics23: 839–861.
31.
LaporteCArbelT (2006) Efficient discriminant viewpoint selection for active bayesian recognition. International Journal of Computer Vision68: 267–287.
32.
LaporteCBrooksRArbelT (2004) A fast discriminant approach to active object recognition and pose estimation. In International Conference on Pattern Recognition (ICPR), Cambridge, UK, 23–26 August 2004.
33.
LeeDCGuptaAHebertMKanadeT (2010) Estimating spatial layout of rooms using volumetric reasoning about objects and surfaces. In Neural Information Processing Systems (NIPS), Vancouver, Canada, 6–11 December 2010.
34.
LeibeBCornelisNCornelisKGoolLV (2007) Dynamic 3d scene analysis from a moving vehicle. In Computer Vision and Pattern Recognition (CVPR), Minneapolis, USA, 18–23 June 2007.
35.
LiCKowdleASaxenaAChenT (2012) Towards holistic scene understanding: Feedback enabled cascaded classification models. IEEE Transactions on Pattern Analysis and Machine Intelligence34(7): 1394–1408.
36.
LiCSaxenaAChenT (2011) θ -mrf: Capturing spatial and semantic structure in the parameters for scene understanding. In Neural Information Processing Systems (NIPS), Granada, Spain, 12–17 December 2011.
37.
LyDSaxenaALipsonH (2012) Co-evolutionary predictors for kinematic pose inference from rgbd images. In Genetic and Evolutionary Computation Conference (GECCO), Philadelphia, USA, July 7–11 2012.
38.
MaJChungTHBurdickJ (2011) A probabilistic framework for object search with 6-dof pose estimation. International Journal of Robotics Research30(10): 1209–1228.
39.
MegerDGuptaALittleJ (2010) Viewpoint detection models for sequential embodied object category recognition. In IEEE International Conference on Robotics and Automation, Anchorage, USA, 3–8 May2010.
40.
MunozDVandapelNHebertM (2009) Onboard contextual classification of 3-d point clouds with learned high-order markov random fields. In IEEE International Conference on Robotics and Automation, Kobe, Japan, 12–17 May 2009.
41.
MurphyKPTorralbaAFreemanWT (2003) Using the forest to see the trees: a graphical model relating features, objects and scenes. Neural Information Processing Systems (NIPS), Whistler, Canada, 8–13 December 2003.
42.
PerkoRLeonardisA (2010) A framework for visual-context-aware object detection in still images. Computer Vision and Image Understanding114: 700–711.
43.
QuigleyMBatraSGouldSKlingbeilELeQVWellmanANgAY (2009) High-accuracy 3d sensing for mobile manipulation: Improving object detection and door opening. In IEEE International Conference on Robotics and Automation, Kobe, Japan, 12–17 May 2009.
44.
RotherCKolmogorovVLempitskyVSzummerM (2007) Optimizing binary mrfs via extended roof duality. In Computer Vision and Pattern Recognition.
45.
RusuRBMartonZCBlodowNDolhaMBeetzM (2008) Towards 3d point cloud based object maps for household environments. Robot. Auton. Syst. 56: 927–941.
46.
SaxenaAChungSNgA (2005) Learning depth from single monocular images. In Neural Information Processing Systems (NIPS), Whistler, Canada, 5–10 December 2005.
47.
SaxenaAChungSNgA (2008) 3-d depth reconstruction from a single still image. International Journal of Computer Vision76(1): 53–69.
48.
SaxenaASunMNgA. Y (2009) Make3d: Learning 3d scene structure from a single still image. IEEE Transactions on Pattern Analysis and Machine Intelligence31(5): 824–840.
49.
ShapovalovRVelizhevA (2011) Cutting-plane training of non-associative markov network for 3d point cloud segmentation. In IEEE International Conference on 3D Imaging, Modeling, Processing, Visualization and Transmission (3DIMPVT), Zurich, Switzerland, 15–17 October 2012.
50.
ShapovalovRVelizhevABarinovaO (2010) Non-associative markov networks for 3d point cloud classification. In ISPRS Commission III Symposium – PCV2010.
51.
SungJPonceCSelmanBSaxenaA (2012) Unstructured human activity detection from rgbd images. In IEEE International Conference on Robotics and Automation, St. Paul, USA, 14–18 May 2012.
52.
SzeliskiRZabihRScharsteinDVekslerOKolmogorovVAgarwalaATappenMRotherC (2008) A comparative study of energy minimization methods for markov random fields with smoothness-based priors. IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 1068–1080.
53.
TaskarBChatalbashevVKollerD (2004) Learning associative markov networks. In International Conference on Machine Learning (ICML), Banff, Canada, July 4–8, 2004.
54.
TaskarBGuestrinCKollerD (2003) Max-margin markov networks. In Neural Information Processing Systems (NIPS), Whistler, Canada, 8–13 December 2003.
55.
TorralbaA (2003) Contextual priming for object detection. International Journal of Computer Vision53(2): 169–191.
56.
TsochantaridisIHofmannTJoachimsTAltunY (2004) Support vector machine learning for interdependent and structured output spaces. In International Conference on Machine Learning (ICML), Banff, Canada, July 4–8, 2004.
57.
XiaoJQuanL (2009) Multiple view semantic segmentation for street view images. In International Conference on Computer Vision (ICCV), Kyoto, Japan, 29 September–2 October 2009.
58.
XiongXHuberD (2010) Using context to create semantic 3d models of indoor environments. In British Machine Vision Conference (BMVC), Aberystwyth (Wales, UK), 30 August–2 September 2010.
59.
XiongXMunozDBagnellJAHebertM (2011) 3-d scene analysis via sequenced predictions over points and regions. In IEEE International Conference on Robotics and Automation, Shanghai, China, 9–13 May 2011.
