Transparent objects are encountered frequently in our daily lives, yet recognizing them poses challenges for conventional vision sensors due to their unique material properties, not being well perceived from RGB or depth cameras. Overcoming this limitation, thermal infrared cameras have emerged as a solution, offering improved visibility and shape information for transparent objects. In this paper, we present TRansPose, the first large-scale multispectral dataset that combines stereo RGB-D, thermal infrared (TIR) images, and object poses to promote transparent object research. The dataset includes 99 transparent objects, encompassing 43 household items, 27 recyclable trashes, 29 chemical laboratory equivalents, and 12 non-transparent objects. It comprises a vast collection of 333,819 images and 4,000,056 annotations, providing instance-level segmentation masks, ground-truth poses, and completed depth information. The data was acquired using an FLIR A65 thermal infrared camera, two Intel RealSense L515 RGB-D cameras, and a Franka Emika Panda robot manipulator. Spanning 87 sequences, TRansPose covers various challenging real-life scenarios, including objects filled with water, diverse lighting conditions, heavy clutter, non-transparent or translucent containers, objects in plastic bags, and multi-stacked objects. Supplementary material can be accessed from the following link: https://sites.google.com/view/transpose-dataset.
BashkirovaDAbdelfattahMZhuZ, et al. (2022) Zerowaste dataset: towards deformable object segmentation in cluttered scenes. Proc. IEEE Conf. On Comput. Vision and Pattern Recog, Silver Spring, MD, USA, 21–23 June 2022.
2.
CalliBSinghABruceJ, et al. (2017) Yale-CMU-Berkeley dataset for robotic manipulation research. The International Journal of Robotics Research36(3): 261–268.
3.
ChenGHanKKwan-Yee KennethW, et al (2018) Tom-net: learning transparent object matting from a single image. Proc. IEEE Conf. On Comput. Vision and Pattern Recog,Salt Lake City, UT, USA, 18–23June 2018.
4.
ChenXZhangHYuZ, et al. (2022) Clearpose: large-scale transparent object dataset and benchmark. Proc. European Conf. On Comput. Vision, Tel Aviv, Israel, 23–27 October 2022.
5.
DaiQZhangJLiQ, et al. (2022) Domain randomization-enhanced depth simulation and restoration for perceiving and grasping specular and transparent objects. Proc. European Conf. On Comput. Vision, Tel Aviv, Israel, 23–27 October 2022.
6.
FangHFangHSXuS, et al. (2022) Transcg: a large-scale real-world dataset for transparent object depth completion and a grasping baseline. IEEE Robotics and Automation Letters7(3): 7383–7390.
7.
HuoDWangJQianY, et al. (2023) Glass segmentation with RGB-thermal image pairs. IEEE Transactions on Image Processing32: 1911–1926.
8.
IchnowskiJAvigalYKerrJ, et al. (2022) Dex-nerf: using a neural radiance field to grasp transparent objects. Proc. Conf. On Robot Learning, Auckland, New Zealand, 14–18 December 2022
9.
JiangJCaoGDoTT, et al. (2022a) A4t: hierarchical affordance detection for transparent objects depth reconstruction and manipulation. IEEE Robotics and Automation Letters7(4): 9826–9833.
10.
JiangJCaoGButterworthA, et al. (2022c) Where shall I touch? Vision-guided tactile poking for transparent object grasping. IEEE28(1): 233–244.
11.
JiangJShan Ll (2022b) Robotic perception of object properties using tactile sensing. Tactile Sensing, Skill Learning, and Robotic Dexterous Manipulation. Amsterdam: Elsevier.
12.
KalraATaamazyanVRaoSK, et al. (2020) Deep polarization cues for transparent object segmentation. Proc. IEEE Conf. On Comput. Vision and Pattern Recog,Seattle, WA, USA, 13–19June2020.
13.
KasperAXueZDillmannR (2012) The KIT object models database: an object model database for object recognition, localization and manipulation in service robotics. The International Journal of Robotics Research31(8): 927–934.
14.
LiWHKleemanL (2011) Segmentation and modeling of visually symmetric objects by robot actions. The International Journal of Robotics Research30(9): 1124–1142.
15.
LinJHeZLauRWH, et al. (2021) Rich context aggregation with reflection prior for glass surface detection. Proc. IEEE Conf. On Comput. Vision and Pattern Recog,Nashville, TN, USA, 20–25June 2021.
16.
LiuXJonschkowskiRAngelovaA, et al. (2020) Keypose: multi-view 3d labeling and keypoint estimation for transparent objects. Proc. IEEE Conf. On Comput. Vision and Pattern Recog,Seattle, WA, USA, 13–19June2020.
17.
LiuXIwaseSKitaniKM, et al. (2021) Stereobj-1m: large-scale stereo image dataset for 6D object pose estimation. Proc. IEEE Intl. Conf. On Comput. Vision, Montreal, Canada, 11–17October2021.
18.
MeiHYangXWangY, et al. (2020) Don’t hit me! glass detection in real-world scenes. Proc. IEEE Conf. On Comput. Vision and Pattern Recog,Seattle, WA, USA, 13–19June2020.
19.
MeiHDongBDongW, et al. (2022) Glass segmentation using intensity and spectral polarization cues. Proc. IEEE Conf. On Comput. Vision and Pattern Recog, New Orleans, LA, USA, 18–24 June 2022.
20.
NovkovicTFurrerFPanjekM, et al. (2019) CLUBS: an RGB-D dataset with cluttered box scenes containing household objects. The International Journal of Robotics Research38(14): 1538–1548.
21.
ProençaPFSimoesP (2020) Taco: Trash Annotations in Context for Litter Detection. arXiv preprint arXiv:2003.06975.
22.
SajjanSMooreMPanM, et al. (2020) Clear grasp: 3d shape estimation of transparent objects for manipulation. Proc. IEEE Intl. Conf. On Robot. andAutomat, Philadelphia, PA, USA, 23–27May 2022.
23.
SaponaroPSorensenSRheinS, et al. (2015) Improving calibration of thermal stereo cameras using heated calibration boardProc. Intl. Conf. On Image Processing,Quebec City, Quebec, Canada, 27–30September 2015.
24.
SaxenaADriemeyerJNgAY (2008) Robotic grasping of novel objects using vision. The International Journal of Robotics Research27(2): 157–173.
25.
SinapovJBergquistTSchenckC, et al. (2011) Interactive object recognition using proprioceptive and auditory feedback. The International Journal of Robotics Research30(10): 1250–1262.
26.
TsaiRYLenzR (1989) A new technique for fully autonomous and efficient 3 d robotics hand/eye calibration. IEEE Transactions on Robotics and Automation5(3): 345–358.
27.
WangPJungHJLiY, et al. (2022) Phocal: a multi-modal dataset for category-level object pose estimation with photometrically challenging objects. Proc. IEEE Conf. On Comput. Vision and Pattern Recog,New Orleans, LA, USA, 18–24June2022.
28.
XieEWangWWangW, et al. (2020) Segmenting transparent objects in the wild. Proc. European Conf. On Comput. Vision,Glasgow, UK, 23–28August 2020.
29.
XuYNagaharaHShimadaA, et al. (2015) Transcut: transparent object segmentation from a light-field image. Proc. IEEE Intl. Conf. On. Santiago, Chile: Vision, Comput., 7–13. (accessed December 2015).
30.
XuHWangYREppelS, et al. (2022) Seeing glass: joint point-cloud and depth completion for transparent objects. Proc. Conf. On RobotLearning, London, UK, 8–11November 2022.
31.
ZengASongSYuKT, et al. (2022) Robotic pick-and-place of novel objects in clutter with multi-affordance grasping and cross-domain image matching. The International Journal of Robotics Research41(7): 690–705.
32.
ZhangZ (1999) Flexible camera calibration by viewing a plane from unknown orientations. Proceedings of IEEE International Conference on Computer Vision1: 666–673.
33.
ZhouZChenXJenkinsOC (2020) Lit: light-field inference of transparency for refractive object localization. IEEE Robot. and Automat. Lett5(3): 4548–4555.
