Depth sensors have been used extensively for perception in robotics. Typically these sensors have a fixed angular resolution and field of view (FOV). This is in contrast to human perception, which involves foveating: scanning with the eyes’ highest angular resolution over regions of interest (ROIs). We build a scanning depth sensor that can control its angular resolution over the FOV. This opens up new directions for robotics research, because many algorithms in localization, mapping, exploration, and manipulation make implicit assumptions about the fixed resolution of a depth sensor, impacting latency, energy efficiency, and accuracy. Our algorithms increase resolution in ROIs either through deconvolutions or intelligent sample distribution across the FOV. The areas of high resolution in the sensor FOV act as artificial fovea and we adaptively vary the fovea locations to maximize a well-known information theoretic measure. We demonstrate novel applications such as adaptive time-of-flight (TOF) sensing, LiDAR zoom, gradient-based LiDAR sensing, and energy-efficient LiDAR scanning. As a proof of concept, we mount the sensor on a ground robot platform, showing how to reduce robot motion to obtain a desired scanning resolution. We also present a ROS wrapper for active simulation for our novel sensor in Gazebo. Finally, we provide extensive empirical analysis of all our algorithms, demonstrating trade-offs between time, resolution and stand-off distance.
CampbellATIIIFussellDS (1990) Adaptive mesh generation for global diffuse illumination. ACM SIGGRAPH Computer Graphics24: 155–164.
9.
CarloneLKaramanS (2017) Attention and anticipation in fast visual–inertial navigation. In: 2017 IEEE International Conference on Robotics and Automation (ICRA). IEEE, pp. 3886–3893.
10.
CharrowBKahnGPatilS, et al. (2015) Information-theoretic planning with trajectory optimization for dense 3d mapping. In: Robotics: Science and Systems.
11.
Del BimboADiniFLisantiGPerniciF (2010) Exploiting distinctive visual landmark maps in pan–tilt–zoom camera networks. Computer Vision and Image Understanding114(6): 611–623.
12.
DenzlerJBrownCM (2002) Information theoretic sensor data selection for active object recognition and state estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence24(2): 145–157.
13.
DeselaersTDreuwPNeyH (2008) Pan, zoom, scan–time-coherent, trained automatic video cropping. In: IEEE Conference on Computer Vision and Pattern Recognition, 2008 (CVPR 2008). IEEE, pp. 1–8.
14.
Durrant-WhyteHBaileyT (2006) Simultaneous localization and mapping: part I. IEEE Robotics and Automation Magazine13(2): 99–110.
15.
FergusRTorralbaAFreemanWT (2006) Random lens imaging. MIT CSAIL Technical Report2006-058.
16.
FlatleyTP (2015) Spacecube: A family of reconfigurable hybrid on-board science data processors. In: NASA/ESA Conference on Adaptive Hardware and Systems (AHS-2012).
17.
GoodmanJW (2005) Introduction to Fourier Optics. Roberts and Company Publishers.
18.
GschwandtnerMKwittRUhlAPreeW (2011) BlenSor: Blender sensor simulation toolbox. In: International Symposium on Visual Computing. Berlin: Springer, pp. 199–208.
19.
GuptaMNayarSKHullinMBMartinJ (2015) Phasor imaging: A generalization of correlation-based time-of-flight imaging. ACM Transactions on Graphics34(5): 156.
20.
HaltermanRBruchM (2010) Velodyne HDL-64e lidar for unmanned surface vehicle obstacle detection. Technical Report, Space and Naval Warfare Systems Center, San Diego, CA.
21.
HarkerMO’LearyP (2015) Regularized reconstruction of a surface from its measured gradient field. Journal of Mathematical Imaging and Vision51(1): 46–70.
22.
HeideFHullinMBGregsonJHeidrichW (2013) Low-budget transient imaging using photonic mixer devices. ACM Transactions on Graphics32(4): 45.
23.
HollingerGASukhatmeGS (2014) Sampling-based robotic information gathering algorithms. The International Journal of Robotics Research33(9): 1271–1287.
24.
IzadiSKimDHilligesO, et al. (2011) KinectFusion: Real-time 3D reconstruction and interaction using a moving depth camera. In: Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology. New York: ACM Press, pp. 559–568.
25.
JonesAMcDowallIYamadaHBolasMDebevecP (2007) Rendering for an interactive 360 degree light field display. In: SIGGRAPH. New York: ACM Press.
26.
JulianBJKaramanSRusD (2014) On mutual information-based control of range sensing robots for mapping applications. The International Journal of Robotics Research33(10): 1375–1392.
27.
KasturiAMilanovicVAtwoodBHYangJ (2016) UAV-borne LiDAR with MEMS mirror-based scanning capability. Proceedings of SPIE9832: 98320M.
28.
KoenigNHowardA (2004) Design and use paradigms for Gazebo, an open-source multi-robot simulator. In: 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Vol. 3, pp. 2149–2154
29.
KoppalSJGkioulekasIYoungT (2013) Toward wide-angle microvision sensors. IEEE Transactions on Pattern Analysis and Machine Intelligence12: 2982–2996.
30.
KoppalSJYamazakiSNarasimhanSG (2012) Exploiting DLP illumination dithering for reconstruction and photography of high-speed scenes. International Journal of Computer Vision96(1): 125–144.
31.
KrastevKTVan LieropHWSoemersHMSandersRHMNellissenAJM (2013) Mems scanning micromirror. US Patent 8,526,089.
32.
LevinAFergusRDurandFFreemanWT (2007) Image and depth from a conventional camera with a coded aperture. ACM Transactions on Graphics26: 70.
33.
MilanovićVKasturiASiuNRadojičićMSuY (2011) “MEMSeye” for optical 3D tracking and imaging applications. In: 2011 16th International Solid-State Sensors, Actuators and Microsystems Conference (TRANSDUCERS). IEEE, pp. 1895–1898.
34.
MilanovićVKasturiAYangJHuF (2017) A fast single-pixel laser imager for VR/AR headset tracking. Proceedings of SPIE10116: 101160E–1.
35.
MuBGiamouMPaullLAgha-mohammadiALeonardJHowJ (2016) Information-based active SLAM via topological feature graphs. In: 2016 IEEE 55th Conference on Decision and Control (CDC), pp. 5583–5590.
36.
MurrayDJenningsC (1997) Stereo vision based mapping and navigation for mobile robots. In: Proceedings of International Conference on Robotics and Automation, Vol. 2. IEEE, pp. 1694–1699.
37.
NayarSKBranzoiVBoultTE (2006) Programmable imaging: Towards a flexible camera. International Journal of Computer Vision70(1): 7–22.
38.
NehabDSanderPVIsidoroJR (2006) The real-time reprojection cache. In: ACM SIGGRAPH 2006 Sketches. New York: ACM Press, p. 185.
39.
NgR (2005) Fourier slice photography. ACM Transactions on Graphics24: 735–744.
40.
O’TooleMAcharSNarasimhanSGKutulakosKN (2015) Homogeneous codes for energy-efficient illumination and imaging. ACM Transactions on Graphics34(4): 35.
41.
O’TooleMHeideFXiaoLHullinMBHeidrichWKutulakosKN (2014) Temporal frequency probing for 5D transient analysis of global light transport. ACM Transactions on Graphics33(4): 87.
42.
PalettaLPrantlMPinzA (2000) Learning temporal context in active object recognition using Bayesian analysis. In: Proceedings 15th International Conference on Pattern Recognition (ICPR-2000), Vol. 1. IEEE, pp. 695–699.
43.
PérezPGangnetMBlakeA (2003) Poisson image editing. ACM Transactions on Graphics22(3): 313–318.
44.
PetersenKE (1980) Silicon torsional scanning mirror. IBM Journal of Research and Development24(5): 631–637.
45.
PoultonCVYaacobiAColeDB, et al. (2017) Coherent solid-state LiDAR with silicon photonic optical phased arrays. Optics Letters42(20): 4091–4094.
46.
PrincipeJCEulianoNRLefebvreWC (2000) Neural and Adaptive Systems: Fundamentals Through Simulations, Vol. 672. New York: John Wiley & Sons, Inc.
47.
QiCRSuHMoKGuibasLJ (2017) PointNet: Deep learning on point sets for 3D classification and segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 652–660.
48.
RaskarRAgrawalATumblinJ (2006) Coded exposure photography: Motion deblurring using fluttered shutter. ACM Transactions on Graphics25(3): 795–804.
49.
RaskarRWelchGCuttsMLakeAStesinLFuchsH (1998) The office of the future: A unified approach to image-based modeling and spatially immersive displays. In: Proceedings of the 25th Annual Conference on Computer Graphics and Interactive Techniques. New York: ACM Press, pp. 179–188.
50.
RosmanGRusDFisherJW (2016) Information-driven adaptive structured-light scanners. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 874–883.
51.
SandnerTBauligCGrasshoffT, et al. (2015) Hybrid assembled micro scanner array with large aperture and their system integration for a 3D TOF laser camera. In: MOEMS and Miniaturized Systems XIV ( Proc. SPIE, Vol. 9375). International Society for Optics and Photonics, p. 937505.
52.
ScharsteinDSzeliskiR (2003) High-accuracy stereo depth maps using structured light. In: Proceedings 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Vol. 1. IEEE.
53.
StannBLDammannJFDel GiornoM, et al. (2014) Integration and demonstration of MEMS–scanned LiDAR for robotic navigation. Proceedings of SPIE9084: 90840J.
54.
TasneemZWangDXieHKoppalSJ (2018) Directionally controlled time-of-flight ranging for mobile sensing platforms. In: Robotics: Science and Systems XIV.
55.
TerzopoulosDVasilescuM (1991) Sampling and reconstruction with adaptive meshes. In: Proceedings IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 1991 (CVPR’91). IEEE, pp. 70–75.
56.
ThrunSBurgardWFoxD (2005) Probabilistic Robotics. Cambridge, MA: MIT Press.
VeltenAWuDMasiaB, et al. (2016) Imaging the propagation of light through scenes at picosecond resolution. Communications of the ACM59(9): 79–86.
59.
WillPMPenningtonKS (1972) Grid coding: A novel technique for image processing. Proceedings of the IEEE60(6): 669–680.
60.
YangBZhouLZhangXKoppalSXieH (2017) A compact MEMS-based wide-angle optical scanner. In: 2017 International Conference on Optical MEMS and Nanophotonics (OMN). IEEE, pp. 1–2.
61.
YangHXiLSamuelsonSXieHYangLJiangH (2013) Handheld miniature probe integrating diffuse optical tomography with photoacoustic imaging through a MEMS scanning mirror. Biomedical Optics Express4(3): 427–432.
62.
ZabataniASurazhskyVSperlingE, et al. (2019) Intel RealSenseâ„¢ SR300 coded light depth camera. IEEE Transactions on Pattern Analysis and Machine Intelligence. DOI: 10.1109/TPAMI.2019.2915841.
63.
ZhangNLiMHongB (2006) Active mobile robot simultaneous localization and mapping. In: 2006 IEEE International Conference on Robotics and Biomimetics, pp. 1676–1681.
64.
ZhangYXiongZCongPWuF (2014) Robust depth sensing with adaptive structured light illumination. Journal of Visual Communication and Image Representation25(4): 649–658.
