The task of teleoperating a robot over a wireless video link is known to be very difficult. Teleoperation becomes even more difficult when the robot is surrounded by dense obstacles, or speed requirements are high, or video quality is poor, or wireless links are subject to latency. Due to high-quality lidar data, and improvements in computing and video compression, virtualized reality has the capacity to dramatically improve teleoperation performance — even in high-speed situations that were formerly impossible. In this paper, we demonstrate the conversion of dense geometry and appearance data, generated on-the-move by a mobile robot, into a photorealistic rendering model that gives the user a synthetic exterior line-of-sight view of the robot, including the context of its surrounding terrain. This technique converts teleoperation into virtual line-of-sight remote control. The underlying metrically consistent environment model also introduces the capacity to remove latency and enhance video compression. Display quality is sufficiently high that the user experience is similar to a driving video game where the surfaces used are textured with live video.
Anderson, D., Howard, T., Apfelbaum, D., Herman, H. and Kelly, A. ( 2008). Coordinated control and range imaging for mobile manipulation . Proceedings of the International Symposium on Experimental Robotics (ISER).
2.
Anderson, D. and Kelly, A. ( 2005). Experimental characterization of commercial flash lidar devices. Proceedingsof the International Conference on Sensing Technologies, Palmerston North, New Zealand.
3.
Bejczy, A., Kim, W. and Venema, S. ( 1990). The phantom robot: predictive displays for teleoperation with time delay. Proceedings of the IEEE International Conference on Robotics and Automation (ICRA).
4.
Buss, M., Peer, A., Schaub, T., Stefanov, N. and Unterhinninghofen, U. ( 2008). Multi-modal multi-user telepresence and teleaction system . Intelligent Robots and Systems, 4137-4138.
5.
Chatila, R., Lacroix, S., Simion, T. and Herrb, M. ( 1995). Planetary exploration by a mobile robot: mission teleprogramming and autonomous navigation. Autonomous Robots2(4): 333-344.
6.
Chen, S.E. and Williams, L. ( 1993). View interpolation for image synthesis. Proceedings of SIGGRAPH93, pp. 279-288.
7.
Daily, M. et al. (1988). Autonomous cross country navigation with the ALV. Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), pp. 718-726.
8.
Debevec, P., Taylor, C. and Malik, J. ( 1996). Modeling and rendering architecture from photographs . Proceedings of SIGGRAPH, pp. 11-20.
9.
Dima, C., Vandapel, N. and Hebert, M. ( 2004). Classifier fusion for outdoor obstacle detection. Proceedings of the International Conference on Robotics and Automation (ICRA), pp. 665-671.
10.
Edwards, L., Sims, M., Kunz, C., Lees, D. and Bowman, J. ( 2005). Photo-realistic terrain modeling and visualization for mars exploration rover science operations. IEEE Systems, Man, and Cybernetics, Waikoloa, Hawaii .
11.
El-Hakim, S.F., Boulanger, P., Blais, F. and Beraldin, J.A. ( 1997). A system for indoor 3D mapping and virtual environments . Videometrics V, SPIE Volume 3174. pp. 21-35.
12.
Ferland, F., Pomerleau, F., Le Dinh, C. and Michaud, F. ( 2009). Egocentric and exocentric teleoperation interface using real-time, 3D video projection. Proceedings of the 4th ACM/IEEE Inter-national Conference on Human Robot Interaction, pp. 37-44.
13.
Fong, T., Pangels, H. and Wettergreen, D. ( 1995). Operator interfaces and network-based participation for Dante II. Proceedings of the SAE 25th International Conference on Environmental Systems, 11(1): 77-85
14.
Fong, T., Thorpe, C. and Baur, C. ( 2001). Advanced interfaces for vehicle teleoperation: collaborative control, sensor fusion displays, and remote driving tools. Autonomous Robots, 77-85.
15.
Früh, C. and Zakhor, A. ( 2004). An automated method for large-scale, ground-based city model acquisition. International Journal of Computer Vision ,60(1), 5-24.
16.
Funda, J., Lindsay, T. and Paul, R. ( 1992). Teleprogramming: toward delay-invariant remote manipulation . Presence: Teleoperation. Virtual Environonments,1(1), 29-44.
17.
Hart, S.G. and Staveland, L.E. ( 1987). Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research (Vol. Human mental workload ), Meshkati, P. H. (eds). Amsterdam: Elsevier.
18.
Hine, B., Hontalas, P., Fong, T., Piguet, L., Nygren, E. and Kline, A. ( 1995). VEVI: A Virtual Environment Teleoperations Interface for planetary exploration. Proceedings of the SAE 25th International Conference on Environmental Systems, San Diego, CA.
19.
Hine, B., Stocker, C. and Sims, M. ( 1994). The application of telepresence and virtual reality to subsea exploration. Proceedings of the 2nd Workshop on Mobile Robots for Subsea Environments (Proc. ROV’94), Monterey, CA.
20.
Ho, N. and Jarvis, R. ( 2007). Large scale 3D environmental modelling for stereoscopic walk-through visualisation. Proceedings of 3DTV07, pp. 1-4.
21.
Hoiem, A., Efros, A. and Hebert, M. ( 2005). Geometric context from a single image. Proceedings of the International Conference on Computer Vision (ICCV), pp. 654-661.
22.
Howard, T., Green C., and Kelly A.Receding Horizon Model-Predictive Control for Mobile Robot Navigation of Intricate Paths, FSR09.
23.
Hu, S.Y. and Neumann, U. ( 2003). Approaches to large-scale urban modeling. IEEE Computer Graphics and Applications (CG&A),23(6), 62-69.
24.
Huber, D., Herman, H., Kelly, A., Rander, P. and Warner, R. ( 2009). Real-time photorealistic visualization of 3D environments for enhanced teleoperation of vehicles. Proceedings of the 2nd International Conference on 3D Digital Imaging and Modeling, Kyoto, Japan.
25.
Jakel, L., Krotkov, E., Perschbacher, M., Pippine, J. and Sullivan, C. ( 2006). The DARPA LAGR program: goals, challenges, methodology, and phase I results. Journal of Field Robotics, 23, 945-973.
26.
Johnson, J., Alberts, J., Berkemeier, M., and Edwards, J. ( 2008). Manipulator autonomy for EOD robots. Proceedings of the 26th Army Science Conference.
27.
Kanade, T., Rander, P. and Narayanan, P.J. ( 1997). Virtualized reality: constructing virtual worlds from real scenes. IEEE MultiMedia, 4(1): 34-47.
28.
Kelly, A. and Stentz, A. ( 1998). Rough terrain autonomous mobility, part 2: an active vision, predictive control approach. Autonomous Robots, 5, 163-198.
29.
Kelly, A., Stentz, A., Amidi, O., Bode, M., Bradley, D., Diaz-Calderon, A. ( 2006). Toward reliable off-road autonomous vehicles operating in challenging environments. The International Journal of Robotics Research, 25(5-6), 449-483.
30.
Kim, W.S. ( 1993). Advanced teleoperation, graphics aids, and application to time delay environments. Proceedings of the 1st Industrial Virtual Reality Show and Conference (IVR ’93), Makuhari Meese, Japan, pp. 23-25.
31.
Kim, W.S. ( 1996). Virtual reality calibration and preview / predictive displays for telerobotics. Presence: Teleoperators and Virtual Environments, 5(2), 173-190.
32.
Lacase, A., Murphy, K. and DelGiorno, M. ( 2002). Autonomous mobility for the Demo III experimental unmanned vehicles. Proceedings of the Assoc. for Unmanned Vehicle Systems Int. Conf. on Unmanned Vehicles (AUVSI 02).
33.
Lewis, R.A. and Johnston, A.R. ( 1977). A scanning laser rangefinder for a robotic vehicle. Proceedsings of the 5th International Joint Conference on Artificial Intelligence.
34.
Lloyd, J.E., Beis, J.S., Pai, D.K. and Lowe, D.G. ( 1997). Modelbased telerobotics with vision. Proceedings of the International Conference on Robotics and Automation (ICRA) , pp. 1297-1304.
35.
Matusik, W., Beuhler, C., Raskar, R., Gortler, L. and McMillan, L. ( 2000). Image-based visual hulls. Proceedings of SIGGRAPH, pp. 369-374.
36.
McMillan, L. and Bishop, G. ( 1995). Plenoptic modeling: an image-based rendering system. Proceedings of ACM SIGGRAPH, pp. 39-46.
37.
Messuri, D.A. and Klein, C.A. ( 1985). Automatic body regulation for maintaining stability of a legged vehicle during terrain locomotion. IEEE Journal of Robotics and Automation, 1(3): 132-141.
38.
Milgram, P., Yin, S. and Grodski, J. ( 1997). An augmented reality based teleoperation interface for unstructured environments. Proceedings of the American Nuclear Society (ANS) 7th Topical Meeting on Robotics and Remote Systems, pp. 966-973.
39.
Milgram, P., Zhai, S., Drascic, D. and Grodski, J. ( 1993). Applications of augmented reality for human-robot communication . Proceedings of the International Conference on Intelligent Robots and Systems (IROS), Yokohama, Japan, pp. 1467-1472.
40.
Möller, T., Kraft, H., Frey, J., Albrech, M. and Lange, R. ( 2005). Robust 3D measurement with pmd sensors. Proceedings of the First Range Imaging Research Dat at ETH Zurich .
41.
Mordohai, P., Frahm, J.M. and Akbarzadeh, A. ( 2007). Real-time video-based reconstruction of urban environments . Proceedings of the ISPRS Workshop on 3D Virtual Reconstruction and Visualization of Complex Architectures.
42.
Nielsen, C.W., Goodrich, M.A. and Ricks, B. ( 2007). Ecological interfaces for improving mobile robot teleoperation . IEEE Transactions on Robotics and Automation, 23(5), 927-941.
43.
Rander, P. ( 1998). A multi-camera method for 3D digitization of dynamic, real-world events. Ph.D. Thesis, Pittsburgh, PA, Robotics Institute, Carnegie Mellon University.
44.
Ricks, B., Nielsen, C. and Goodrich, M. ( 2004). Ecological displays for robot interaction: a new perspective . IEEE/RSJ Proceedings of the International Conference on Intelligent Robots and Systems (IROS), pp. 2855-2860.
45.
Rohlf, J. and Helman, J. ( 1994). IRIS Performer: a high performance multiprocessing toolkit for real-time 3D graphics. Proceedings of SIGGRAPH, pp. 381-394.
46.
Ross, B., Bares, J., Stager, D., Jacker, L. and Perschbacher, M. ( 2008). An advanced teleoperation testbed. Field and Service Robotics. 42: 397-304.
47.
Ryde, J. and Hu, H. ( 2008). 3D laser range scanner with hemispherical field of view for robot navigation. Proceedings of the 2008 IEEE/ASME International Conference on Advanced Intelligent Mechatronics, Xi’an, China.
48.
Schneider, M. and Klein, R. ( 2008). Enhancing textured digital elevation models using photographs . Proceedings of the Intl. Symp. on 3D Data Processing, Visualization, and Transmission (3DPVT), pp. 261-268.
49.
Se, S. and Jasiobedzki, P. ( 2008). Stereo-vision based 3D modeling and localization for unmanned vehicles. International Journal of Intelligent Control and Systems, 13(1), 47-58.
50.
Segal, M., Korobkin, C., van Widenfelt, R., Foran, J. and Haeberli, P. ( 1992). Fast shadows and lighting effects using texture mapping . Proceedings of SIGGRAPH, pp. 249-252.
51.
Sheridan, T. ( 1986). Human supervisory control of robot systems. Proceedings of the IEEE International Conference on Robotics and Automation.
52.
Sheridan, T. ( 1993). Space teleoperation through time delay: review and prognosis . IEEE Trans. Robotics Automation, 9, 592-606.
53.
Smallman, H. and St. John, M. ( 2005). Naive realism: misplaced faith in realistic displays . Ergonomics in Design, 13: 6-13.
54.
Stamos, I. and Allen, P.K. ( 2002). Geometry and texture recovery of scenes of large scale . Computer Vision and Image Understanding (CVIU), 88(2), 94-118.
55.
Stewart, B., Ko, J. and Konolige, K. ( 2003). The revisiting problem in mobile robot map building: a hierarchical bayesian approach. Proceedings of the Conference on Uncertainty in Artificial Intelligence.
56.
Theunissen, E., Koeners, G., Roefs, F., Rademaker, R., Jinkins, R. and Etherington, T. ( 2005). Guidance, situation awareness and integrity monitoring with an SVS+EVS. Proceedings of the AIAA Guidance, Navigation, and Control Conference and Exhibit, San Francisco, CA.
57.
Wellington, C., Courville, A. and Stentz, A. ( 2006). A generative model of terrain for autonomous navigation in vegetation. Proceedings of the Intl. Journal of Robotics Research (IJRR), 25(1), 1287-1304.
58.
Wellington, C. and Stentz, A. ( 2004). Online adaptive rough-terrain navigation in vegetation . Proceedings of the IEEE Int. Conf. on Robotics and Automation (ICRA).
59.
Westover, L. ( 1990). Footprint evaluation for volume rendering. in computer graphics. Proceedings of SIGGRAPH, pp. 367-376.
60.
Williams, L. ( 1978). Casting curved shadows on curved surfaces. Proceedings of ACM SIGGRAPH, pp. 270-274.
