A multi-sensor fusion and object tracking algorithm for self-driving vehicles

Abstract

Vehicles need to detect threats on the road, anticipate emerging dangerous driving situations and take proactive actions for collision avoidance. Therefore, the study on methods of target detection and recognition are of practical value to a self-driving system. However, single sensor has its weakness, such as poor weather adaptability with lidar and camera. In this article, we propose a novel spatial calibration method based on multi-sensor systems, and the approach utilizes rotation and translation of the coordinate system. The validity of the proposed spatial calibration method is tested through comparisons with the data calibrated. In addition, a multi-sensor fusion and object tracking algorithm based on target level to detect and recognize targets is tested. Sensors contain lidar, radar and camera. The multi-sensor fusion and object tracking algorithm takes advantages of various sensors such as target location from lidar, target velocity from radar and target type from camera. Besides, multi-sensor fusion and object tracking algorithm can achieve information redundancy and increase environmental adaptability. Compared with the results of single sensor, this new approach is verified to have the accuracy of location, velocity and recognition by real data.

Keywords

Self-driving vehicles spatial calibration multi-sensor fusion object tracking

Get full access to this article

View all access options for this article.

References

Kelly

Stentz

Amidi

, et al. Toward reliable off road autonomous vehicles operating in challenging environments. Int J Robot Res 2006; 25: 449–483.

Darms

Rybski

Baker

, et al. Obstacle detection tracking for the urban challenge. In: Proceedings of the 12th IEEE conference on intelligent transportation systems, St. Louis, MO, 3–7 October 2009. New York: IEEE.

Petrovskaya

Thrun

. Model based vehicle detection and tracking for autonomous urban driving. Autonom Robot 2009; 26: 123–139.

Yang

Jiang

Zhao

, et al. Intelligent connected vehicles: current status future perspectives. Sci China 2018; 61: 20–45.

Wang

Mirchandani

Wang

. The VISTA project and its applications. IEEE Intell Transp Syst 2002; 17: 72–75.

Wang

, et al. Creating a digital-vehicle proving ground. IEEE Intell Transp Syst 2003; 18: 12–15.

Song

Wang

, et al. IVS 05: new developments and research trends for intelligent vehicles. IEEE Intell Transp Syst 2005; 20: 10–14.

Yim

. Three-feature based automatic lane detection algorithm (TFALDA) for autonomous driving. IEEE Trans Intell Transp Syst 2003; 4: 219–225.

Jeong

Kim

Yoon

, et al. Real-time lane detection for autonomous vehicle. In: Proceedings of the IEEE international symposium on industrial electronics, vol. 3, Pusan, South Korea, 12–16 June 2001, pp.1466–1471. New York: IEEE.

10.

Huang

Chen

C-J

Hsiao

P-Y

, et al. On-board vision system for lane recognition and front-vehicle detection to enhance driver’s awareness. In: Proceedings of the IEEE international conference on robotics and automation, vol. 3, New Orleans, LA, 26 April–1 May 2004, pp.2456–2461. New York: IEEE.

11.

McCall

Trivedi

. An integrated, robust approach to lane marking detection and lane tracking. In: Proceedings of the IEEE intelligent vehicles symposium, Parma, 14–17 June 2004, pp.533–537. New York: IEEE.

12.

Chen

Huang

Bai

, et al. Multi-Sensor information fusion algorithm with central level architecture for intelligent vehicle environmental perception system. SAE technical paper 2016-01-1894, 2016.

13.

Zhao

Xiao

Yuan

, et al. Fusion of 3D-LIDAR and camera data for scene parsing. J Vis Commun Imag Rep 2014; 25: 165–183.

14.

Aufrere

Gowdy

Mertz

, et al. Perception for collision avoidance and autonomous driving. Mechatronics 2003; 13: 1149–1161.

15.

Monteiro

Premebida

Peixoto

, et al. Tracking and classification of dynamic obstacles using laser range finder and vision, 2006, https://www.academia.edu/18267582/Tracking_and_Classification_of_Dynamic_Obstacles_Using_Laser_Range_Finder_and_Vision

16.

Miller

Campbell

Fujishima

. Team cornell’s skynet: robust perception and planning in an urban environment. J Field Robot 2008; 25: 493–527.

17.

Leonard

How

Teller

, et al. A perception-driven autonomous urban vehicle. J Field Robot 2008; 25: 727–774.

18.

Mahlisch

Schweiger

Ritter

, et al. Sensorfusion using spatio-temporal aligned video and lidar for improved vehicle detection. In: Proceedings 2006 IEEE intelligent vehicles symposium, Tokyo, Japan, 13–15 June 2006. New York: IEEE.

19.

Montemerlo

Becker

Bhat

, et al. Junior: the stanford entry in the urban challenge. J Field Robot 2008; 25: 569–597.

20.

Premebida

Ludwig

Nunes

. Lidar and vision-based pedestrian detection system. J Field Robot 2009; 26: 696–711.

21.

Spinello

Siegwart

. Human detection using multimodal and multidimentional features. In: Proceedings of the 2008 IEEE international conference on robotics and automation, Pasadena, CA, 19–23 May 2008. New York: IEEE.

22.

Spinello

Siegwart

. A Multi-sensor fusion system for moving object detection and tracking in urban driving environments. In: Proceedings of the 2014 IEEE international conference on robotics & automation (ICRA), Hong Kong, China, 31 May–7 June 2014. New York: IEEE.