Object ground lines regression and mapping from fisheye images to around view image for the AVP

Abstract

The automated valet parking (AVP) based on the around view monitor (AVM) has attracted increasing attention from the research community due to its high practicality and low cost. However, the distortion correction and inverse perspective transformation of the around view images result in the truncated and deformed objects, which leads to poor detection performance. To address this problem, we propose a method of object ground lines regression and mapping from fisheye images to around view images. There are two key steps for the proposed method, regressing ground lines in the fisheye images and mapping ground lines to the around view image. To detect the objects and regress ground lines at the same time, we explored three new detectors based on YOLOv5 (i.e. YOLOv5-VGG, YOLOv5-Point, and YOLOv5-Ratio). Among them, YOLOv5-VGG adds a VGG-based regression network behind YOLOv5, YOLOv5-Point directly adds a regression head to share the convolutional features, and YOLOv5-Ratio combines the 2D box and ground line to predict its ratios. After that, the ground lines from four different views are mapped and fused to the around view image. Additionally, to facilitate the study of the object detection in the around view image, a large-scale labeled dataset is established, which comprises 9828 fisheye images collected from typical indoor and outdoor parking lots. For each image, the 2D bounding boxes and ground lines of objects are carefully labeled. Experiments show that the YOLOv5-Ratio obtains the best performance and can effectively detect the truncated and deformed object in the around view image with the precision rate of 92.28% and recall rate of 85.09% on our collected dataset.

Keywords

Automated valet parking object detection deep learning around view image fisheye image

Get full access to this article

View all access options for this article.

References

Schwesinger

Bürki

Timpner

, et al. Automated valet parking and charging for e-mobility. In: 2016 IEEE intelligent vehicles symposium (IV), Gothenburg, Sweden, 19–22 June 2016, pp.157–164. New York, NY: IEEE.

Kato

Suzuki

Fujita

, et al. Image synthesis display method and apparatus for vehicle camera. U.S. Patent US7139412B2, 2006.

Cao

Yan

, et al. Vacant parking slot detection in the around view image based on deep learning. Sensors 2020; 20: 2138.

Deng

Yang

, et al. Semantic segmentation-based lane-level localization using around view monitoring system. IEEE Sens J 2019; 19(21): 10077–10086.

Shao

Zhang

Zhang , et al. A tightly-coupled semantic SLAM system with visual, inertial and surround-view sensors for autonomous indoor parking. In: Proceedings of the 28th ACM international conference on multimedia (MM’20), Seattle, WA, 12–16 October 2020, pp.2691–2699. New York, NY: ACM.

Heimberger

Horgan

Hughes

, et al. Computer vision in automated parking systems: design, implementation and challenges. Image Vis Comput 2017; 68: 88–101.

LeCun

Bengio

Hinton

. Deep learning. Nature 2015; 521(7553): 436–444.

Tong

Tang

, et al. FisheyeDet: a self-study and contour-based object detector in fisheye images. IEEE Access 2020; 8: 71739–71751.

Arsenali

Viswanath

Novosel

. RotInvMTL: rotation invariant multinet on fisheye images for autonomous driving applications. In: IEEE/CVF international conference on computer vision workshop (ICCVW), Seoul, South Korea, 27–28 October 2019, pp.2373–2382. New York, NY: IEEE.

10.

Yahiaoui

Rashed

Mariotti

, et al. FisheyeMODNet: moving object detection on surround-view cameras for autonomous driving, arXiv:1908.11789 [Online], 2019, https://arxiv.org/abs/1908.11789

11.

Krizhevsky

Sutskever

Hinton

. ImageNet classification with deep convolutional neural networks. Commun ACM 2017; 60(6): 84–90.

12.

Deng

Dong

Socher

, et al. ImageNet: a large-scale hierarchical image database. In: IEEE conference on computer vision and pattern recognition, Miami, FL, 20–25 June 2009, pp.248–255. New York, NY: IEEE.

13.

Girshick

Donahue

Darrell

, et al. Rich feature hierarchies for accurate object detection and semantic segmentation. In: 2014 IEEE conference on computer vision and pattern recognition (CVPR), Columbus, OH, 23–28 June 2014, pp.580–587. New York, NY: IEEE.

14.

Zhang

Ren

, et al. Spatial pyramid pooling in deep convolutional networks for visual recognition. IEEE Trans Pattern Anal Mach Intell 2015; 37(9): 1904–1916.

15.

Girshick

. Fast R-CNN. In: Proceedings of the 2015 IEEE international conference on computer vision (ICCV), Santiago, Chile, 7–13 December 2015, pp.1440–1448. New York, NY: IEEE.

16.

Ren

Girshick

, et al. Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Trans Pattern Anal Mach Intell 2017; 39(6): 1137–1149.

17.

Liu

Anguelov

Erhan

, et al. SSD: single shot multibox detector. In: European conference on computer vision (ECCV), Amsterdam, The Netherlands, 2016, pp.21–37. Springer: Cham.

18.

Lin

Goyal

Girshick

, et al. Focal loss for dense object detection. IEEE Trans Pattern Anal Mach Intell 2020; 42(2): 318–327.

19.

Redmon

Divvala

Girshick

, et al. You only look once: unified, real-time object detection. In: IEEE conference on computer vision and pattern recognition (CVPR), Las Vegas, NV, 27–30 June 2016, pp.779–788. New York, NY: IEEE.

20.

Redmon

Farhadi

. YOLO9000: better, faster, stronger. In: IEEE conference on computer vision and pattern recognition (CVPR), Honolulu, HI, 21–26 July 2017, pp.6517–6525. New York, NY: IEEE.

21.

Redmon

Farhadi

. YOLOv3: an incremental improvement, arXiv:1804.02767. [Online], 2018, https://arxiv.org/abs/1804.02767

22.

Bochkovskiy

Wang

Liao

. YOLOv4: optimal speed and accuracy of object detection, arXiv:2004.10934v1 [Online], 2020, https://arxiv.org/abs/2004.10934v1

23.

Jocher

Nishimura

Mineeva

, et al. YOLOv5 [Online], 2022, https://github.com/ultralytics/yolov5

24.

Chiang

Wang

Chen

. Efficient pedestrian detection in top-view fisheye images using compositions of perspective view patches. Image Vis Comput 2021; 105: 104069.

25.

Rashed

Mohamed

Sistu

, et al. Generalized object detection on fisheye cameras for autonomous driving: dataset, representations and baseline. In: IEEE winter conference on applications of computer vision (WACV), Waikoloa, HI, 3–8 January 2021, pp.2271–2279. New York, NY: IEEE.

26.

Zhu

Wan

, et al. Object detection and localization in 3D environment by fusing raw fisheye image and attitude data. J Vis Commun Image Represent 2019; 59: 128–139.

27.

Kim

Park

S-Y

. Expandable spherical projection and feature fusion methods for object detection from fisheye images. In: 17th international conference on machine vision and applications (MVA), Aichi, Japan, 25–27 July 2021, pp.1–5. New York, NY: IEEE.

28.

Coors

Condurache

Geiger

. SphereNet: learning spherical representations for detection and classification in omnidirectional images. In: 15th European conference on computer vision (ECCV), Munich, Germany, 8–14 September 2018, pp.525–541. Cham: Springer.

29.

Kannala

Brandt

. A generic camera model and calibration method for conventional, wide-angle, and fish-eye lenses. IEEE Trans Pattern Anal Mach Intell 2006; 28(8): 1335–1340.

30.

Rufli

Scaramuzza

Siegwart

. Automatic detection of checkerboards on blurred and distorted images. In: IEEE/RSJ international conference on intelligent robots and systems, Nice, France, 22–26 September 2008, pp.3121–3126. New York, NY: IEEE.

31.

Feng

Wei

, et al. Calibration and stitching methods of around view monitor system of articulated multi-carriage road vehicle for intelligent transportation. SAE technical paper 2019-01-0873, 2019.

32.

Yogamani

Hughes

Horgan

, et al. WoodScape: a multi-task, multi-camera fisheye dataset for autonomous driving. In: IEEE/CVF international conference on computer vision (ICCV), Seoul, South Korea, 27 October–2 November 2019, pp.9307–9317. New York, NY: IEEE.

33.

Everingham

Eslami

SMA

Van Gool

, et al. The pascal visual object classes challenge: a retrospective. Int J Comput Vis 2015; 111(1): 98–136.