Nighttime vehicle detection based on direction attention network and bayes corner localization

Abstract

Detecting vehicle at night is critical to both assistant driving systems and autonomous driving systems. In this paper, we propose a deep network scheme assisted by light information with good generalization to detect vehicle at night. Our approach is divided into two branches, the object stream and the pixel stream. The object stream generates a batch of bounding boxes, and the pixel stream utilizes the vehicle light information to calibrate the bounding boxes of the object stream. In the object stream, we propose a new structure, Direction Attention Pooling (DAP), to improve the accuracy of the prior boxes. DAP leads into attention mechanism. The feature maps obtained from backbone network is divided into two branches. One branch obtains direction perception information through IRNN layer, and the other branch learns attention weights. The weights are multiplied with the direction perception features in an element-wise manner. In the pixel stream, we propose a corner localization algorithm based on Bayes to get more accurate corners with the vehicle light pixels. The locations of the corners are considered as a discrete random variable. When the mask of the object is known, solving the probability distribution of the corner of the object is the next step. The corners with the highest probability is the correct corner. On the nighttime vehicle detection datasets CHUK and SYSU, our method achieves the accuracy of 97.2% and 96.86%, which outperforms other state-of-the-art methods by at least 0.31% and 0.34%.

Keywords

Nighttime vehicle detection advanced driver-assistance systems attention mechanism deep learning

1 Introduction

Intelligent Transport System (ITS) aims to achieve optimum traffic efficiency by avoiding traffic problems to the extent possible. The automated system fueled by Advanced Driver-Assistance Systems(ADAS) is proven to be able to reduce road fatalities, by minimizing the human error [1]. Although the traffic volume is lower at night, American National Highway Transportation Safety Administration (NHTSA) statistics show that, 52% of fatal crashes occur after dark [2]. Further more, several studies supported the conclusion that collisions generally are more severe at nighttime than during the day [3, 4]. From the above, nighttime vehicle detection is a major area of interest within the field of intelligent transportation systems (ITS). In this paper, we introduce a new deep network scheme, which aims to improve the accuracy of vehicle detection at night.

There are a number of mature image feature extraction approaches, in daytime scenes, the earlier method is Scale-invariant feature transform (SIFT) [61]. These features have scale invariance and rotation invariance, especially when the illumination and 3D camera’s angle of view change. They are suitable for different images or scenes. The following speed up robust features algorithm(SURF) [62] simplifies the Gaussian second-order differential in SIFT, so that the convolution smoothing operation only needs to be converted into addition and subtraction operation, so that surf algorithm has good robustness and low time complexity. Other subsequent algorithms are also widely used, such as histogram of oriented gradients (HOG) [5], local binary patterns (LBP) [6], and deformable parts model (DPM) [7]. However, nighttime images have low contrast between background and object, which increases the difficulty of object detection. In addition, the low luminosity leads some features of vehicles (e.g., edge, color, and shape features of vehicles) to become obscure. Without enhancement applicable to nighttime images, the traditional image feature extraction methods designed with daytime images do not perform well in nighttime scenes [8 –10]. Therefore, in order to achieve good performance, additional auxiliary information should be added during training.

Daytime object detection methods rely heavily on the color information, gradient features, and texture features of objects. However, at night, the above mentioned visual features are weakened and most conspicuous areas are the bright areas, mainly car light areas. Therefore, it is necessary to assist vehicle detection at night with vehicle light saliency information. At present, some night vehicle detection methods are based on light detection [11 –13]. However, this kind of method may leave out the vehicle lights or capture street lights and traffic lights in complex scenes, which demonstrates the difficulty of accurate vehicle light detection. To solve this problem, Chen et al. [10] proposed a strategy to distinguish vehicle lights and street lights. Specifically, they utilize the Nakagami [14] image to locate vehicle light areas and RPN (RegionProposal Network) based on CNN (Convolutional Neural Networks) feature maps to obtain the vehicle object proposals. Then, the two results are combined to generate the RoIs (Region of Interesting). We observe the fusion segmentation image of Nakagemi [49] image and HSI (Hue-Saturation-Intensity) image, and find that the vehicle light areas are very significant. In our work, the vehicle light information is very important. Then, we utilize the fusion segmentation image of Nakagemi [49] image and HSI image as instrument to obtain vehicle light information to assist nighttime vehicle detection. In the image dehazing task, there has been a similar idea. One image processing result feature is used to finetune another. Singh et al. [67] proposed a gradient profile prior to remove the haze from remote sensing images. The coarse estimated atmospheric veil has been refined by using gradient-based guided image filter. They combined transmission map estimation and atmosphere light estimation to get final image. In the follow-up work, they refined the transmission map by developing a local activity-tuned anisotropic diffusion based filter [68] and using a guided L₀ filter [69].

Nighttime vehicle detectors based on nighttime image enhancement methods and SVM (Support Vector Machine) classifiers have achieved state-of-the-art results [8, 15]. For SYSU nighttime vehicle detection dataset, the accuracy of Combining Extraction method [8] achieved is 94.71%, which is 0.33% higher than the second best method. For CHUK nighttime vehicle detection dataset, the accuracy of Tensor Decomposition method [15] achieved is 95.48%, which is 1.41% higher than the second best method. The above methods are traditional computer recognition methods and they have the following limitations. First of all, there must be a process of feature extraction and expression, and then the features are put into the learning algorithm to train the classifier. Manual feature design requires a lot of experience, and the quality of feature engineering will affect the quality of classifier learning. Secondly, we should have a reasonable classifier. It is very difficult to fulfill the above two requirements to achieve the best effect. Image feature extraction based on CNN structure can avoid the design of complex manual features. The experimental results of Faster R-CNN [16], YOLO [17] and SSD [18], etc. frameworks show that CNN can achieve good performance in the field of object detection. Table 1 shows a attribute comparison with the CNN methods. However, in the night vehicle detection task, only using CNN feature extraction combined with SVM classifier can not get better results than manual feature extraction [8]. In this paper, we propose a new deep network, which is suitable for night vehicle detection tasks and improves the accuracy of night vehicle detection.

Table 1
Attribute comparison with the CNN methods

Method Stage Anchor Attention

Faster-RCNN [32] two ✓ ×

D-FCN [56] two × ×

Mask R-CNN [57] two ✓ ×

Casecade-RCNN [33] two ✓ ×

CBNet [73] two ✓ ×

SSD513 [18] one ✓ ×

YOLOV3 [35] one ✓ ×

DETR [74] one ✓ ✓

EfficientDet [36] one ✓ ×

FCOS [39] one × ×

FreeAnchor [75] one × ×

CornerNet [19] one × ×

Cornernet-Lite [58] one × ✓

CenterNet [37] one × ×

Method	Stage	Anchor	Attention
Faster-RCNN [32]	two	✓	×
D-FCN [56]	two	×	×
Mask R-CNN [57]	two	✓	×
Casecade-RCNN [33]	two	✓	×
CBNet [73]	two	✓	×
SSD513 [18]	one	✓	×
YOLOV3 [35]	one	✓	×
DETR [74]	one	✓	✓
EfficientDet [36]	one	✓	×
FCOS [39]	one	×	×
FreeAnchor [75]	one	×	×
CornerNet [19]	one	×	×
Cornernet-Lite [58]	one	×	✓
CenterNet [37]	one	×	×

CornerNet [19] is a classic object detection network that can achieve a balance between speed and accuracy, and does not need to set a large number of anchor hyper parameters. The improvement based on CornerNet will be suitable for our nighttime vehicle detection task with high requirements of accuracy and speed. In addition, the detection box is rectangular, and the corner position can reflect the information of the rectangular. ConerNet is an anchor-free object detection method which detects an object bounding box as a pair of keypoints: the top-left corner and the bottom-right corner. Fig. 1 illustrates the overall pipeline of our approach.

Fig. 1

Architecture of nighttime vehicle detection network. A convolutional backbone network applies Direction Attention Pooling to output two corner heatmaps. Similar to CornerNet, a pair of detected corners and the similar embeddings are used to detect a potential bounding box. The instance mask obtained from fusion image segmentation is used for refining the corners by the Bayes corner localization algorithm.

In training phase, RGB images are inputted to object stream and pixel stream respectively. In the object stream, the features are extracted by backbone network, and then go through a special pooling layer—Direction Attention Pooling. The network outputs a heatmap for all top-left corners, a heatmap for all bottom-right corners, and an embedding vector for each detected corner. The potential bounding box can be obtained by matching the embedding vectors of the top-left corners and the bottom-right corners. In the pixel stream, the HSI segmentation image and the Nakagami segmentation image obtained from the RGB image are fused to obtain the segmentation image of the vehicle lights. An accurate bounding box must contain all pixels of the light mask. According to the aforementioned analyses, pixel-level information has the potential to benefit the detection task. To this end, we develop a formulation for corner localization based on the Bayes’ theorem. In this formulation, we utilize vehicle light segmentation from the pixel stream to select more accurate bounding box of each object.

The contributions of this paper are summarized as follows:

1) Different from the previous method [8 , 15] which extracts manual features from the image, and then uses SVM to classify the features, we use deep network to detect vehicles at night. A new special pooling layer—Direction Attention Pooling is proposed to make the corner points determination more accurate.

2) We use vehicle light segmentation image to finetune the bounding box of nighttime vehicle detection.

The rest of this paper is organized as follows. Related work is described in Section II. The details of the proposed nighttime vehicle detection approach based on deep network including network structure, loss function and corner localization algorithm based on Bayes theorem are introduced in Section III. The experiment results are shown and discussed in Section IV. Finally, Section V provides the conclusions and discusses future work.

2 Related work

Early stage night vehicle detection is based on the detection of headlights and taillights [20] which are the most salient areas of the night image. Pham et al. [72] noticed that bright blobs that may be vehicle lights are segmented in the captured image. Then, they proposed a machine learning-based method to classify whether the bright blobs are headlights, taillights, or other illuminant objects. Subsequently, the detected vehicle lights are tracked to further facilitate the determination of the vehicle position. The methods for vehicle light detection are divided into four parts: blobs detection, feature extraction, lights pairing and classification based on the common pipeline[20]. Antonio L‘opez et al. [12] used an image sensor to obtain different appearance features, which were used as the input of a classifier-based module. For each candidate blob, the module generates similarity with vehicle lighting to determine whether the blob is a vehicle light. Eum S et al. [21] integrated the partial region of the auto exposure(AE) image confined by the lane detection information and the low-exposure(LE) image. This approach enhances the performance of detecting the distant light blobs. VeDANt [22] employed AdaBoost cascade classifiers for detecting hypothesis blobs using gray-scale images only. Nakagami image combing with an HSI segmentation method is utilized to locate vehicle light areas [10]. Blobs feature extraction is used for lights pairing and classification [23 –25]. Through these blobs feature extraction methods one can get the luminance, appearance, geometric and other attributes of blobs. Pairing is used to distinguish the vehicle light areas from other luminous areas. Andrea Fossati et al. [26] proposed pair filter to filtrate out pair bond which cannot be a vehicle of candidate light blobs. Yen-Lin Chen et al. [27] grouped the luminous regions by projection-based spatial clustering method to obtain potential pairing vehicle lights. Classification is used to distinguish between vehicle lights, other light sources and background. The most commonly used methods are boosting [12] and support vector machine [23].

Different from the transformation of vehicle detection at night into vehicle lights detection, there are some methods which are based on effective feature extraction and machine learning to directly detect the whole vehicle at night. Hulin Kuang et al. [8] proposed a nighttime vehicle detection approach, which combines vehicle light areas obtained by Nakagami segmentation image and object proposals by Edge Boxes [28] with nighttime image enhancement based on improved MSR(Multi-Scale Retinex, a method of image enhancement). In the following research [9], they developed a nighttime image enhancement method which was inspired by the retinal mechanism and a weighted feature fusion technique. Subsequent study [29] proposed a Bayes saliency-based object proposal generator for nighttime images to generate an exact set of proposals, which are more likely to be the locations of the vehicles. In the latest study, Hulin Kuang et al. [15] present a vehicle detection system which focuses on four types of vehicles based on tensor decomposition. They selected proposals by feature ranking after tensor decomposition and only extract features from these selected proposals. For the night vehicle detection task, Hossein Tehrani et al. [30] optimized the visible and potential structure of the detected object in the process of using deformable part models (DPM) to capture salient features.

Since the development of deep learning, CNN has shown good performance in object detection tasks, and a series of excellent frameworks have emerged. For example, the two-stage detection network RCNN [31], Faster R-CNN [16], FPN [32], Cascade R-CNN[33], CBNet [73], etc. And the single-stage network YOLO [17 , 35], SSD [18], EfficientDet [36], DETR(DEtection Transformer) [74], etc. One-stage detectors place anchor boxes densely over an image and generate final box predictions by scoring anchor boxes. Anchor box structure has two disadvantages: one is that the number of anchor boxes is large, which is easy to cause uneven positive and negative samples; the other is that it will introduce many hyper parameters, such as quantity, size and aspect ratio. Therefore, anchor free methods[19 , 75] which balance accuracy and speed began to emerge. In the field of automatic driving, the method of CNN feature extraction has achieved good performance [40 –42] in the daytime vehicle detection task of KITTI [43] dataset. For example, Recurrent Rolling Convolution (RRC) [63] enhances single vehicle detection under occlusion. And Fan Yang al. [64] proposed Scale Dependent Pooling (SDP)to enhance the accuracy of proposal and Cascaded Rejection Classifiers (CRC) to eliminate negative samples to improve the detection accuracy. Inputting only CNN features into SVM to train a night-time vehicle classifier [31] is also effective. The latest innovative method used GAN(Generative Adversarial Network) structure. SHAO et al. [70] proposed a FTE(feature translate-enhancement) module based on CycleGAN and the OD(object detection) module to improve the accuracy of vehicle detection at night. Long Chen et al. [10] proposed a method to get nighttime vehicle proposals by fusing the light region and the proposals which obtained by using RPN method, and then classify them to get vehicle bounding boxes. In subsequent research, Miao et al. [71] combined the MSR image enhancement method with YOLOv3 to improve the accuracy of night object detection. According to CornerNet [19] and [10], the vehicle light information is used to assist network training. However, there is a problem when this advanced object detection network is applied to our nighttime vehicle detection task. When detecting a single class of objects, there will be repeated detection. To be specific, when the correct objects are detected, there will be a larger bounding box covering them, as example is shown in Fig. 2. To address the problem, we design improvement scheme in the object stream and the pixel stream.

Fig. 2

Repeated detection. Red indicates the correct bounding boxes, and blue indicates the repeated bounding box.

Our method takes RGB images as input, which are conveyed to object stream and pixel stream respectively. The object stream detects the candidate region based on a convolution network. The convolution network is used to detect the heatmaps of corner locations of the candidate regions. In the process of detecting the heatmaps, we draw lessons from attention mechanism. Its core goal is to select the information that is more critical to the current task from a lot of information, and reach an agreement with our goal of finding the key points in the picture. And attention mechanism is also well used in the object detection task. For example, the last ImageNet champion model: SENet [65], proposed a new convolution operator, which learns new feature map from input feature map through convolution kernel. AC-FPN [66] designs a new network structure, which is called attention oriented context feature pyramid network. By fusing multiple different receptive field features, the network structure not only increases the receptive field of objects, but also makes use of the context information of objects. In addition, DSC [44] is used for shadow detection task. For each feature grid, the information of its surrounding grid can be obtained, so as to distinguish the shadow. Each corner has its own embedding vector. By matching the embedding vectors of the top-left corner and the bottom-right corner, whether the two corners belong to the same object can be determined. Pixel stream uses the vehicle lights segmentation image to assist the candidate regions which are generated in the object stream to get more accurate bounding boxes.

3 The proposed nighttime vehicle detection method

Our proposed nighttime vehicle detection method includes two important parts: 1) Object Stream; and 2)Pixel Stream. They are described in detail in this section.

3.1 Object stream

The object stream is used to detect the candidate regions in the nighttime image. The structure of object stream is similar to that of CornerNet. The features of the original image are extracted by backbone(Hourglass) network, and then the features pass Direction Attention Pooling to obtain direction perception feature. And then, the direction perception features pass a ReLU layer and a convolution layer to obtain three branches. They are heatmaps for locating corner key points, embeddings for pairing and offsets for position calibration, respectively. The procedure shown in Fig. 3.

Fig. 3

The main structure of object stream. The features are processed separately into three branches for different usages: generating the heatmaps, extracting embeddings and predicting offsets.

3.1.1 Detecting Corners

The most critical structure used by CornetNet to detect key points is a special layer called Corner Pooling. The mechanism of Corner Pooling is as follows. Take the situation of detecting top-left corners as example (and similar in bottom-right case). There are two parallel feature maps from backbone network used in the procedure, one put through horizontal and the other put through vertical pooling, and the output is the element-wise sum of the two pooled maps. The horizontal pooling is that, given a grid in the feature map, the value is set as the maximum among the values to its right-side. In a similar manner, vertical pooling selects the maximum among down-side values. The scheme is shown in Fig. 4. The principle is that the intersection of the upper boundary and the left boundary of an object is the top-left corner. The maximum value in the Corner Pooling reflects the information of the top and left boundaries.

Fig. 4

The top-left Corner Pooling layer.

It is obvious from Fig. 2 (b) that the straight line extending horizontally from the bottom-right corner of the blue bounding box intersects the lower boundary of the silver car, and the vertical extension straight line intersects the right boundary of another red car. At the same time, this repeated detection bounding box(blue) does meet the operation criteria of Corner Pooling. It can be explained that the Corner Pooling has defects to locate the wrong location of corner points. And Law et al. [19] replace the predicted heatmaps with the ground-truth heatmaps improves the AP from 38.4% to 73.1%, indicating that the main bottleneck of CornerNet is detecting corners. We analyze the reasons and find that Corner Pooling only focuses on the grid with the maximum value and ignores the information of other grids. But not all the information carried by other pixels is useless.

Inspired by the shadow detection method DSC [44], we propose Direction Attention Pooling(DAP). The direction-aware spatial context module (DSC) module pays attention to the difference between each pixel and the surrounding pixels. DSC focuses on the information carried by other grids in four directions of a pixel, and gives them different attention weights. We proposed method Direction Attention Pooling is different from it. We change its method of perceiving the surroundings into sensing the right and down directions (for the topleft corner), as shown in Fig. 5. The features of the original image after passing through the backbone network are divided into two branches, one for learning attention weight W, the other for computing direction-aware spatial context by adopting a spatial RNN. To learn spatial context in a direction-aware manner, we formulate the direction-aware attention mechanism in a spatial RNN to learn attention weights and generate features. Spatial RNN is a special RNN structure. It can sense the context information of other spatial directions. The operation of the structure is explained as follows. As shown in Fig. 6. Input feature $h_{i, j}^{direction}$ can be updated from four directions (i.e. up, down, left and right). Take right-side as example, the new $h_{i, j}^{right}$ can be calculated by

$h_{i, j}^{right} = \max {M_{hw}^{right} h_{i, j + 1}^{right} + h_{i, j}^{right}, 0}$ (1)

$M_{hw}^{right}$ is the recurrent weight matrix. Sean Bel et al. [45] thinks that the 1 × 1 convolution from input layer to hidden layer and from hidden layer to output layer contains rich context information, so the recurrent weight matrix can be eliminated. $M_{hw}^{right}$ is set to be the identity matrix. It was named IRNN. Naturally formula (1) becomes

$h_{i, j}^{right} = \max {h_{i, j + 1}^{right} + h_{i, j}^{right}, 0}$ (2)

Fig. 5

The schematic illustration of the Direction Attention Pooling. (top-left corner).

Fig. 6

The schematic illustration of how information propagates in IRNN from two direction.

Formulas (1) and (2) are proposed in [45]. Our method is different from the original use of IRNN. Sean Bel et al. [45] originally wanted to use IRNN layer twice to make each grid of feature map get the global receptive field. We only focus on the information in the right and down directions of each grid. When detecting the heatmap of top-left corner, the features which are obtained from the backbone network pass the IRNN layer to get two new features h^right and h^down. Then, we split the W learned by the other branch into two maps of attention weights, denoted by W^right and W^down. The two maps of weights W^right and W^down are multiplied with the spatial context features h^right and h^down in an element-wise manner. Finally, we concatenate the features from the two directions. The above is the overall structure and operation mechanism of Direction Attention Pooling. Algorithm 1 is the Direction Attention Pooling pseudocode.

Algorithm 1.

Direction Attention Pooling calculation
process(top-left corner)
Input:H is the feature map from backbone;h_i,j ∈ H
is the eigenvalue of the position (i, j) , i ∈ I, j ∈ J
Output: the direction attention feature map H^DA
1. (WeightBranch, IRNNBranch)← input(H)
2. WeightBranch
3. W← two 3 × 3conv(H)
4. W′← 1 × 1conv(W)
5. (W_right, W_down)← split(W′)
6. IRNNBranch
7. H′← 1 × 1conv(H)
8. fori ← 0 to I, j ← 0 to J,h_i,j ∈ H′ do
9. $h_{i, j}^{right}$ ← formula(2)( $h_{i, j}^{right}, h_{i, j - 1}^{right}$ )
10. $h_{i, j}^{down}$ ← formula(2)( $h_{i, j}^{down}, h_{i - 1, j}^{down}$ )
11. end for
12. H^right^′←element-wise(H^right, W_right),
H^down^′←element-wise(H^down, W_down)
13. H″←concatenate(H^right^′,H^down^′)
14. H^DA←1 × 1conv(H″)

For each corner, there is one ground-truth positive location, and the other locations are negative. p_cij denotes the probability of the pixel at location (i, j) belonging to a class c object. In our task, there are C types, and no background. y_cij is the ground-truth heatmap augmented with the unnormalized Gaussians $σ = \frac{1}{3}$ and the value represents the probability of location (i, j) being a corner point. (i_G, j_G) is the ground-truth. The loss designed by CornerNet retains the principle of focal loss [46], which reduces the impact of simple samples on the overall loss. The closer the location (i, j) is to the ground-truth, the closer the y_cij is to 1. And vice versa. According to Law H et al.’s point of view [19], the location (i, j) closer to the center is more similar to the positive sample, and has a relatively high IOU with ground-truth. When the negative sample training does not need to pay too much attention to these areas, using (1 - y_cij) ^β to reduce its weight.

$y_{cij} = e^{- \frac{{(i - i_{G})}^{2} + {(j - j_{G})}^{2}}{2 σ^{2}}}$ (3) $\begin{matrix} L_{\det} = \frac{- 1}{N} \sum_{c = 1}^{C} \sum_{i = 1}^{H} \sum_{j = 1}^{W} {\begin{matrix} {(1 - p_{cij})}^{α} \log (p_{cij}), y_{cij} = 1 \\ {(1 - y_{cij})}^{β} {(p_{cij})}^{α} \log (1 - p_{cij}), other \end{matrix} \end{matrix}$ (4)

Formulas (3) and (4) are proposed in [19], where N is the number of objects in an image, and α and β are the hyper-parameters which control the contribution of each point (we set α to 2 and β to 4 in all experiments).

When the feature extraction process of the original image is equivalent to down sampling the original image, the output is generally smaller than the original image. The original image position (x, y) is mapped to position $(⌊ \frac{x}{n} ⌋, ⌊ \frac{y}{n} ⌋)$ in the feature map. The n value of the task in this paper is 4. When the position (x, y) of the feature map remapping original image position, some information will be lost, which will make it hard to predict small bounding boxes accurately. To solve this problem, we use L1 Loss to train and predict the position offset, so that we can adjust the corner position before remapping.

$o_{k} = (\frac{x_{k}}{n} - ⌊ \frac{x_{k}}{n} ⌋, \frac{y_{k}}{n} - ⌊ \frac{y_{k}}{n} ⌋)$ (5) $L_{off} = \frac{1}{N} \sum_{k = 1}^{N} {SmoothL}_{1} Loss (o_{k}, {\hat{o}}_{k})$ (6)

Formulas (5) and (6) are proposed in [19], where o_k is the offset, x_k and y_k are the coordinate for corner k. o_k is the true value and ${\hat{o}}_{k}$ is the predicted value.

3.1.2 Grouping Corners

The embedding vector is used for matching to determine whether the detected top-left corner and bottom-right corner belong to the same object. The relationship between feature map of heatmap and embedding vector is shown in Fig. 7. According to Newell et al. [47], the matching principle is that the relative distance between the top-left corner and the bottom-right corner of the same object will be smaller. Therefore, we can match the corners of the same object by "pull" loss, and separate the corners of different objects by "push" loss in training. e_tk and e_bk are the embedding for the corner of object k. e_k is the average of e_tk and e_bk. Δ is set to be 1. Formulas (7) and (8) are proposed in [47].

$L_{pull} = \frac{1}{N} [{(e_{tk} - e_{k})}^{2} + {(e_{bk} - e_{k})}^{2}]$ (7) $L_{push} = \frac{1}{N (N - 1)} \sum_{k = 1}^{N} \sum_{j = 1, j \neq k}^{N} \max (0, Δ - | e_{k} - e_{j} |)$ (8)

Fig. 7

The network outputs a heatmap for corners and embedding vectors.

3.2 Pixel stream

For the automatic driving nighttime image, various light sources are significant information, and the light information has a strong positive impact on vehicle detection. In this section, we demonstrate how to enhance the detection results by utilizing the pixel-level vehicle light information. First of all, we use the same method in the article [10] to get the segmentation image of the vehicle light. As shown in Fig. 8. The contrast of the original image Fig. 8(a) is enhanced by the method [48] to get Fig. 8(b), which makes the distinction between the object and the background more obvious. Because Nakagami image [49] emphasizes the light and its surrounding scattering area, the Nakagami image Fig. 8(c) after processing can be used to detect the light location in the image. The segmentation result shown in Fig. 8(d) is obtained from the Nakagami images using an adaptive threshold [25]. Nakagami image contains other lights besides the vehicle lights, so it is necessary to distinguish the vehicle lights from other lights. When the nakagami image is converted to HSI color space, there are different hue values for different lights. According to the hue distribution of the article [10], most of the turn lights and tail lights are distributed between 0 to 0.05, 0.45 to 0.6, 0.96 to 1. We set the threshold based on this range. The segmentation results of the HSI color space are shown in Fig. 8(e). For HSI segmentation and Nakagami image segmentation, we use and(&) operation to fuse them to get Fig. 8(f). And then, for fusion image,we use 8-connected component labeling method to get vehicle light locations. We use this method to preprocess all the images in the dataset.

Fig. 8

Vehicle lights segmentation image acquisition process: (a) original image, (b) enhanced image, (c) Nakagami image, (d) Nakagami image segmentation, (e) HSI segmentation, (f) fusion image.

From the above processing for the original image, we obtain the areas of vehicle light. From an intuitive point of view, if a bounding box contains more vehicle light pixels, this box is more likely to be the best bounding box. But in fact, most of the bounding boxes detected by the object stream contain all the vehicle light pixels. Secondly, we proposed the repeated detection problem in the related work, most of which also include all the vehicle light pixels. Therefore, the detection accuracy can not be improved by simply judging whether the vehicle light pixels are included or not, and more or less. Next, we will introduce how to use the pixel level information of the vehicle lights to calibrate bounding box of the vehicle detection. Inspired by mask based boundary modification module (MBRM) [50], we develop the formulation for corner localization based on the Bayes theorem.

Although the bounding box detected in the object stream have certain location error, it has provided rich prior information for the final accurate bounding box. Therefore, our formula combines the detection results of object stream and the segmentation result of pixel stream. Specifically, we consider the locations of the corners as a discrete random variable. From the probabilistic perspective, an object corner location is the argmax of the probability of a coordinate where the corner locates, namely

$z = argmaxP (Z = z_{x, y} | M^{'})$ (9) where z is the discrete random variable for the corner coordinate of the top-left corner, M′ is the output of pixel stream, that is, the set of pixels at the vehicle light position. P (Z = z_x,y|M′) denotes the posterior probability given the corresponding instance mask M′.

Now we take the calculation of the top-left corner as an example. After corner embedding vectors matching through the object stream, a top-left corner has been matched with a bottom-right corner, so a top-left corner can represent a bounding box. Following the Bayes’ theory, we have

$P (Z = z_{x, y} | M^{'}) = \frac{P (Z = z_{x, y}) P (M^{'} | Z = z_{x, y})}{\sum_{t \in R} P (Z = t_{x, y}) P (M^{'} | Z = t_{x, y})}$ (10) where P (Z = z_x,y) and P (M′|Z = z_x,y) are the corresponding prior and likelihood probabilities. R is the set of prior corners.

For P (Z = z_x,y), the definition is similar to section 3.1.1, and it reflects the distance between the detection point and the ground-truth. We simply adopt a discrete Gaussian distribution

$P (Z = z_{x, y}) = α e^{- \frac{{(z_{x} - z_{x_{G}})}^{2} + {(z_{y} - z_{y_{G}})}^{2}}{2 σ^{' 2}}}$ (11) where α is the normalization coefficient. z_{x
_G}, z_{y
_G} is the top-left corner coordinate of the ground-truth. Obviously, the distribution of the corner location is related to the instance scale, thus we set

$σ^{'} = γ \min (w_{h}, w_{w})$ (12)γ specifies the weighting. σ′ delimits a range in which bounding box and ground truth have higher IOU. In our experiment, we set γ to $\frac{1}{3}$ . w_h, w_w is the height and width of the ground truth. Likelihood probabilities represent the vehicle light instance mask prediction of the posterior probability of the given corner location. We defined it as $P (M^{'} | Z = z_{x, y}) = \frac{{Pixel}_{light}}{{Pixel}_{bbox}}, {Pixel}_{light} \in {Pixel}_{bbox}$ (13) Pixel_bbox represents the collection of pixels in the bounding box with Z = z_x,y as the top-left corner, which is equivalent to the area of the bounding box. Pixel_light is the set of pixels belonging to the vehicle light location in the bounding box. Next, we illustrate the role of this likelihood probability in solving the repeated detection problem. As shown in Fig. 9. We have a single object detection bounding box(red) and a repeated detection bounding box(blue). When we calculate P (Z = z_x,y|M′) respectively, since the denominator of the bounding box belonging to the same object is the same, according to the control variable method, it is assumed that P (Z = z_x,y) is also the same. So the difference is P (M′|Z = z_x,y). According to the calculation method of likelihood probability, we can get inequality (14). Normally, ${Pixel}_{light}^{orange} ⪡ ({Pixel}_{bbox}^{blue} - {Pixel}_{bbox}^{red})$ . The number of numerator growth is smaller than that of denominator, which will lead to smaller original number. Finally, the P (Z = z_x,y|M′) of single object detection bounding box is bigger than that of the repeated detection one.

Fig. 9

Calculation diagram about single object detection bounding box and repeated detection one.

$\begin{matrix} \frac{{Pixel}_{light}^{yellow}}{{Pixel}_{bbox}^{red}} > \\ \frac{{Pixel}_{light}^{yellow} + {Pixel}_{light}^{orange}}{{Pixel}_{bbox}^{red} + ({Pixel}_{bbox}^{blue} - {Pixel}_{bbox}^{red})} = \\ \frac{{Pixel}_{light}^{yellow} + {Pixel}_{light}^{orange}}{{Pixel}_{bbox}^{blue}} \end{matrix}$ (14) During training, the ground-truth corner is transformed to one-hot format of the image, and we use cross entropy loss L_mask between true value and predicted value. Only the representations of positive corners from object stream value are fed into the pixel stream module to calculate P (Z = z_x,y|M′). And then we choose the top-10 corners,which are then cropped with the expanded ground-truth boxes and used to calculate L_mask. Algorithm 2 is the Bayes Corner Localization pseudocode.

Algorithm 2.

Bayes Corner Localization Algorithm
Input: priori bounding boxes set B,b ∈ B;their top-left
corners set Z,z_x,y ∈ Z; light vehicle pixels set
P; top-left corners ground-truth set Z_G,z_{x_G,y_G} ∈ Z_G;
Output: the top-10 bounding boxes b and their top-left
corners z_x,y
1. for each z_x,y ∈ Zdo
2. P (Z = z_x,y)←formula(11)(z_x,y,z_{x_G,y_G},b)
3. P (M′\|Z = z_x,y)←formula(13)(z_x,y,b,P)
4. P (Z = z_x,y\|M′)←formula(10)(P (Z = z_x,y),
P (M′\|Z = z_x,y))
5. end for
6. z_x,y← top10(formula(9)(P (Z = z_x,y\|M′)))
7. b← data processing(z_x,y)

4 Experiments

4.1 Training and inference

We implement our method in PyTorch. The network is randomly initialized under the default setting of PyTorch, except that we use pretrained backbone Hourglass [19]. As we apply focal loss, we follow [47] to set the biases in the convolution layers that predict the corner heatmaps. In training, we use Hourglass to be backbone network to extract features. We set the input resolution of the network to 511 × 511, which leads to an output resolution of 128 × 128. To reduce overfitting, we adopt standard data augmentation techniques including random horizontal flipping, random scaling, random cropping and random color jittering, which includes adjusting the brightness, saturation and contrast of an image. Finally, we apply PCA [59] to the input image. We use Adam [60] to optimize the full training loss: $L = L_{\det} + α L_{pull} + β L_{push} + γ L_{off} + η L_{mask}$ (15) where α, β, γ and η are the weights for the pull, push,offset and mask loss respectively. We set both α and β to 0.1, γ to 1 and η to 0.2. We find that 1 or larger values of α and β lead to poor performance. At the beginning of the training, we only used L_det, L_pull, L_push and L_off. L_mask joins the training later as a model fine-tuning. We use a batch size of 10 and train the network on 2 Titan X (PASCAL) GPUs. To conserve GPU resources, in our ablation experiments, we train the networks for 20k iterations with a learning rate of 5 × 10^-4. When we compare our results with other detectors, we train the networks for an extra 20k iterations and reduce the learning rate to 5 × 10^-5 for the last 4k iterations. During inference, we only use the object stream [19]. We use a simple post-processing algorithm to generate bounding boxes from heatmaps, embeddings and offsets. We use a 3 × 3 max pool layer to apply non maximum suppression (NMS) on the corner heatmaps. Then we select the top-50 top-left corners and bottom-right corners from the heatmaps. The corner position is adjusted by the corresponding offset. We calculate the L1 distance between the top-left and bottom-right corners. Pairs with distances greater than 0.5 are rejected. Instead of resizing the image to a fixed size, we keep the original resolution of the image and fill in zeros around border before sending it to our network. Both the original image and the flipped image are used for testing. We apply soft NMS to suppress redundant detection. Only the top 50 bounding boxes are reported. The average inference time is 227ms per image on a Titan X (PASCAL) GPU.

4.2 Dataset

We use Sun Yat-sen University Night-time 1 Vehicle Dataset. This dataset contains 5576 images including over 12000 nighttime vehicles in two different traffic situations, i.e., urban road with streetlight and without streetlight. All these images are captured by driving recorders set on the front windshield of vehicles. We used 4460 images as training set samples and 1116 images as test samples. The dataset is used to evaluate the proposed object proposal approach. In order to verify the generalization ability of this method, we also tested on another data set collected at night in Hong Kong named CHUK 2 and compared with other detection methods. This is a multi-class category dataset which contains 836 high resolution images with pixels. We use random clipping, flipping, and mirroring to enhance and expand dataset.

4.3 Results for Nighttime Vehicle Detection

4.3.1 Results of the Object Stream Proposal

In our method, the object stream provides proposal for nighttime vehicle detection task. The performance of the proposed method was measured using the detection rate versus number of proposals (ROIs) curve, as shown in Fig. 10. The method in reference [8] is superior to the original Edge boxes [28]. The method based on Bayesian saliency in reference [29] is superior to other methods based on vehicle light detection in references [51 –53]. The performance of the method combining edgeboxes, local contrast feature and image region similarity proposed in reference [15] is better than those in reference [8, 29]. In this paper, we compare the proposed method with those in references [15]. We find that when we only use the 18 proposals, our proposal generation method can obtain 100% detection rate, while the other method in references [15] obtain the same detection rate when using 20 proposals. Meanwhile, when the number of proposals is less than 20, the detection rate of our method will be higher with the same number of proposals. The results show that our proposed method outperforms other methods.

Fig. 10

Evaluations of our proposed object proposal method and the object proposal methods in [8 , 29].

4.3.2 Results of Miss Rate vs.FPPI

When our method is applied to vehicle detection at night, we use the false positives per image (FPPI) vs. miss rate curve of each image to evaluate the detection performance of some of the most advanced vehicle detection methods. In this curve, under the same FPPI, the lower the miss rate is, the better the method is. As shown in Fig. 11(a), we compare the detection results of CornerNet, without pixel stream and using pixel stream. The latter shows a lower miss rate than the former two under the same FPPI. Therefore, it can be inferred that the pixel stream is effective for the detection task. In order to further verify the role of image enhancement in vehicle detection, we compare the missed detection rate and false positives per image (FPPI) curves in Fig. 11(b). For comparison, we evaluate the proposed detection methods on the test set. In Fig. 11(b), the curve of our method is lower than that of all other curves, and shows a lower miss rate than the other five methods under the same FPPI. These results also show that our night detection method is more effective than other methods in vehicle detection at night.

Fig. 11

Comparison of the detection performance of our proposed method and other methods.(a) Detection performance of our proposed method. (b) Comparison with some state-of-the-art night-time vehicle detection approaches.

4.3.3 Visualization of Results

Fig. 12 shows some examples of the detection results of our proposed method. The bounding box of each object in each image has the highest IOU with the real ground-truth. From Fig. 12, we can see that the proposed method can detect vehicles with different sizes [Fig. 12(a)], different numbers [Fig. 12(b)], and different backgrounds [Fig. 12(c)]. Even partially obscured and blurred vehicles can be detected successfully. Multi-class detection in the CHUK dataset can also be detected [Fig. 12(d)]. The yellow, green and red bounding boxes represents the bus, the car and the taxi, respectively. The proposed method can detect cars, taxis, buses and minibuses successfully. But there are also cases of error detection, such as similar taxis and cars. [Fig. 12(e)] For example, the left car in the left picture is detected as a taxi, and the left most taxi in the right picture is detected as a car.

Fig. 12

The detection results of our proposed method.

4.4 Comparisons with State-of-the-art Detectors

In this section, we report the results of our proposed nighttime vehicle detection method and compare it with some of the most advanced detection methods in terms of vehicle recognition accuracy and vehicle detection performance. As shown in Table 2. And We name our method DANet+BCL (Direction Attention Network and Bayes Corner Localization).The accuracy represents the ratio of correct detection bounding boxes to all ground-truth bounding boxes. And if the IoU of a bounding box and ground-truth is greater than 0.5, we judge it as the correct. In the same dataset, we retain the same number of detected bounding boxes as the number of gound-truth in each image. Many effective methods of night detection are using manual feature extraction. For example, vehicle light blobs classifier [8, 72] combines vehicle lights detection and image enhancement, [9] uses bio-inspired image to enhance candidate region and [15] conducts feature extraction based on tensor decomposition. There are also methods based on CNN framework, such as CNN feature extraction combined with SVM classification method, CNN combined with nakagmi image extraction fusion feature method [10], and single-stage network YOLOv3 [34] and image-enhanced YOLOv3 [71]. The GAN nighttime object detection method FTE [70] has also been added for comparison. From Table 2, we can see that the accuracy of our proposed method is 97.2%, which is 0.31% and 1.62% higher than that of the best manual feature extraction method Tensor Decomposition [15] and the best CNN detection method [10] respectively on CHUK dataset. The accuracy of our proposed method is 96.86%, which is 0.34% and 1.65% higher than that of the best manual feature extraction method and the best CNN detection method respectively on SYSU dataset.

Table 2
Comparisons Of Accuracy(%) on SYSU Dataset and CUHK Dataset

Method CHUK SYSU

CNN+SVM [54] 89.02 88.59

The method in [72] 91.38 91.03

The method in [8] 92.47 92.09

The method in [9] 94.51 94.19

YOLO v3 [35] 95.39 95.02

CNN+ImEn [10] 95.58 95.21

YOLO v3+MSR [71] 96.03 95.78

YOLO v3+FteOD [70] 96.72 96.17

Tensor Decomposition [15] 96.89 96.52

DANet+BCL 97.2 96.86

Method	CHUK	SYSU
CNN+SVM [54]	89.02	88.59
The method in [72]	91.38	91.03
The method in [8]	92.47	92.09
The method in [9]	94.51	94.19
YOLO v3 [35]	95.39	95.02
CNN+ImEn [10]	95.58	95.21
YOLO v3+MSR [71]	96.03	95.78
YOLO v3+FteOD [70]	96.72	96.17
Tensor Decomposition [15]	96.89	96.52
DANet+BCL	97.2	96.86

We tested the inference speed of our algorithm and other solutions on NVIDIA Titan X single GPU. Table 3 shows a comparison with detection time. For each different method, we first use the train set for training and test set for verification. When the accuracy reaches the value of the paper, stop training. And then, to ensure fair comparison, we set the image input size of each method to 511 × 511. We record the inference time for each picture. For our method, the average inference time is 227 ms per image. At the same time, the use of Hourglass-52 backbone network can speed up the inference speed. Our Hourglass-52 takes an average of 180 ms to process one image, which is faster than Hourglass-104. When using GPU, the method of manual feature extraction has not improved greatly in speed. For example, the method [15] needs 180 ms per image in CPU and 165 ms per image in GPU. This is because the data transmission will have a great cost, and GPU is slower than CPU in processing data transmission. And GPU’s specialty which is computing in deeper and wider network can not be clearly reflected in simple parameter calculation. Compared with the original CornerNet, the learning speed does not significant increase in training time. Because learning attention weights and IRNN layer in the attention pool module is parallel computing. In addition, the calculation method of IRNN is faster than that of CNN which is because the element-wise calculation is time-consuming.

Table 3

Detection Time of Methods Compared in This Paper

Method	Time(seconds in per image)
CNN+SVM [54]	1.2
CNN+ImEn [10]	0.87
The method in [9]	0.47
The method in [8]	0.35
The method in [72]	0.31
CornerNet[19]	0.244
Tensor Decomposition [15]	0.165
YOLOv3 [35]	0.1
YOLOv3+MSR [71]	0.12
YOLOv3+FteOD [70]	0.36
DANet+BCL(Hourglass-104)	0.227
DANet+BCL(Hourglass-52)	0.18

Since our method is based on deep network, we also need to compare it with advanced deep network. We use the open source network that has been trained, and fine-tune it with the train set of CHUK dataset. When the loss of train set and test set no longer decreases and are small enough, the training will be stopped. Table 4 shows a comparison with state-of-the-art detectors on the CHUK test-dev set. We use AP metrics to characterize the performance of a detector. AP represents the average precision rate for all categories. The precision rate is the ratio of correct detection bounding boxes to all detection bounding boxes. And if the IoU of a bounding box and ground is greater than threshold, we judge it as the correct. And we calculated the AP when the IoU threshold is set to be 0.3,0.5 and 0.8. Compared with the reference CornerNet [19], our proposed method has significant improvement. For example, Our Hourglass-101 (i.e., the input image resolution is 511 × 511, and the backbone is Hourglass-101) reports that under the same settings, our test AP, AP_0.3, AP_0.5 and AP_0.8 is 98.43%, 97.2%, 80.68% respectively, which is 3.45%, 3.09%, 2.83% and 4.45% higher than origin CornerNet. When the mild network (Hourglass-52) was used, the AP improvement of CornerNet was 4.4% (from 86.96% to 91.27%). We also replaced the backbone with ResNet-101 [55], and the corresponding AP is 90.63%. These results demonstrate the effectiveness of our method and Hourglass. Our method achieves 92.1% testing AP. Compared to traditional two-stage approaches, our method is 4.93% higher than Faster-RCNN/FPN [32]. Compared with other relatively advanced two-stage approaches D-FCN [56], Mask R-CNN [57],Casecade-RCNN [33] and CBNet [73], our method improved by 4.33%, 2.46%, 2.01% and 1.5% respectively. As for single-stage network, our method is 7.93% and 5.41% higher than YOLOV3 and SSD513 [18] respectively. In the case of anchor-free network, our AP is 4.96% and 1.2% higher than FCOS [39] and FreeAnchor [75]. Compared with a series of improved networks based on CornerNet, such as Cornernet-Lite [58] and CenterNet [37], we perform better. Even with the most advanced single-stage network DETR [74] and EfficientDet [36], our Hourglass-101 AP is 3.68% and 0.4% higher.

Table 4

Comparisons with object detection network

Method	backbone	Test Input Size	AP	AP _0.3	AP _0.5	AP _0.8
Faster-RCNN/FPN[32]	ResNet-101	1000 × 600	87.17	94.97	94.32	72.24
D-FCN[56]	Inception-Restnet	1000 × 600	87.77	95.46	94.18	73.67
Mask R-CNN[57]	ResNeXt-101	800 × 600	89.64	95.32	94.39	79.2
Casecade-RCNN[33]	ResNet-101	800 × 600	90.09	95.89	94.67	79.72
CBNet[73]	Dual-ResNeXt152	1333 × 800	90.6	96.35	95.63	79.83
YOLOV3[35]	DarkNet-53	544 × 544	84.17	95.95	95.09	61.49
SSD513[18]	ResNet-101	513 × 513	86.69	96.21	95.23	68.63
FCOS[39]	ResNet-101	1024 × 800	87.14	94.78	94.34	72.29
DETR[74]	FFN	1000 × 600	88.42	95.76	94.94	74.57
EfficientDet[36]	EfficientNet	512 × 512	91.7	98.14	96.9	80.23
CornerNet	Hourglass-52	511 × 511	86.96	93.98	93.23	73.67
CornerNet[19]	Hourglass-101	511 × 511	88.65	95.34	94.37	76.23
Cornernet-Lite[58]	Hourglass-101	511 × 511	90.12	96.78	95.32	78.25
CenterNet[37]	Hourglass-101	511 × 511	90.52	96.92	95.26	79.39
FreeAnchor[75]	ResNeXt-101	800 × 800	90.9	96.87	95.53	80.392
DANet+BCL	ResNet-101	511 × 511	90.63	95.97	95.7	80.21
DANet+BCL	Hourglass-52	511 × 511	91.27	97.32	96.92	79.56
DANet+BCL	Hourglass-101	511 × 511	92.1	98.43	97.2	80.68

4.5 Ablation Study

Our work mainly improves the night vehicle detection in two parts: the direction attention pooling layer of object stream and the pixel stream. In order to analyze the contribution of each individual component, we do the ablation studies. The standard is the accuracy when IOU is 0.5. We split the details of these two modules, as shown in Table 5. The direction attention pooling layer is divided into two small details, one is learning attention weights, the other is using IRNN layer. The pixel stream is also divided into two details, one is using pixels to assist training, the other is the Bayes corner localization algorithm. We take the proposed method CornerNet as the original method. In the later experiment, we use the original network, delete the modules that need to be discussed, and then retrain them. In the first experiment, we use CornerNet as object stream and add the pixel stream, and the accuracy is 95.12%. Compared with the original 94.37%, it is increased by 0.75%. In the second experiment, we only use the object stream containing direction attention pooling layer. The result is 94.67% which is 0.3% higher than original. In the third experiment, we got the bounding box from the object stream. Instead of using corner localization algorithm based on Bayes, number of vehicle light pixels covered in each bounding box is used for sifting. We determine that the top-10 bounding boxes containing the most pixels of vehicle light is the best bounding boxes. The result is 94.26%, which is worse than that of the second experiment. The reason is that some bounding boxes may cover the lights of other objects, resulting in a lot of pixels covered, and these improper boxes are retained. The fourth experiment is the method proposed in this paper, it is increased by 2.83% compared to the original method. In the fifth experiment, in the direction attention pooling layer of object stream, we do not use IRNN layer, but pass through two 3 × 3convolution layer. The final result is 93.38%, which is 3.82% lower than the method proposed in this paper. In the sixth experiment, in the direction attention pooling layer, attention weights were not learned. The final result is 89.72%. Through the fifth and sixth experiments, the performance is even worse than the original. We can see that the attention weights must cooperate with IRNN layer to obtain better effect. According to these experiments, the attention pool of the object stream has a great impact on the performance of the algorithm.

Table 5
The Result of Ablation Study

Pool DAP-Attention Weight DAP-IRNN Pixel Stream Pixel Stream-Bayes Accuracy

Original 94.37

1 ✓ ✓ 95.12

2 ✓ ✓ 94.67

3 ✓ ✓ ✓ 94.26

4 ✓ ✓ ✓ ✓ 97.2

5 ✓ ✓ ✓ 93.38

6 ✓ ✓ ✓ 89.72

Pool	DAP-Attention Weight	DAP-IRNN	Pixel Stream	Pixel Stream-Bayes	Accuracy
Original					94.37
1			✓	✓	95.12
2	✓	✓			94.67
3	✓	✓	✓		94.26
4	✓	✓	✓	✓	97.2
5	✓		✓	✓	93.38
6		✓	✓	✓	89.72

5 Conclusion

In this paper, we propose a new method for vehicle detection at night. Due to the characteristics of low contrast and low brightness between the background and the object of night image, unmodified CNN feature extraction method tends to suffer from information loss occurred mainly during down sampling. For this reason, most of the night vehicle detection methods are using complex manual feature extraction. Our method provides remedy for this problem by adding vehicle light information. We used Nakagami segmentation and HSI segmentation fusion to get the vehicle light segmentation image, and then use the Bayes Corner Localization algorithm to finetune the bounding boxes of the detection network.We improved the reference network CornerNet by introducing the directional attention pooling into the corner detection module, and obtained better detection performance than the original. In order to verify the generalization of the model, we experimented on two different datasets and achieved state-of-the-art accuracy. Finally, we also compared with the current advanced network, and show superior performance than them.

In the future work, the existing night vehicle datasets have the problem of insufficient scenes and low resolution. We plan to build a new data set covering more scenes, more categories and of higher resolution. In addition, brightness of the training images is still enough for relatively reliable detection. However, there are some extreme circumstances in reality where there is hardly any light. How to deal with the detection task in such situation will be investigated next.

Footnotes

Acknowledgments

This work is supported by National Key R&D Program of China (2018YFC0309400), National Natural Science Foundation of China (61871188), Guangzhou City Science and Technology Research Projects (20190 2020008).

The SYSU dataset can be found in wwww.carlib.net/?page_id=35

The CHUK dataset can be found in .

References

Abdul Hamid

U.Z.

, Ahmad Zakuan

F.R.

, Akmal Zulkepli

, Zulfaqar Azmi

and Aizzat Zakaria

, Autonomous emergency braking system with potential field risk assessment for frontal collision mitigation. In 2017 IEEE Conference on Systems, Process and Control (ICSPC), 2017.

National Highway Traffic Safety Administration. National highway traffic safety administration’s traffic safety facts annual report. Technical report, 2018.

Arditi

, Eun Lee

and Polat

, Fatal accidents in nighttime vs. daytime highway construction work zones, Journal of Safety Research 38(4) (2007), 399–405.

Gray

R.C.

, Quddus

M.A.

and Evans

, Injury severity analysis of accidents involving young male drivers in great britain, J Safety Res 39(5) (2008), 483–495.

Dalal

, Histograms of oriented gradients for human detection. Proc of Cvpr, 2005.

Ahonen

, Hadid

and Pietikainen

, Face description with local binary patterns: Application to face recognition, IEEE Transactions on Pattern Analysis & Machine Intelligence 28(12) (2006), 2037–2041.

Felzenszwalb

P.F.

, Girshick

R.B.

, McAllester

D.R.

and Deva. Object detection with discriminatively trained part-based models, IEEE Transactions on Pattern Analysis & Machine Intelligence, 2010.

Kuang

, Long

, Feng

, Chen

, Chan

and Hong

, Combining region-of-interest extraction and image enhancement for nighttime vehicle detection, IEEE Intelligent Systems 31(3) (2016), 57–65.

Kuang

, Zhang

, Li

Y.J.

, Lai Hang Chan

and Yan

, Nighttime vehicle detection based on bioinspired image enhancement and weighted score-level feature fusion, IEEE Transactions on Intelligent Transportation Systems 18(4) (2017), 927–936.

10.

Chen

, Hu

, Xu

, Kuang

and Li

, Turn signal detection during nighttime by cnn detector and perceptual hashing tracking, IEEE Transactions on Intelligent Transportation Systems PP(12) (2017), 1–12.

11.

Juri´c

and Lončari´c , Sven A method for on-road nighttime vehicle headlight detection and tracking. In 2014 International Conference on Connected Vehicles and Expo (ICCVE), 2014.

12.

Lopez

Antonio M.

, Hilgenstock

, Busse

, Baldrich

and Serrat

, Nighttime vehicle detection for intelligent headlight control. In International Conference on Advanced Concepts for Intelligent Vision Systems, 2008.

13.

Chen

Y.L.

and Chiang

C.Y.

, Embedded on-road nighttime vehicle detection and tracking system for driver assistance. In Proceedings of the IEEE International Conference on Systems, Man and Cybernetics, Istanbul, Turkey, 10-13 October 2010, 2010.

14.

Charash

, Reception through nakagami fading multipath channels with random delays, IEEE Transactions on Communications 27(4) (1979), 657–670.

15.

Kuang

, Chen

and Hang

L.L.

, Chan, Ray C. C. Cheung and H. Yan, Feature selection based on tensor decomposition and object proposal for night-time multiclass vehicle detection, IEEE Transactions on Systems Man & Cybernetics 49(1) (2019), 71–80.

16.

Ren

, He

, Girshick

and Sun

, Faster r-cnn: Towards real-time object detection with region proposal networks, IEEE Transactions on Pattern Analysis & Machine Intelligence 39(6) (2017), 1137–1149.

17.

Redmon

, Divvala

, Girshick

and Farhadi

, You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, (2016), pp. 779–788.

18.

Liu

, Anguelov

, Erhan

, Szegedy

and Berg.

Alexander C.

, Ssd: Single shot multibox detector. In European Conference on Computer Vision, 2016.

19.

Law

and Deng

, Cornernet: Detecting objects as paired keypoints. In Proceedings of the European Conference on Computer Vision (ECCV), 2018.

20.

Juri´c

and Lončari´c , Sven A method for on-road nighttime vehicle headlight detection and tracking. In 2014 International Conference on Connected Vehicles and Expo (ICCVE), 2014.

21.

Eum

and Jung

H.G.

, Enhancing light blob detection for intelligent headlight control using lane detection, IEEE Transactions on Intelligent Transportation Systems 14(2) (2013), 1003–1011.

22.

Kumar Satzoda

and Manubhai Trivedi

, Looking at vehicles in the night: Detection and dynamics of rear lights, IEEE Transactions on Intelligent Transportation Systems, 2016.

23.

Alcantarilla

Pablo F.

, Bergasa

Luis M.

, Jimenez

, Sotelo

M.A.

, Parra

, Fernandez

and Mayoral

S.S.

, Night time vehicle detection for driving assistance lightbeam controller. In 2008 IEEE Intelligent Vehicles Symposium, (2008), pp. 291–296. IEEE.

24.

Connell

Jonathan H.

, Herta

, Pankanti

, Hess

and Pliefke

, A fast and robust intelligent headlight controller for vehicles. In 2011 IEEE Intelligent Vehicles Symposium (IV), (2011), pp. 703–708. IEEE.

25.

Alcantarilla

Pablo F.

, Bergasa

Luis M.

, Jimenez

, Sotelo

M.A.

, Parra

, Fernandez

and Mayoral.

S.S.

, Night time vehicle detection for driving assistance lightbeam controller. In 2008 IEEE Intelligent Vehicles Symposium, (2008), pp. 291–296. IEEE.

26.

Fossati

, Schonmann

and Fua

, Real-time vehicle tracking for driving assistance, Machine Vision and Applications 22(2) (2011), 439–448.

27.

Chen

Y-L.

and Chiang

C-Y.

, Embedded visionbased nighttime driver assistance system. In 2010 International Symposium on Computer, Communication, Control and Automation (3CA) 2 (2010), pp. 199–203. IEEE.

28.

Lawrence Zitnick

and Dollar

, Edge boxes: Locating object proposals from edges. In European conference on computer vision (2014), pp. 391–405. Springer, 2014.

29.

Kuang

, Yang

K.-F.

, Chen

, Li

Y-J.

, Lai Hang Chan

and Yan

, Bayes saliency-based object proposal generator for nighttime traffic images, IEEE Transactions on Intelligent Transportation Systems 19(3) (2017), 814–825.

30.

Tehrani

, Kawano

and Mita

, Car detection at night using latent filters. In 2014 IEEE Intelligent Vehicles Symposium Proceedings, (2014), pp. 839–844. IEEE.

31.

Girshick

, Donahue

, Darrell

and Malik

, Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2014), pp. 580–587.

32.

Lin

T.-Y.

, Dollar

and Girshick. Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, (2017), pp. 2117–2125.

33.

Cai

and Vasconcelos

, Cascade r-cnn: Delving into high quality object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, (2018), pp. 6154–6162.

34.

Redmon

and Farhadi

, Yolo9000: better, faster, stronger. In Proceedings of the IEEE conference on computer vision and pattern recognition, (2017), pp. 7263–7271.

35.

Redmon

and Farhadi

, Yolov3: An incremental improvement. arXiv preprint arXiv:1804.02767, 2018.

36.

Tan

, Pang

and Le

Q.V.

, Efficientdet: Scalable and efficient object detection. arxiv 2019. arXiv preprint arXiv:1911.09070.

37.

Duan

, Bai

, Xie

, Qi

, Huang

and Tian

, Centernet: Keypoint triplets for object detection. In Proceedings of the IEEE International Conference on Computer Vision, (2019), pp. 6569–6578.

38.

Kong

, Sun

, Liu

, Jiang

, Li

and Shi

, Foveabox: Beyound anchor-based object detection, IEEE Transactions on Image Processing 29 (2020), 7389–7398.

39.

Tian

, Shen

, Chen

and He

, Fcos: Fully convolutional one-stage object detection. In Proceedings of the IEEE international conference on computer vision, (2019), pp. 9627–9636.

40.

Pang

, Morris

and Radha

, Clocs: Cameralidar object candidates fusion for 3d object detection. ArXiv preprint arXiv:2009.00784, 2020.

41.

, Ang

Marcelo H.

, Karaman

and Rus

, A general pipeline for 3d detection of vehicles. In 2018 IEEE International Conference on Robotics and Automation (ICRA), (2018), pp. 3194–3200. IEEE.

42.

, Zeng

, Huang

, Hua

X-S.

and Zhang

, Structure aware single-stage 3d object detection from point cloud. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, (2020), pp. 11873–11882.

43.

Geiger

, Lenz

and Urtasun

, Are we ready for autonomous driving? the kitti vision benchmark suite. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, (2012), pp. 3354–3361. IEEE.

44.

, Zhu

, Fu

C-W.

, Qin

and Heng

P.-A.

, Direction-aware spatial context features for shadow detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (2018), pp. 7454–7462.

45.

Bell

, Zitnick

C.L.

, Bala

and Girshick

, Inside-outside net: Detecting objects in context with skip pooling and recurrent neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, (2016), pp. 2874–2883.

46.

Lin

T-Y.

, Goyal

, Girshick

, He

and Dollar

, Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, (2017), pp. 2980–2988.

47.

Newell

, Huang

and Deng

, Associative embedding: End-to-end learning for joint detection and grouping. In Advances in neural information processing systems, (2017), pp. 2277–2287.

48.

Nakagami

, The m-distributiona general formula of intensity distribution of rapid fading. In Statistical methods in radio wave propagation, (1960), pp. 3–36. Elsevier.

49.

Chen

D-Y.

, Lin

Y-H.

and Peng

Y-J.

, Nighttime brake-light detection by nakagami imaging, IEEE Transactions on Intelligent Transportation Systems 13(4) (2012), 1627–1637.

50.

Wang

, Gong

, Xing

, Huang

and Hu.

, Rdsnet: A new deep architecture forreciprocal object detection and instance segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, (2020), pp. 12208–12215.

51.

O’Malley

, Jones

and Glavin

, Rear-lamp vehicle detection and tracking in low-exposure color video for night conditions, IEEE Transactions on Intelligent Transportation Systems 11(2) (2010), 453–462.

52.

Chen

Y-L.

, Nighttime vehicle light detection on a moving vehicle using image segmentation and analysis techniques, WSEAS Trans Comput 8(3) (2009), 506–515.

53.

Wang

, Sun

and Guo

, A region tracking-based vehicle detection algorithm in nighttime traffic scenes, Sensors 13(12) (2013), 16474–16493.

54.

Sharif Razavian

, Azizpour

, Sullivan

and Carlsson

, Cnn features off-the-shelf: an astounding baseline for recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp. 806–813.

55.

, Zhang

, Ren

and Sun

, Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, (2016), pp. 770–778.

56.

Dai

, Qi

, Xiong

, Li

, Zhang

, Hu

and Wei

, Deformable convolutional networks. In Proceedings of the IEEE international conference on computer vision, (2017), pp. 764–773.

57.

, Gkioxari

, Dollar

and Girshick

, Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, (2017), pp. 2961–2969.

58.

Law

, Teng

, Russakovsky

and Deng

, Cornernet-lite: Efficient keypoint based object detection. arXivpreprint arXiv:1904.08900, 2019.

59.

Krizhevsky

, Sutskever

and Hinton

Geoffrey E.

, Imagenet classification with deep convolutional neural networks. In Communications of the ACM, (2017), pp. 84–90.

60.

Kingma

Diederik P.

and Ba

, Adam: A method for stochastic optimization. In arXiv preprint arXiv:1412.6980, 2014.

61.

Lowe

David G.

, Distinctive image features from scale-invariant keypoints, International journal of computer vision, Springer 60(2) (2004), 91–80.

62.

Bay

, Tuytelaars

and Gool

, Luc Surf: Speeded up robust features European conference on computer vision, Springer, (2006), 404–417.

63.

Ren

, Chen

, Liu

, Sun

, Pang

, Yan

, Tai

Y.-W.

and Xu

, Accurate single stage detector using recurrent rolling convolution. IEEE Conference on Computer Vision and Pattern Recognition (2017), 5420–5428.

64.

Yang

, Choi

and Lin

, Exploit all the layers: Fast and accurate cnn object detector with scale dependent pooling and cascaded rejection classifiers. IEEE conference on computer vision and pattern recognition, (2016), 2129–2137.

65.

, Shen

and Sun

, Squeeze-and-excitation networks. IEEE conference on computer vision and pattern recognition (2018), 7132–7141.

66.

Junxu,

Q. Chen

, Guo

and Shi

, Attention guided Context Feature Pyramid Network for Object Detection-Cao, arXiv preprint arXiv:2005.11475, 2020.

67.

Singh

and Kumar

, A novel dehazing model for remote sensing images, In Computers & Electrical Engineering, (2018), pp. 14–27(69). Elsevier.

68.

Singh

and Kumar

, Image dehazing using Moore neighborhood-based gradient profile prior, In Signal Processing: Image Communication, (2019), pp. 131–144(70). Elsevier.

69.

Kaur

, Singh

, Kumar

and Sun

, Color image dehazing using gradient channel prior and guided L0 filter, In Information Sciences, (2020), pp. 326–342(521). Elsevier.

70.

Shao

, Wei

, Shen

and Wang

, Feature Enhancement Based on CycleGAN for Nighttime Vehicle Detection, In IEEE Access. IEEE, 2020.

71.

Miao

, Liu

, Hou

, Liu

and Liu

, A Nighttime Vehicle Detection Method Based on YOLO v3, In 2020 Chinese Automation Congress (CAC), (2020), pp. 6617–6621. IEEE.

72.

Pham

T-A.

and Yoo

, Nighttime Vehicle Detection and Tracking with Occlusion Handling by Pairing Headlights and Taillights, In Applied Sciences, (2020), pp. 3986(10), 11. Multidisciplinary Digital Publishing Institute.

73.

Liu

, Wang

, Liang

T.T.

, Zhao

, Tang

and Ling

, Cbnet: A novel composite backbone network architecture for object detection, In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34 (2020), pp. 11653–11660.

74.

Carion

, Massa

, Synnaeve

, Usunier

, Kirillov

and Zagoruyko

, End-to-end object detection with transformers. In European conference on computer vision, (2020), pp. 213–229. Springer.

75.

Zhang

, Wan

, Liu

, Ji angyang

and Ye

, Learning to match anchors for visual object detection, In IEEE Transactions on Pattern Analysis and Machine Intelligence, (2021), pp. 213–229. IEEE.

Nighttime vehicle detection based on direction attention network and bayes corner localization

Abstract

Keywords

1 Introduction

3.1 Object stream

4.1 Training and inference

4.3 Results for Nighttime Vehicle Detection

4.3.1 Results of the Object Stream Proposal

Table 5 The Result of Ablation Study Pool DAP-Attention Weight DAP-IRNN Pixel Stream Pixel Stream-Bayes Accuracy Original 94.37 1 ✓ ✓ 95.12 2 ✓ ✓ 94.67 3 ✓ ✓ ✓ 94.26 4 ✓ ✓ ✓ ✓ 97.2 5 ✓ ✓ ✓ 93.38 6 ✓ ✓ ✓ 89.72

Footnotes

Acknowledgments

References

Table 5
The Result of Ablation Study

Pool DAP-Attention Weight DAP-IRNN Pixel Stream Pixel Stream-Bayes Accuracy

Original 94.37

1 ✓ ✓ 95.12

2 ✓ ✓ 94.67

3 ✓ ✓ ✓ 94.26

4 ✓ ✓ ✓ ✓ 97.2

5 ✓ ✓ ✓ 93.38

6 ✓ ✓ ✓ 89.72