DW: Detected weight for 3D object detection

Abstract

It is a generic paradigm to treat all samples equally in 3D object detection. Although some works focus on discriminating samples in the training process of object detectors, the issue of whether a sample detects its target GT (Ground Truth) during training process has never been studied. In this work, we first point out that discriminating the samples that detect their target GT and the samples that don’t detect their target GT is beneficial to improve the performance measured in terms of mAP (mean Average Precision). Then we propose a novel approach name as DW (Detected Weight). The proposed approach dynamically calculates and assigns different weights to detected and undetected samples, which suppresses the former and promotes the latter. The approach is simple, low-calculation and can be integrated with available weight approaches. Further, it can be applied to almost 3D detectors, even 2D detectors because it is nothing to do with network structures. We evaluate the proposed approach with six state-of-the-art 3D detectors on two datasets. The experiment results show that the proposed approach improves mAP significantly.

Keywords

Object detection 3D weight

1. Introduction

With the development of 3D sensors and increased requirements for 3D understanding [22], 3D recognition [24], object detection [6,10], as well as segmentation [16] have become important issues of computer vision. Many applications such as auto pilot [26,27], robotics [26–28], and augmented reality [28], are heavily dependent on 3D understanding [3–5,7,8,11,15,17,20,23,25,29]. 3D object detection is usually the first and crucial step, as well as a basic problem for these tasks [22]. Some works have gained state-of-the-art 3D detection performance of point cloud or RGB-D images with advances in convolutional neural network (CNN) and deep learning techniques. In this work, we aim to improving the 3D object detection performance by assigning different weights to different samples during training process.

During most of training procedures of 3D detectors, massive samples are generated. Usually, these samples can be divided into two categories: positive samples and negative samples. If the IoU (Intersection over Union) between a sample and its target GT (Ground Truth) box is up to a specific thresh value, the sample is a positive sample. Otherwise, it is a negative sample. No matter positive samples or negative samples, usually only part of them is selected to participate in training because of limited computing resources and capabilities. Usually, this is implemented by assigning weight 0 to the unselected samples and weight 1 to the selected ones. For a selected negative sample, only its classification needs to be trained because it does not overlap enough with and isn’t assigned any GT box. For a selected positive sample, both its classification and bounding box regression need to be trained because it is assigned a GT as its target.

In inference, 3D Object detectors output many detections composed of classification and 3D bounding box, some of which detects a GT and others don’t detect any GT. A prediction must meet two conditions before it can be judged as a detected one: (1) its classification must be correct, i.e., the score of the correct class must be the highest among all classes; (2) the IoU between its 3D bounding box and a GT box must be up to a threshold, such as 0.25, 0.5, 0.7, and so on, depending on different datasets or scenes. Similarly, the samples generated during the training process can also be judged as detected or undetected. In detail, the samples meeting the two conditions above are detected, and others are undetected. The detected samples indicate that the detector has been able to detect their target GT correctly and precisely, and the undetected ones indicate that the detector has not been able to detect their target GT. The contribution of the detected samples makes the score of the correct class higher, and the IoU with their target GT box higher. However, to some extent, these improvement does not improve object detection performance because they have been able to detect their target GT. And the contribution of the undetected samples is relatively more important because they are unable to detect their target GT. Consequently, the detected samples should contribute less to the detector, on the contrary, the undetected samples should contribute more.

Based on the analysis above, we proposed an approach that assigns small weights to the detected samples, and large weights to the undetected samples during the training process of 3D detectors. The proposed approach is named as DW (Detected Weight), which makes detected samples contribute less and the undetected samples contribute more. As a result, more GT can be correctly and precisely detected by the detectors, and object detection performance can be improved. For the first time, a sample’s weight is computed based on whether it is detected, which differs from existing methods and is the main contribution of this work.

The rest of this paper is organized as the following: in Section 2, we describe related works on 3D object detection and the approaches of assigning different weights to different samples; Section 3 describes our approach in detail; in Section 4, the experiments are described and the results are analyzed; finally, our work is concluded in Section 5.

2. Related work

The main task of 3D object detection is localizing and recognizing objects in a 3D scene [14]. Some works have achieved state-of-the-art performances in 3D object detection [19]. In this section, we describe related works on 3D object detection. In addition, the approaches of assigning different weights to different samples are also described.

VoteNet is proposed as an end-to-end network combining deep point set networks with Hough voting [15]. By the voting mechanism, new points that are near to objects centers are generated, grouped, and finally aggregated to generate box proposals. In detail, the input point cloud is passed through a backbone network, then a set of seed points are sampled. And votes are generated from the sampled seed points to reach object centers. Finally, vote clusters emerge near object centers, and are aggregated to generate box proposal. ImVoteNet [14] is proposed on the basis of VoteNet. It is specialized for RGB-D scenes by combining 2D votes in images with 3D votes in point clouds. Both geometric and semantic features are extracted from the 2D images, and lifted to 3D with camera parameters. And a multi-towered network structure is introduced to balance the information from 2D and 3D sources, and fully utilize the 2D and 3D features. ImVoxelNet [19] is a fully convolutional method on the basis of posed monocular or multi-view RGB images. Different from available detectors using multi-view inputs only for inference, ImVoxelNet uses multi-view inputs for both inference and training. A voxel representation of the 3D space is constructed, and used to detect objects in indoor as well as outdoor scenes. This is implemented by choosing between an indoor and outdoor head with the same meta-architecture. And final predictions are generated from 3D feature maps. Part-aware and aggregation neural network (Part-A2 net) [21] is proposed based on PointRCNN. It consists of two stages: the part-aware stage and the part-aggregation stage. The former is used to predict good quality 3D proposals and precise intra-object part locations. The latter is used to reevaluate the box and refine the location of the box. MVX-Net [22] proposed PointFusion and VoxelFusion, which are two simple and effective early-fusion approaches in order to combine the RGB with point cloud modalities based on VoxelNet architecture. SECOND [25] is an improved sparse convolution method which is proposed to increases the speed of both training and inference significantly. A new form of angle loss regression as well as a new data augmentation approach are also proposed. The former aims to improve the orientation estimation performance. And the latter aims to enhance the convergence speed and performance.

The method of assigning different weights to different samples has been widely used in the training of object detectors. As mentioned before, selected samples are assigned weight 1 and the unselected samples are applied weight 0 during the training process. The contribution of a bounding box ( ${BB}_{i}$ ) is: $\begin{array}{r} (1) & w_{i} CE (p_{s}) \end{array}$ Here $w_{i}$ is weight, CE() is cross-entropy loss, $p_{s}$ is the predicted probability for the ground-truth class, and $w_{i} \in {0, 1}$ . In some works, $w_{i} = [0, 1]$ , i.e. the weight is used to adjust the contribution of a sample [13]. A straightforward method is to use constant coefficients. For example, YOLO [18] halves the loss of examples from the background class, i.e. $wi = 0.5$ if sample i is a background sample. Focal Loss [12] assigns more weight to the hard examples dynamically as following: $\begin{array}{r} (2) & w_{i} = {(1 - p_{s})}^{γ} \end{array}$ Thus, harder examples are promoted because lower ps implies a larger error. And $γ = 2$ is demonstrated to be a good trade-off between hard examples and easy ones [12]. However, these methods above are only useful for classification task.

Gradient Harmonizing Mechanism (GHM) [9] is a counting-based approach, which suppresses gradients generated by negatives and easy positives. The loss of a sample is suppressed if its gradient is similar norm with many samples’ gradient norm as the following: $\begin{array}{r} (3) & w_{i} = \frac{1}{G ({BB}_{i}) / m} \end{array}$ Here $G ({BB}_{i})$ is the count of samples whose gradient norm is similar with that of ${BB}_{i}$ ; and m is the number of bounding boxes which are in the batch. GHM is useful for both classification task and regression task. PrIme Sample Attention (PISA) [2] promotes the negatives with larger foreground classification scores. First, the examples for each class are ranked based on their IoU or classification score. Then the normalized rank of sample i, $r_{i}$ , is calculated as following: $\begin{array}{r} (4) & \overline{r_{l}} = \frac{N_{max} - r_{i}}{N_{max}} \end{array}$ Here $N_{max}$ is the maximum number of samples over all classes in the batch. And the weight of i is calculate as: $\begin{array}{r} (5) & w_{i} = {((1 - β) r_{i} + β)}^{γ} \end{array}$ Here β is used to adjust the contribution of the normalized rank, and γ is also a modulating factor. However, these methos handle classification task and regression task separately, instead of treating the two tasks of a sample as a whole, don’t consider the issue of whether a sample detects its target GT, and aren’t widely discussed and used.

3. Detected weight

In this section, we firstly describe the motivation to propose DW. Then we describe how to calculate a weight of a sample depending on whether it detects its GT, as well as how to combine the weight with different loss function of variant 3D detector.

3.1. Motivation

As mentioned before, a sample must meet two conditions before it can be judged as a detected one: (1) its classification must be correct, i.e., the score of the correct class must be the highest among all classes; (2) the IoU between its 3D bounding box and its target GT box must be up to a threshold. The detected samples indicate that they have been able to detect their target GT correctly and precisely. In contrast, the undetected ones indicate that they are still unable to detect their target GT. The contribution of the former makes the score of the correct class higher, and the IoU with their target GT boxes higher. However, to a certain degree, these improvements are unable to improve object detection performance because they have been able to detect their target GT correctly and precisely. In contrast, the contribution of the latter is relatively more important because they are still unable to detect their target GT. Therefore, the former should contribute less to the detector, and the latter should contribute more to the detector. However, most available methods don’t consider this issue, and don’t distinguish detected samples and undetected samples during training process. Although Focal Loss [12] assigns more weight to the hard examples dynamically based on the predicted probability for the ground-truth class [13], it only considers classifications, and does not consider whether a sample can detect its target GT, i.e, it is unable to consider whether classification is correct, and meanwhile IoU is up to threshold.

In this work, we consider the issue of whether a sample can detect its target GT during training process for the first time, and proposed a novel weight approach named as DW (Detected Weight), which assigns small weight to detected samples and large weight to undetected samples. During the process of training, many boxes are proposed by 3D detectors. Then the IoU between the proposed boxes and GT boxes are computed. And the proposed boxes having high IoU with a certain GT box are selected and used as samples, as shown in Fig. 1. In the figure, the green boxes are GT boxes, and the red boxes are the bounding boxes of samples. As can be seen in sub figure (a), the left sample’s bounding box overlap enough with its target GT box, and its classification is also correct, i.e. the sample detects its target GT correctly and precisely. In contrast, the right sample’s bounding box does not overlap enough with its target GT box and its classification is also incorrect, i.e. the sample does not detects its target GT. Sub figure (b) is the result if we assign the same weight 1 to the two samples in this iteration. As can be seen in sub figure (b), the left sample overlaps more with its target GT box, and its classification is more correct. And the right sample overlaps more with its target GT, and its score of the correct class is higher. However, its IoU is still lower than the specific threshold, and its classification is still incorrect, i.e., it is still unable to detect its target GT. Sub figure (c) is the result if we assign a small weight to the left sample and a large sample to the right sample in this iteration. As can be seen in sub figure (c), the left sample still can detect its target GT although it’s suppressed compared to sub figure (b). However, the right sample is promoted, its IoU is up to the specific value and its classification is correct, i.e., it can detect its target GT. As a result, more GT can be detected and the performance measured in terms of mAP is improved.

Fig. 1.

Assigning small weight to detected samples and large weight to undetected samples can detect more GT, and improve mAP. Here sub figure (a) is the detections before an iteration. Sub figure (b) is the result if we assign the same weight 1 to the two samples in this iteration. And sub figure (C) is the result if we assign a small weight to the left sample and a large weight to the right sample in this iteration.

3.2. Calculation of detected weight

For each selected sample, we firstly judge whether its classification is correct, and calculate its IoU with its target GT box. If its classification is correct and its IoU is up to specific thresh value, then it is judged as a detected one. Otherwise, it is judged as an undetected one. For sample i, we calculate its weight as following: $\begin{array}{r} (6) & w_{i} = α e^{1 - β w c_{i} w i_{i}} \end{array}$

Here α and β are modulating factors used to adjust the weight, ${wc}_{i}$ represents classification weight, and ${wi}_{i}$ represents IoU weight. ${wc}_{i}$ is calculated as following: $\begin{array}{r} (7) & {wc}_{i} = \{\begin{array}{ll} c_{argmax (i)}, & argmax (i) = index_g t (i) \\ 0, & argmax (i) \neq index_g t (i) \end{array} \end{array}$

Here $c_{argmax (i)}$ is the maximum of all classes, $argmax (i)$ is the index of the maximum, index_gt(i) is the index of ground-truth class. ${Wi}_{i}$ is calculated as following: $\begin{array}{r} (8) & w i_{i} = \{\begin{array}{ll} {iou}_{i} - i_{t}, & {iou}_{i} - i_{t} > 0 \\ 0, & {iou}_{i} - i_{t} ⩽ 0 \end{array} \end{array}$

Here ${iou}_{i}$ is the IoU between the bounding box of sample i and its target GT box, and $i_{t}$ is the specific threshold value of IoU.

As can be seen from the formulas above, if i is an undetected sample, i.e, its classification is incorrect or its IoU is lower than specific threshold, then ${wc}_{i} = 0$ or ${wi}_{i} = 0$ , and $w_{i} = α e$ . On the contrary, if i is a detected sample, then ${wc}_{i} > 0$ , ${wi}_{i} > 0$ , and $w_{i} < α e$ . Thus, the undetected samples contribute more than the detected ones. Furthermore, for detected sample i, the greater ${wc}_{i}$ or ${wi}_{i}$ is, the smaller $w_{i}$ is. That is to say, the barely detected samples contribute more than the confidently detected samples.

We set $α = 0.5$ because $w_{i} = e$ if i is an undetected sample, $w_{i} < e$ even $w_{i} < 2$ if i is a detected sample. Thus, we estimate that the average of $w_{i}$ is about 2, and set $α = 0.5$ to keep the average of all items roughly unchanged. In Fig. 2, we assume that $i_{t} = 0.25$ . Although the actual situation of wc is very complicated, we simplify the problem and assume that $argmax (i) = index_gt (i)$ when $c_{argmax (i)} > 0.25$ . Then we increase β from 0.5 to 3.0. As can be seen in the figure, the weight dynamically decreases as wi and wc increase once the sample detects its GT. And the greater β, more sharply the weight decreases.

Fig. 2.

The weight dynamically decreases as wi and $w_{c}$ increase once the sample detects its GT. And the greater β, more sharply the weight decreases. Here $β = 0.5$ , 1.0, 2.0 and 3.0 in sub figure (a), (b), (C) and (d) respectively.

3.3. Combination of weight

Due to different network structures, available 3D detectors have their specific loss functions. These loss functions usually involve many terms, some of which are not directly used to predict 3D bounding boxes, such as the classification loss or semantic loss for each point, vote loss, etc. Consequently, we only combine DW with the terms that are directly used to predict 3D bounding boxes, such as classification loss and regression loss for bounding box, classification or regression loss for direction of bounding box, and so on. For these terms, we directly multiply them by DW. In this sense, DW can be integrated with available weight, such as focal loss. Further, DW is nothing to do with network structures of 3D detectors. Thus, it can be applied to almost 3D detectors, even 2D detectors.

4. Experiments

In this section, we evaluate DW by experiments. As mentioned before, the proposed weight can be applied to almost 3D detectors. Thus, we use indoor dataset Sun RGBD and outdoor dataset KITTI to carry out experiments. The following 6 state-of-the-art 3D detectors are adopted as our baselines: VoteNet [15], ImVoteNet [14], MVXNet [22], Part-A2 [21], SECOND [25] as well as ImVoxelNet [19]. We compare and analyze their original performance and the performance achieved by DW. Additionally, parameters selection as well as ablation studies are carried out, and qualitative results are also shown.

4.1. Datasets

Sun RGBD is an RGB-D dataset, which is indoor, monocular and for 3D scene understanding [14]. It consists of 37 object categories, more than 10k RGB-D images as well as their annotation.

KITTI is a challenging outdoor 3D detection benchmark [30], including 7k training samples and 7k test samples that have images and lidar point clouds. The training samples are further divided into train split and val split, nearly 4k images each. It consists 3 object categories: car, pedestrian and cyclist.

4.2. Baselines and implementation

We apply DW to the selected detectors, train and test them with the two datasets above respectively. Concretely, indoor detectors VoteNet and ImVoteNet are trained and tested with Sun RGBD, outdoor detectors MVXNet, Part-A2, SECOND as well as ImVoxelNet are trained and tested with KITTI, as shown in Table 1.

Table 1
Detectors selected to carry out experiment

Detectorst Dataset Indoor Outdoor

VoteNet Sun RGBD ✓

ImVoteNet Sun RGBD ✓

MVXNet KITTI ✓

Part-A2 KITTI ✓

SECOND KITTI ✓

ImVoxelNet KITTI ✓

Detectorst	Dataset	Indoor	Outdoor
VoteNet	Sun RGBD	✓
ImVoteNet	Sun RGBD	✓
MVXNet	KITTI		✓
Part-A2	KITTI		✓
SECOND	KITTI		✓
ImVoxelNet	KITTI		✓

Each selected detector has its specific loss function. As mentioned before, we only combine DW with the terms that are directly used to predict 3D bounding boxes, such as classification loss and regression loss for bounding box, classification or regression loss for direction of bounding box, and so on. To implement DW, we adopt and modify the code of MMDetection3D [1] on GitHub, which is based on PyTorch. We find that VoteNet and ImVoteNet have exactly the same loss function, and use the same code to calculate it in MMDetection3D. Thus, we study the terms of the loss function, and determine which terms need to be combined with DW and which items do not, as shown in Table 2, in which the terms needed to combine are marked with ✓, and the terms unneeded to combine are marked with ×. MVXNet, Part-A2, SECOND and ImVoxelNet have exactly the same loss function, and use the same code to calculate it in MMDetection3D. We determine which terms need to be combined with DW and which terms do not, as shown in Table 3.

Table 2

Combination of the loss function for VoteNet and ImVoteNet

Objectness Loss	Center Loss	Direction Class Loss	Direction Residual Loss	Size Class Loss	Size Residual Loss
✓	×	✓	✓	✓	✓

Table 3

Combination of the loss function for MVXNet, part-A2, SECOND and ImVoxelNet

Classification Loss	Bounding Box Loss	Direction Class Loss
×	✓	✓

We only add or modify about 20 lines of code for each loss function, which is very easy to implement. And the amount of calculation increased is negligible compared to the total amount of calculation for training.

4.3. Parameter selection

For the original parameters, we follow the setting of MMDetection3D. As mentioned in Section 3.2, we set $α = 0.5$ and increases β from 0.5 to 3.0. Therefore, for $β = 0.5$ , 1.0, 2.0, 3.0, we train and test VoteNet for each value respectively. The results are shown in Table 4. As can be seen in the table, the performance achieved by $β = 1.0$ is the best. Therefore, we set $β = 1.0$ in the following experiments.

Table 4
Result for different value of β

β AP@0.25 AP@0.50

0.5 60.6 37.3

1.0 60.3 38.6

2.0 60.5 37.9

3.0 59.4 36.8

β	AP@0.25	AP@0.50
0.5	60.6	37.3
1.0	60.3	38.6
2.0	60.5	37.9
3.0	59.4	36.8

As known to all, Sun RGBD has two evaluation protocols: IoU = 0.25 and IoU = 0.50. We adopt IoU = 0.25 so that more samples can be affected by DW. For KITTI, we follow its standard evaluation protocol IoU = 0.70. The values of it for different detectors are shown in Table 5.

Table 5

The values of $i_{t}$ for different detectors

Detectors	Dataset	Values of $i_{t}$
VoteNet	Sun RGBD	0.25
ImVoteNet	Sun RGBD	0.25
MVXNet	KITTI	0.70
Part-A2	KITTI	0.70
SECOND	KITTI	0.70
ImVoxelnet	KITTI	0.70

4.4. Main results

Experimental results on Sun RGBD and KITTI are shown in Table 6–8, in which the results with suffix -DW are the results achieved with DW, and the results without suffix -DW are the original results. The original results are referenced from MMDetection3D except VoteNet and ImVoteNet, which are retrained and tested by us because MMDetection3D does not provide AP@0.25 or AP@0.50 of the two detectors. Note that MVXNet, Part-A2 and SECOND are trained and tested for 3 classes of KITTI, while ImVoxelNet is only for car, which we follow MMDetection3D.

As can be seen in the three tables, DW improves 0.5–3.8 percentage in terms of mAP, 1.6 percentage on average. It shows that the effectiveness of DW is pronounced in terms of mAP, especially on Sun RGBD with the evaluation protocol IoU = 0.50. This is because DW suppresses detected samples and promotes undetected samples. Thus, more samples detect their target GT and mAP is improved.

Table 6
Results of AP@0.25 on Sun RGBD

Detectors Bed Table Sofa Chair Toilet Desk Dresser Nightstand Bookshelf Bathtub AP@0.25

VoteNet 84.6 51.2 67.5 77.9 87.0 24.8 30.3 60.5 33.5 79.3 59.7

ImVoteNet 87.6 50.3 72.7 79.3 90.7 31.8 39.7 65.9 42.5 74.2 63.5

VoteNet-DW 85.2 (+0.6) 50.8 (−0.4) 67.2 (−0.3) 77.2 (−0.7) 90.1 (+3.1) 25.7 (+0.9) 31.3 (+1.0) 62.4 (+1.9) 31.3 (−2.2) 80.7 (+1.4) 60.2 (+0.5)

ImVoteNet-DW 89.1 (+1.5) 49.8 (−0.5) 73.9 (+1.2) 79.0 (−0.3) 90.9 (+0.2) 32.8 (+1.0) 42.2 (+2.5) 67.7 (+1.8) 44.2 (+1.7) 73.9 (−0.3) 64.3 (+0.8)

Detectors	Bed	Table	Sofa	Chair	Toilet	Desk	Dresser	Nightstand	Bookshelf	Bathtub	AP@0.25
VoteNet	84.6	51.2	67.5	77.9	87.0	24.8	30.3	60.5	33.5	79.3	59.7
ImVoteNet	87.6	50.3	72.7	79.3	90.7	31.8	39.7	65.9	42.5	74.2	63.5
VoteNet-DW	85.2 (+0.6)	50.8 (−0.4)	67.2 (−0.3)	77.2 (−0.7)	90.1 (+3.1)	25.7 (+0.9)	31.3 (+1.0)	62.4 (+1.9)	31.3 (−2.2)	80.7 (+1.4)	60.2 (+0.5)
ImVoteNet-DW	89.1 (+1.5)	49.8 (−0.5)	73.9 (+1.2)	79.0 (−0.3)	90.9 (+0.2)	32.8 (+1.0)	42.2 (+2.5)	67.7 (+1.8)	44.2 (+1.7)	73.9 (−0.3)	64.3 (+0.8)

Table 7

Results of AP@0.50 on Sun RGBD

Detectors	Bed	Table	Sofa	Chair	Toilet	Desk	Dresser	Nightstand	Bookshelf	Bathtub	AP@0.50
VoteNet	50.2	19.6	47.4	56.7	63.7	4.1	16.5	39.8	6.1	40.9	34.5
ImVoteNet	56.0	17.9	49.6	54.9	64.3	6.3	16.4	42.6	7.2	43.0	35.8
VoteNet-DW	55.3 (+5.1)	22.6 (+3.0)	48.9 (+1.5)	58.0 (+1.3)	72.7 (+9.0)	26.4 (+2.3)	18.4 (+1.9)	40.3 (+0.5)	9.0 (+2.9)	51.6 (+10.7)	38.3 (+3.8)
ImVoteNet-DW	60.3 (+4.3)	20.5 (+2.6)	55.1 (+5.5)	57.5 (+2.6)	73.6 (+9.3)	6.8 (+0.5)	16.9 (+0.5)	47.0 (+4.4)	10.7 (+3.5)	42.4 (−0.6)	39.1 (+3.3)

Table 8

Results on KITTI

Detectors	Pedestrian	Cyclist	Car	mAP
MVXNet	56.8	56.0	76.9	63.2
Part-A2	55.6	70.5	79.0	68.3
SECOND	56.7	62.4	76.6	65.2
ImVoxelNet	–	–	17.3	–
MVXNet-DW	58.0 (+1.2)	57.2 (+1.2)	76.9 (+0.0)	64.0 (+0.8)
Part-A2-DW	56.0 (+0.4)	73.0 (+2.5)	79.2 (+0.2)	69.4 (+1.1)
SECOND-DW	55.6 (−1.1)	67.5 (+5.1)	78.0 (+1.4)	67.0 (+1.8)
ImVoxelNet-DW	–	–	18.0 (+0.7)	–

4.5. Ablation studies

DW involves two critical factors: classification weight its wci and IoU weight wii. First, we only consider classification weight wci. That is to say, for detected sample i, we calculate its weight as following: $\begin{array}{r} (9) & w_{i} = α e^{1 - β w c_{i}} \end{array}$

Then we only consider IoU weight wii and calculate the weight as following: $\begin{array}{r} (10) & w_{i} = α e^{1 - β w i_{i}} \end{array}$

Similar with parameters selection, we perform ablation studies with VoteNet. Table 9 shows the experimental results of ablation studies, in which the results with suffix -DW-c are the results achieved with equation (9), and the results with suffix -DW-i are the results achieved with equation (10). As can be seen in the table, the three results with evaluation protocols IoU = 0.25 are similar. However, the results with evaluation protocols IoU = 0.25 differ greatly. Concretely, the results achieved by the approach considering only IoU weight are almost the same as those achieved by the approach considering both IoU weight and classification weight. The latter is slightly better than the former. And the two approaches obviously outperform the approach considering only classification weight. That is to say, IoU weight is much more important than classification weight, and plays a major role in DW.

Table 9
Ablation results with VoteNet on Sun RGBD

Method AP@0.25 AP@0.50

VoteNet-DW 60.3 38.6

VoteNet-DW-c 60.4 35.4

VoteNet-DW-i 60.4 38.3

Method	AP@0.25	AP@0.50
VoteNet-DW	60.3	38.6
VoteNet-DW-c	60.4	35.4
VoteNet-DW-i	60.4	38.3

4.6. Qualitative results

We present some representative results that are generated by SECOND-DW in Fig. 3. As a contrast, results generated by SECOND are also shown in the figure. The 3D bounding boxes generated by SECOND-DW are marked with blue in the left sub figures, and the ones generated by SECOND are marked with green in the right sub figures. As can be seen in the figure, our proposed DW approach detects more objects, and meanwhile, false detections are avoided.

Fig. 3.

Comparison of SECOND-DW with SECOND. The 3D bounding boxes generated by SECOND-DW are marked with blue in the left sub figures, and the ones generated by SECOND are marked with green in the right sub figures. The comparison shows that DW approach detects more objects, and meanwhile, false detections are also avoided.

5. Conclusion

In this work, we point out that discriminating detected and undetected samples is beneficial to improve the performance measured in terms of mAP. Consequently, we propose a novel approach to compute and assign different weights to samples based on whether it detects its GT, which suppresses detected samples and promotes undetected samples. The approach is named as DW (Detected Weight), which is simple, low-calculation and can be integrated with available weight approaches. Finally, we perform experiments with six state-of-the-art 3D detectors on two datasets. And the experimental results show the significant effectiveness of the proposed approach is pronounced in terms of mAP. For future work, we plan to evaluate DW with state-of-the-art 2D detectors. Additionally, we plan to apply the idea to other computer vision applications, such as image classification and segmentation.

Footnotes

Acknowledgement

The author would like to thank to the editors and reviewers for their considerations, comments and suggestions.

References

https://github.com/open-mmlab/mmdetection3d.

Cao,

Chen,

C.C.

Loy and

Lin, Prime sample attention in object detection, in: CVPR, 2020, pp. 11580–11588.

Cheng,

Qiu,

Zou,

Pu and

Xue, Understanding depth map progressively: Adaptive distance interval separation for monocular 3d object detection, in: IJCNN, 2023.

Hazarika,

Vyas,

Rahmati and

Wang, Multi-camera 3D object detection for autonomous driving using deep learning and self-attention mechanism, IEEE Access 11 (2023), 64608–64620. doi:10.1109/ACCESS.2023.3288112.

Huang,

Wang,

Tang and

Sun, Boundary-aware set abstraction for 3D object detection, in: IJCNN, 2023.

Ji,

Liu and

Zhao, Monocular 3D object detection via estimation of paired keypoints for autonomous driving, Multim. Tools Appl. 81(4) (2022), 5973–5988. doi:10.1007/s11042-021-11801-3.

J.U.

Kim,

H.-I.

Kim and

Y.M.

Ro, Stereoscopic vision recalling memory for monocular 3D object detection, IEEE Transactions on Image Processing 32 (2023), 2749–2760. doi:10.1109/TIP.2023.3274479.

A.H.

Lang,

Vora,

Caesar,

Zhou,

Yang and

Beijbom, PointPillars: Fast encoders for object detection from point clouds, in: CVPR, 2019, pp. 12697–12705.

Li,

Liu and

Wang, Gradient harmonized single-stage detector, in: AAAI, 2019, pp. 8577–8584.

10.

Li,

Zhou,

Liu,

Yang,

Li and

Zhu, Detection-friendly dehazing: Object detection in real-world hazy scenes, IEEE Trans. Pattern Anal. Mach. Intell 45(7) (2023), 8284–8295.

11.

Li,

Ma,

Tang,

Zhao,

Chen and

Huang, Geo-localization with transformer-based 2D-3D match network, IEEE Robotics and Automation Letters 8(8) (2023), 4855–4862. doi:10.1109/LRA.2023.3290526.

12.

T.-Y.

Lin,

Goyal,

R.B.

Girshick,

He and

Dollár, Focal loss for dense object detection, IEEE Trans. Pattern Anal. Mach. Intell. 42(2) (2020), 318–327. doi:10.1109/TPAMI.2018.2858826.

13.

Oksuz,

B.C.

Cam,

Kalkan and

Akbas, Imbalance problems in object detection: A reviewk, IEEE Trans. Pattern Anal. Mach. Intell. 43(10) (2021), 3388–3415. doi:10.1109/TPAMI.2020.2981890.

14.

C.R.

Qi,

Chen,

Litany and

L.J.

Guibas, ImVoteNet: Boosting 3D object detection in point clouds with image votes, in: CVPR, 2020, pp. 4403–4412.

15.

C.R.

Qi,

Litany,

He and

L.J.

Guibas, Deep hough voting for 3D object detection in point clouds, in: ICCV, 2019, pp. 9276–9285.

16.

C.R.

Qi,

Su,

Mo and

L.J.

Guibas, PointNet: Deep learning on point sets for 3D classification and segmentation, in: CVPR, 2017, pp. 77–85.

17.

Qian,

Wang,

Zhuang,

Wang and

Yang, 3-D vehicle detection enhancement using tracking feedback in sparse point clouds environments, IEEE Open Journal of Intelligent Transportation Systems 4 (2023), 471–480. doi:10.1109/OJITS.2023.3283768.

18.

Redmon,

S.K.

Divvala,

R.B.

Girshick and

Farhadi, You only look once: Unified, real-time object detection, in: CVPR, 2016, pp. 779–788.

19.

Rukhovich,

Vorontsova and

Konushin, ImVoxelNet: Image to voxels projection for monocular and multi-view general-purpose 3D object detection, in: WACV, 2022, pp. 1265–1274.

20.

Shi,

Wang,

Li and

Ma, Real-time point cloud object detection via voxel-point geometry abstraction, IEEE Transactions on Intelligent Transportation Systems 24(6) (2023), 5971–5982. doi:10.1109/TITS.2023.3259582.

21.

Shi,

Wang,

Shi,

Wang and

Li, From points to parts: 3D object detection from point cloud with part-aware and part-aggregation network, IEEE Trans. Pattern Anal. Mach. Intell. 43(8) (2021), 2647–2666.

22.

V.A.

Sindagi,

Zhou and

Tuzel, MVX-Net: Multimodal VoxelNet for 3D object detection, in: ICRA, 2019, pp. 7276–7282.

23.

Song,

Wei,

Jia,

Xia,

Li and

Zhang, VP-Net: Voxels as points for 3-D object detection, IEEE Transactions on Geoscience and Remote Sensing 61 (2023), 00.

24.

Xiang,

Kim,

Chen,

Ji,

C.B.

Choy,

Su,

L.J.

Guibas and

Savarese, ObjectNet3D: A large scale database for 3D object recognition, in: ECCV, 2016, pp. 160–176.

25.

Yan,

Mao and

Li, Second: Sparsely embedded convolutional detection, Sensors 18(10) (2018).

26.

Yuan,

Song,

Bai,

Wang and

Ouyang, Temporal-channel transformer for 3D lidar-based video object detection for autonomous driving, IEEE Trans. Circuits Syst. Video Technol. 32(4) (2022), 2068–2078. doi:10.1109/TCSVT.2021.3082763.

27.

Zhang,

He,

Lv,

Jin and

Wang, Infrastructure 3D target detection based on multi-mode fusion for intelligent and connected vehicles, IEEE Access 11 (2023), 72803–72812. doi:10.1109/ACCESS.2023.3292174.

28.

Zhao,

Guo,

Xu and

Sheng, Transformer3D-Det: Improving 3D object detection by vote refinement, IEEE Trans. Circuits Syst. Video Technol. 31(12) (2021), 4735–4736. doi:10.1109/TCSVT.2021.3102025.

29.

Zheng,

Huang,

Zhao,

Hu and

Chen, DTSSD: Dual-channel transformer-based network for point-based 3D object detection, IEEE Signal Processing Letters 30 (2023), 798–802. doi:10.1109/LSP.2023.3283468.

30.

Zhou,

Sun,

Zhang,

Anguelov,

Gao,

Ouyang,

Guo,

Ngiam and

Vasudevan, End-to-end multi-view fusion for 3D object detection in LiDAR point clouds, in: CoRL, 2019, pp. 923–932.

DW: Detected weight for 3D object detection

Abstract

Keywords

1. Introduction

2. Related work

3. Detected weight

3.1. Motivation

4. Experiments

4.1. Datasets

4.2. Baselines and implementation

Table 1 Detectors selected to carry out experiment Detectorst Dataset Indoor Outdoor VoteNet Sun RGBD ✓ ImVoteNet Sun RGBD ✓ MVXNet KITTI ✓ Part-A2 KITTI ✓ SECOND KITTI ✓ ImVoxelNet KITTI ✓

Table 4 Result for different value of β β AP@0.25 AP@0.50 0.5 60.6 37.3 1.0 60.3 38.6 2.0 60.5 37.9 3.0 59.4 36.8

Table 9 Ablation results with VoteNet on Sun RGBD Method AP@0.25 AP@0.50 VoteNet-DW 60.3 38.6 VoteNet-DW-c 60.4 35.4 VoteNet-DW-i 60.4 38.3

Footnotes

Acknowledgement

References

Table 1
Detectors selected to carry out experiment

Detectorst Dataset Indoor Outdoor

VoteNet Sun RGBD ✓

ImVoteNet Sun RGBD ✓

MVXNet KITTI ✓

Part-A2 KITTI ✓

SECOND KITTI ✓

ImVoxelNet KITTI ✓

Table 4
Result for different value of β

β AP@0.25 AP@0.50

0.5 60.6 37.3

1.0 60.3 38.6

2.0 60.5 37.9

3.0 59.4 36.8

Table 9
Ablation results with VoteNet on Sun RGBD

Method AP@0.25 AP@0.50

VoteNet-DW 60.3 38.6

VoteNet-DW-c 60.4 35.4

VoteNet-DW-i 60.4 38.3