Depth-embedded instance segmentation network for urban scene parsing

Abstract

The ubiquitous availability of cost-effective cameras has rendered large scale collection of street view data a straightforward endeavour. Yet, the effective use of these data to assist autonomous driving remains a challenge, especially lack of exploration and exploitation of stereo images with abundant perceptible depth. In this paper, we propose a novel Depth-embedded Instance Segmentation Network (DISNet) which can effectively improve the performance of instance segmentation by incorporating the depth information of stereo images. The proposed network takes binocular images as input to observe the displacement of the object and estimate the corresponding depth perception without additional supervisions. Furthermore, we introduce a new module for computing the depth cost-volume, which can be integrated with the colour cost-volume to jointly capture useful disparities of stereo images. The shared-weights structure of Siamese Network is applied to learn the intrinsic information of stereo images while reducing the computational burden. Extensive experiments have been carried out on publicly available datasets (i.e., Cityscapes and KITTI), and the obtained results clearly demonstrate the superiority in segmenting instances with different depths.

Keywords

Instance segmentation depth embedding scene parsing

1 Introduction

Accurately and expeditiously segmenting countable objects (e.g., individuals, vehicles and bicycles) of stereo imagery, a.k.a. instance segmentation in the computer vision community [7 , 53], is a core problem for autonomous driving. Conventional segmentation approaches refer to segment amorphous image regions by analysing pixels in similar spatial semantics, such as positions and texture information. With the advent of object-based detectors, segmentation tends to interpret pixels at the instance level, which reflects the more detailed discrepancies of instances and greatly promotes the development of instance segmentation. The development of instance segmentation is inseparable from object detection [12 , 42] and semantic segmentation [4 , 57]. Therefore, it not only requires to accurately localise objects but also precise instance mask prediction. With the rapid development of depth sensors in the field of autonomous driving and robotics, additional depth information has potential to become additional information and be incorporated into off-the-shelf segmentation architectures to improve instance segmentation performance, but research in this direction has not yet been explored.

Despite the numerous efforts in improving the performance of instance segmentation, most of the aforementioned methods extract features from the monocular images [55], thereby only capturing limited texture information. Concretely, with monocular vision, the distance between objects is often gauged inaccurately due to the limitation of the ability to perceive the depth from one object to another. Binocular vision can sufficiently estimate the depth between objects through their overlap and displacement, but it lacks concern in the field of instance segmentation. Inspired by the recent successful use of depth information in stereo matching scenarios, we introduce an effective method to apply the depth estimation based on the binocular vision system to the instance segmentation. Specifically, taking the binocular images as input, we first estimate the depth perceptions for the paired images and then ingeniously devise a module to calculate the cost-volume of depth disparities. Analogously, the colour cost-volume can be generated and concatenated with the obtained depth cost-volume, forming a joint cost-volume, which can be regarded as the pre-processing manipulation and fed into the standard instance segmentation architecture (i.e., Mask-RCNN [19]). In addition, we employ Siamese-style Convolutional Neural Networks (CNNs) to mutually learn the inherent semantics implicit in stereo images, and ensure that the computational cost is equivalent to that of ordinary networks based on monocular images. The contributions of our approaches can be summarised as follows:

(1) We propose a novel Depth-embedded Instance Segmentation Network (DISNet) to effectively incorporate the depth information of binocular images without introducing extra supervision information.

(2) We design a new module to compute the depth disparities of stereo images, which can be fused into the colour cost-volume and improve the performance of the instance segmentation.

(3) The shared-weights networks are employed to discover the intrinsic semantics implied in the input stereo images and avoid yielding a large number of parameters.

(4) We conduct extensive experiments on Cityscapes and KITTI datasets to show the effectiveness of our framework. We also present quantitative visualisations to demonstrate the priority of the proposed method in discovering distant objects.

The rest of this paper is organised as follows. A comprehensive investigation of related works will be given in Section 2. Then, we will illustrate and analyse the proposed framework in Section 3. Consequently, experimental results and ablation studies will be shown in Section 4. Finally, we will present a conclusion of the paper.

2 Related work

In this section, we will comprehensively review the literature related to the proposed method from three perspectives, including the RGB image-based instance segmentation methods, the image depth-based instance segmentation methods and Siamese-CNN.

RGB Image-based Instance Segmentation Instance segmentation, as a downstream task of segmentation, aims to learn the masks of the detected objects and has received considerable attention in recent years [7 , 55]. Compared with semantic segmentation, instance segmentation requires to analyse the semantic information represented by each pixel on the basis of accurately recognising all countable objects. Existing instance segmentation methods can be roughly categorised into two folds: segmentation-based methods and proposal-based methods. Segmentation-based methods usually follow a two-stage pipeline, namely, generating a segmentation map of the input image and then distinguishing each object instance. A representative work is the Deep Watershed Transform (DWT) [2] which combines deep learning on the basis of classical watershed transform to generate high-quality energy maps. InstanceCut [25] is another meaningful work, which learns the boundaries of all instances derived from the instance-aware edge detection model.

Proposal-based methods generally refer to the use of object detectors to obtain region proposals, and then estimate appropriate masks for the detected outputs. Early methods relied heavily on hand-made features and classifiers, such as Random Forests [29] or Support Vector Machine (SVM) [17]. Recently, with the advent of deep learning, fully convolutional neural network has gradually replaced traditional methods and becomes the dominant framework for semantic segmentation [7]. Proposed an end-to-end framework that cascades a multi-task network to implement multiple functions simultaneously, including distinguishing instances, determining masks, and classifying objects [28]. Proposed the first differentiable fully convolutional semantic segmentation framework, which inherits the advantages of FCN and instance mask proposals. Mask R-CNN [19] extended the function of Faster R-CNN [42] to generate a pixel-wise mask prediction for each object by aggregating segmented mask branches.

Image Depth-based Instance Segmentation Stereo images are inspired by simulating human binocular vision, which can be captured by a pair of cameras with precise viewing angles and have been widely applied in various research fields, including stereo matching [35], depth estimation [22] and optical flow [50]. Compared with monocular images, binocular images capture depth information as auxiliary features to complete dense results, which benefits to identify two or more objects that share the same or similar geometric positions. Gupta et al. [15] proposed a geocentric embedding method for depth images, which encodes the height above the ground and the angle with gravity for each pixel (excluding horizontal disparity). Then, Uhrig et al. [46] leveraged FCN to predict semantic labels, depth and an instance-based encoding using the direction of each pixel towards its corresponding instance centre. Besides, model distillation was applied to transfer supervision between images from different modalities [16], including distilling supervision from labelled RGB images to unlabelled depth and optical flow. More recently, Kendall et al. [23] were committed to achieving better segmentation results by performing multiple tasks jointly and appropriately weighing each task. However, the aforementioned methods require ground-truth labels for both depth and semantics and involve multi-stage training.

Siamese-CNN The idea of Siamese network was originally proposed by Bromley et al. [3] and was widely used to learn the similarity metric after 2005 [5]. Then, the era of deep learning brings about Siamese-CNN by more efficient convolution fashion. By allowing weights to be shared in two identical networks, Siamese-CNN has been widely applied to solve the problems of computer vision tasks, including learning a general similarity function for comparing image patches [51], obtaining the discriminative appearance-based tracklet affinity models for track multi-object [47] and employing in stereo matching [35].

3 Proposed approach

In this section, we will depict the proposed DISNet in detail. As shown in Fig. 1, our DISNet includes the process of calculating depth and RGB cost-volumes, as well as a backbone network for instance-level segmentation. The pipeline of the entire architecture will be displayed in 3.1. Then, it will introduce the detailed information of each module in 3.2 and 3.3, respectively. In 3.5, it will describe how we design the final objective function.

Fig. 1

Overall architecture of our DISNet. Our DISNet takes binocular images as input, and generates the corresponding cost-volumes based on the estimated depth and RGB information. The obtained cost-volumes will be concatenated and fed into the ongoing backbone to produce the segmentation results at the instance-level.

3.1 Overall architecture

As illustrated in Fig. 1, the proposed architecture (i.e., DISNet) consists of four parts, which are the network used to yield the RGB cost-volume, the depth generation and embedding module, and the segmentation module. Specifically, the RGB cost-volume is generated by following the method introduced in GC-Net [24]. However, the colour cost-volume of monocular images is extremely susceptible to the interference of occluded objects (caused by the single shooting angle of the camera) and objects with similar colours (due to the close proximity of adjacent objects in unstructured scenes).

To tackle the above problems, we explored the way to distinguish the objects with similar colours, and then proposed two adjacent modules (namely, depth generating module and depth embedding module) to extract depth disparity from binocular images. Compared with the latest segmentation scheme that only uses monocular images [14], the binocular image pair is synthesized from the left-view image and the corresponding right-view image, in which contains extensive depth disparities. Effective use of these depth disparities can make up for the lack of object geometry information and benefit to distinguish these objects in identical colour. This motivates us to introduce delegate modules to calculate the depth cost-volume from the stereo image.

Once the depth cost-volume is obtained, it will be concatenated with the RGB cost-volume and sent to the segmentation network to predict the contours of the instances. The optimisation of the entire network tends to search the optimal solution that satisfies the best balance between the disparities and segmentation.

3.2 Depth generating

The depth generation process of DISNet is rendered in blue in Fig. 1. The deep generation is inspired by the structure expressed in GC-Net [24]. Specifically, the input left-view and right-view images are processed by a feature extractor similar to ResNet18 [20]. The dimension of the resulting feature is $\frac{1}{2} H \times \frac{1}{2} W \times F$ where H and W are the height and width of feature maps, F is the number of features and equals to 32. The generated features are subsequently applied to produce the cost-volumes to reflect the adjustment of the left-view and right-view images.

In order to obtain the cost volume from the left-view to the right-view, it needs to move to the right along the epipolar line of the left tensor until the maximum disparity is reached. The left-view and right-view features are concatenated in an pixel-wise manner and yield a 4D cost volume (the dimension is $\frac{1}{2} D \times \frac{1}{2} H \times \frac{1}{2} W \times 2 F$ , where D is the largest disparity). Analogously, the cost volume from the rightview to the left-view can be formed by moving to the left along the outer-polar line of the right tensor until reaching the maximum gap. The generated depth disparity costs are processed by the 3D convolution and deconvolution bottlenecks, and are accompanied by the execution of feature comparison and stereo matching algorithms. Specifically, each bottleneck includes a multi-scale encoder that performs multi-resolution comparison and a jump-connection decoder that captures various information at different resolutions. In order to prevent extra parameters from being generated, it is allowed to share the weights of the left and right bottlenecks during training. After the last decoding layer, the left and right tensors (with dimension of D×H×W×1) are generated by upsampling layers, which incorporates the adjustment cost of the pixels in the paired images.

Once the above process is complete, the natural way to determine the optimal gap for each pixel is to apply the differential Soft-argmax [36] to the adjustment cost. However, the original Soft-argmax function considers returning the index of the maximum value of the entire tensor, which will cause greater errors when processing pixel-level data. To alleviate the impact of the above function, we implemented a multi-layer argmax function based on multi-layer convolution to generate a single value for each pixel. In addition to the multi-layer argmax function, it also requires Sigmoid activation to accurately estimate the pixel gap in actual training.

After obtaining the learned left depth map, we refer to the method introduced in [40] to transform it to the right depth map. Especially, the transformation process needs to abide by the following left and right consistency loss [14] $L_{lr}^{l} = \frac{1}{N} \sum_{i, j} | d_{i, j}^{l} - d_{i, j + d_{i, j}^{l}}^{r} |$ (1) where N represents the total number of pixels meeting the predefined disparity range. d^l and d^r denote the predicted depth map for the left and right images, respectively.

3.3 Depth embedding

After extracting the depth features from the stereo image, we immediately considered how to effectively integrate the obtained information into the existing network. To achieve this goal, we designed a module of depth cost-volume by imitating the calculation process of the RGB cost-volume. Specifically, this module is constructed by a series of convolution and deconvolution layers (see the yellow part in Fig. 1). Furthermore, a Siamese-style CNN is employed to reduce the computational cost and the number of model parameters.

3.4 Module of detection and segmentation

The depth cost-volume generated from the above process will be concatenated with the conventional RGB cost-volume to form a joint representation that can be regarded as the input of detection and segmentation network. To show the effectiveness of our method, we select the network of Mask R-CNN [19] as backbone architecture. Precisely, it involves two functionalities, including a Region Proposal Network (RPN) and a mask classifier. In RPN, it usually defines 9 anchor boxes to generate multiple regions of interest (RoI) and a lightweight binary classifier to predict the confidential scores of detected objects. Non-maximum suppression is another well-known operation applied to anchor points with high objective scores. The RoI Align network outputs multiple bounding boxes and adjusts them to a fixed size. Then, it will extract the features in the bounding box and feed them into the fully connected layer, and accompanied by the softmax function for classification. The regression models are required to further improve the quality of bounding box prediction. Besides, the extracted features are also sent to a mask classifier composed of two CNNs, and a binary mask is output for each RoI. By doing so, the mask classifier allows the network to generate a mask for each category without interference between categories.

3.5 Objective function

The objective function adopted by our method consists of two parts, one of which aims to regularise the accuracy of the calculated disparity, and the other is the segmentation loss function in the Mask R-CNN [19]. Considering that the ground truth of disparity map is available, we choose a smooth $L_{1}$ loss to regularise the process of generating the left depth. $L_{disp} = \frac{1}{N_{mask}} \sum_{i = 1}^{N_{mask}} {smooth}_{L 1} (d_{i} - {\hat{d}}_{i})$ (2) where N_mask represents the total number of pixels that satisfy the predefined disparity range. d_i denotes the pixel from the ground truth map and $\hat{d_{i}}$ indicates the pixel of prediction map.

The loss function introduced in Mask R-CNN $L_{m}$ can be viewed as another part of the overall objective function. According to the paper of Mask R-CNN [19], it aims to simultaneously optimise the classification score $L_{cls}$ , the regression scores $L_{box}$ and $L_{mask}$ of ROI and segmentation respectively, $L_{m} = L_{cls} + L_{box} + L_{mask}$ (3) where, $L_{cls}$ , $L_{box}$ and $L_{mask}$ have the same functions as those in K. He’s Paper [19]. We introduce two hyper-parameters λ_d and λ_m to determine the domination of disparity and segmentation during back-propagation. Formally, it can be written as, $L = λ_{d} L_{disp} + λ_{m} L_{m}$ (4) The regularisation of the stereo depth will inevitably increase time consumption, but it brings a significant improvement in the segmentation results and is summarised in the experimental section.

4 Experimental results and analysis

In this section, detailed descriptions of experimental datasets and the evaluation metrics will be presented. Subsequently, we will introduce implementation details of proposed network. Obtained results will be compared with extensive baselines and state-of-the-art methods. In addition, we will offer quantitative visualisations and ablation studies to comprehensively evaluate the priority of the proposed framework.

Datasets Our experiments are conducted on two publicly available datasets, called Cityscapes datasets [6] and KITTI dataset [11], respectively.

The dataset Cityscapes [6] is a large-scale datasets released for tasks related to urban scene understanding and autonomous driving. It provides rectified stereo image pairs and the corresponding disparity maps pre-calculated by the SGM algorithm[21]. It contains 5,000 high-quality maps for both the left view images which are finely annotated at the pixel level, as well as corresponding right view which are not annotated. These images are split into three non-overlapping sets of 2975, 500, and 1525, which are used for training, validating, and testing, respectively. Eight semantic categories are selected from 19 formally annotated classes to evaluate and report the performance of instance-level segmentation, including rider, car, bus, truck, train, motorcycle and bicycle. Since the depth data is not directly accessible in the Cityscapes dataset, we employ the method introduced in [37] to preprocess the depth map.

KITTI dataset [11] is an outdoor dataset, including 32 training scenes and 29 test scenes by using the Eigen split [10]. The RGB images and depth values are captured by car-mounted stereo cameras and the rotating Velodyne laser scanner, respectively. We evaluated our method with 3,712 training images and reported the performance of 120 validation images and 144 test images split by [41]. In addition, for disparity images, we refer the dedicated work of [37].

Metrics The appropriate measurement method for instance segmentation should consider the degree of refinement of the bounding box. To this end, we exploit average accuracy (AP) to evaluate the performance of our method, and present the results with a COCO-style evaluation matrix [30]. The result of AP can be obtained by: $AP = \frac{TP}{TP + FP}$ . Concretely, it needs to calculate the division of prediction objects and ground truth provided by the official dataset for each category. The final result is equal to adding the values of all classes and dividing by the total number of classes.

Implementation Details We implement our DISNet through the open-source Pytorch [38]. Our framework follows the process of Mask R-CNN [19] and replaces monocular images with paired stereo images to discover the extensive implied depth cues [49]. The model was trained on a single PC equipment with a graphic card of GeForce GTX 1080Ti. For disparity generation, we adopt the original network in GC-Net [24] and the maximum disparity is set as 192. We have retained the model architecture and parameters of Mask R-CNN [19]. In the training phase, we first use pre-trained configures with ImageNet [9] to initialize the ResNet [20] network. The initial learning rate is set to 0.025 and it decays by 10^- 4 every 5 epochs. Due to the limitation of graphical memory, the batch size for Cityscapes is 1 (i.e., a pair of stereo images) and the model has been trained for 64 epochs. By default, the number of sample RoIs is 512. We adopted SGD optimiser [43] with momentum of 0.9 to train the network and the weight decay is set to 0.0005.

4.1 Comparison with baselines and state-of-the-art methods

To show the effectiveness of the proposed method, we compared two variants of our methods with state-of-the-art methods in Table 1. As can be observed,the mean average precision on the DISNet(Mask R-CNN) outperforms the original one by 6.7% and the UPSNet of our method outperforms the original UPSNet [49] by 4.4% in terms of the mean average precision metric. Specifically, the segmentation result has improved significantly in most categories, including the categories of person, train, truck, motorcycle and bicycle. These improvements undoubtedly prove the effectiveness of exploiting the depth information of stereo image pairs.

Table 1
Overall AP and AP₅₀ and precision of each class on Cityscapes validation set. The replicated results are indicated by *

Method(%) AP AP₅₀ person rider car bus train truck motorcycle bicycle

SAIS [18] 17.4 36.7 14.6 12.9 35.7 16.0 23.2 19.0 10.3 7.8

DWT [2] 19.4 35.3 15.5 14.1 31.5 22.5 27.0 22.9 13.9 8.0

DIN [1] 20.0 38.8 16.5 16.7 25.7 20.6 30.0 23.4 17.1 10.1

SGN [32] 25.0 44.9 16.6 9.1 40.0 29.8 46.6 21.1 7.4 6.1

Mask R-CNN [19] 26.0 51.3 26.2 18.6 46.0 23.5 40.6 23.1 16.0 13.7

DISNet (Mask R-CNN) 32.7 60.4 34.4 28.2 53.6 28.8 49.3 26.6 17.0 23.7

UPSNet [49] 32.8 60.3 30.6 24.9 50.5 34.1 52.4 33.5 17.6 18.9

DISNet (UPSNet) 37.2 65.5 36.7 31.0 56.0 34.6 58.7 34.4 21.9 24.7

Method(%)	AP	AP₅₀	person	rider	car	bus	train	truck	motorcycle	bicycle
SAIS [18]	17.4	36.7	14.6	12.9	35.7	16.0	23.2	19.0	10.3	7.8
DWT [2]	19.4	35.3	15.5	14.1	31.5	22.5	27.0	22.9	13.9	8.0
DIN [1]	20.0	38.8	16.5	16.7	25.7	20.6	30.0	23.4	17.1	10.1
SGN [32]	25.0	44.9	16.6	9.1	40.0	29.8	46.6	21.1	7.4	6.1
*Mask R-CNN [19]	26.0	51.3	26.2	18.6	46.0	23.5	40.6	23.1	16.0	13.7
DISNet (Mask R-CNN)	32.7	60.4	34.4	28.2	53.6	28.8	49.3	26.6	17.0	23.7
*UPSNet [49]	32.8	60.3	30.6	24.9	50.5	34.1	52.4	33.5	17.6	18.9
DISNet (UPSNet)	37.2	65.5	36.7	31.0	56.0	34.6	58.7	34.4	21.9	24.7

To obtain an intuitive understanding of our models, we have visualised the segmented images obtained through our DISNet and state-of-the-art methods (i.e., Mask R-CNN [19] and UPSNet [49]) in Fig. 2 and Fig. 3, respectively. From the qualitative results illustrated above, it is not difficult to realize that our method can correctly segment objects even in complex scenes. Note that the algorithm accurately detects more objects than compared methods owing to depth information providing more accurate guiding role. In particular, notice in the second row of Fig. 2 that we produce masks more smoothly for the cars on right part of the image, and the third row of same figure that our network recognizes the bicycle under the person which Mask-RCNN has failed. Moreover in the last row person and bicycles that have not detected by other methods are clearly recognized in our results. Similarly, our method successfully finds out many objects that are ignored by original UPSNet, such as the person on right part of scene in last row of Fig. 3.

Fig. 2

Visualisation results on Cityscapes val set. From left to right, each column of the image represents the input images, the ground-truths, the results of Mask R-CNN [19] and our DISNet (Mask R-CNN), respectively. Best viewed in colour.

Fig. 3

Visualisation results on Cityscapes val set. From left to right, each column of the image represents the input images, the ground-truths, the results of UPSNet [19] and our DISNet (UPSNet), respectively. Best viewed in colour.

4.1.1 Experiments on KITTI dataset

Since the annotations provided by the official KITTI dataset cannot be directly converted to the format corresponding to the COCO style, we only display the experimental results through qualitative illustrations. From Fig. 4, we can see that most objects have been detected and assigned the correct mask. Especially, rear objects in images with similar colours can also be distinctly classified.

Fig. 4

Qualitative visualisations on KITTI dataset [11]. From the left to right, each column of the image represents the input RGB images, the ground truth of instance train id and our results.

4.2 Ablation study

4.2.1 Depth generation module

We also investigated the impact of whether to import the generated depth module on the performance of the experiment. From Fig. 5, we found that learning depth from stereo images by introducing an independent network can significantly improve the segmentation effect. Variants of Mask R-CNN and UPSNet based on our DISNet can bring more than 4% improvement. This indicates that the depth map obtained by the convolutional network can make better use of the depth information and have a positive impact on the segmentation results.

Fig. 5

Effectiveness of adding depth generating module.

4.2.2 Depth cost-volume

To further illustrate the effectiveness of proposed DISNet, we designed an ablation study to investigate whether Depth Cost-volume is employed during training. The results of the ablation study are summarised in Table 2. As can be seen in Table 2, there are notable improvements when we adopt Depth Cost-volume in the Mask R-CNN and UPSNet architecture. For Mask R-CNN, the Average Precision is relatively improved by 6.1%, and the Average Precision₅₀ is improved by 11%. The improvements also can be seen on UPSNet, where the Average Precison and Average Precision₅₀ are improved by 3.8% and 4.1%, respectively. It is worth noting that UPSNet has made several major modifications to Mask R-CNN, changing the internal structure of the layer used to generate the instance Mask, therefore its degree of improvement is not as good as Mask R-CNN.

Table 2
Performance with or without the depth cost-volume, where GDC represents for Generate Depth Cost-volume

Method GDC AP AP₅₀

DISNet (Mask R-CNN) False 26.6 51.4

DISNet (Mask R-CNN) True 32.7 60.4

DISNet (UPSNet) False 33.4 61.4

DISNet (UPSNet) True 37.2 65.5

Method	GDC	AP	AP₅₀
DISNet (Mask R-CNN)	False	26.6	51.4
DISNet (Mask R-CNN)	True	32.7	60.4
DISNet (UPSNet)	False	33.4	61.4
DISNet (UPSNet)	True	37.2	65.5

4.2.3 Hyper-parameters

Since the objective function we introduced earlier is composed of two parts, different weights will affect the dominance of different parts during model optimisation, thereby affecting the final result. In order to better choose the appropriate weights for each part, we designed ablation experiments for hyper-parameters. As the results shown in Table 3, the best results are achieved with setting the values of λ_d and λ_m to 0.4 and 0.6, respectively. This indicates that when the detection module loss takes major role in loss term, the results can be better.

Table 3
Comparison of accuracy using different hyper-parameters in the loss function

λ_d λ_m AP AP₅₀

0.3 0.7 26.3 51.0

0.4 0.6 26.6 51.4

0.5 0.5 26.4 51.0

0.6 0.4 26.1 50.6

0.7 0.3 25.8 50.2

λ_d	λ_m	AP	AP₅₀
0.3	0.7	26.3	51.0
0.4	0.6	26.6	51.4
0.5	0.5	26.4	51.0
0.6	0.4	26.1	50.6
0.7	0.3	25.8	50.2

4.2.4 Backbones

Taking into account the characteristics of different networks, we provide a comparison of segmentation results using three mainstream network structures. From Table 4, we can see that the best accuracies achieved by utilizing the ResNeXt [20]. The reason is that the ResNeXt architecture replaces the standard residual block with the "split-transform-merge" strategy used in the Inception model [45]. The input of the block is projected into a series of low-dimensional representations. Before merging the results, it applies several convolution filters separately instead of performing convolution on the entire input feature map.

Table 4
Comparison of accuracy using different backbone networks

Backbone AP AP₅₀

ResNet-50 [20] 26.4 51.0

ResNet-101 [20] 26.6 51.4

ResNeXt [48] 26.8 51.8

Backbone	AP	AP₅₀
ResNet-50 [20]	26.4	51.0
ResNet-101 [20]	26.6	51.4
ResNeXt [48]	26.8	51.8

5 Conclusions

In this paper, we have proposed a novel network architecture that takes advantages of the depth information extracted from stereo images to improve the quality and accuracy of instance segmentation. To this end, we have introduced a new module for computing the depth cost-volume, which can be integrated with the colour cost-volume to jointly capture useful disparities of stereo images. Experiments conducted on the challenging datasets suggested that our method can outperform the baselines and state-of-the-art methods in different metrics. Shortly, we will explore ways to deploy our framework to tasks related to 3D instance segmentation.

Acknowledgment

This work was supported in part by National Natural Science Foundation of China under the Grant Number 61872187, 62077023 and 62072246, in part by Natural Science Foundation of Jiangsu Province under the Grant Number BK20201306, and in part by the “111 Program” under Grant No. B13022.

References

Arnab

and Torr

P.H.S.

, Pixelwise instance segmentation with a dynamically instantiated network, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (2017), 441–450.

Bai

and Urtasun

, Deep watershed transform for instance segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (2017), 5221–5229.

Bromley

, Guyon

, LeCun

, Säckinger

and Shah

, Signature verification using a"siamese” time delay neural network. In Advances in neural information processing systems, (1994), 737–744.

Chen

L.-C.

, Papandreou

, Kokkinos

, Murphy

and Yuille

A.L.

, Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs, IEEE transactions on pattern analysis and machine intelligence 40(4) (2017), 834–848.

Chopra

, Hadsell

and LeCun

, Learning a similarity metric discriminatively, with application to face verification, In 2005 IEEE Computer Society Conference on Computer Vision andPattern Recognition (CVPR’05), volume 1, pages 539–546. IEEE, (2005).

Cordts

, Omran

, Ramos

, Rehfeld

, Enzweiler

, Benenson

, Franke

, Roth

and Schiele

, The cityscapes dataset for semanticurban scene understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (2016), 3213–3223.

Dai

, He

and Sun

, Instance-aware semantic segmentation via multi-task network cascades, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (2016), 3150–3158.

Danielczuk

, Matl

, Gupta

, Li

, Lee

, Mahler

and Goldberg

, Segmenting unknown 3d objects from real depth images using mask rcnn trained on synthetic data, In 2019 International Conference on Robotics and Automation (ICRA), 7283–7290. IEEE, (2019).

Deng

, Dong

, Socher

, Li

L.-J.

, Li

and Fei-Fei

, Imagenet: A large-scale hierarchical image database, In 2009 IEEE conference on computer vision and pattern recognition, 248–255. Ieee, (2009).

10.

Eigen

, Puhrsch

and Fergus

, Depth map prediction from a single image using a multi-scale deep network, In Advances in neural information processing systems, (2014), 2366–2374.

11.

Geiger

, Lenz

, Stiller

and Urtasun

, Vision meets robotics: The kitti dataset, The International Journal of Robotics Research 32(11) (2013), 1231–1237.

12.

Girshick

, Fast r-cnn. In Proceedings of the IEEE international conference on computer vision, (2015), 1440–1448.

13.

Girshick

, Donahue

, Darrell

and Malik

, Rich feature hierarchies for accurate object detection and semantic segmentation, In Proceedings of the IEEE conference on computer vision and pattern recognition, (2014), 580–587.

14.

Godard

, Aodha

O.M.

and Brostow

G.J.

, Unsupervised monocular depth estimation with left-right consistency, InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (2017), 270–279.

15.

Gupta

, Girshick

, Arbeláez

and Malik

, Learning rich features from rgb-d images for object detection and segmentation, In European conference on computer vision, 345–360. Springer, (2014).

16.

Gupta

, Hoffman

and Malik

, Cross modal distillation for supervision transfer, In Proceedings of the IEEE conference on computer vision and pattern recognition, (2016), 2827–2836.

17.

Hai-xiang

X.U.

, Guang-xi

, Jin-wen

, Xiang

and Fu-yuan

, Image segmentation based on support vector machine, Journal of Electronic Science and Technology 3(3) (2005), 226–230.

18.

Hayder

, He

and Salzmann

, Boundary-aware instance segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages (2017), 5696–5704.

19.

, Gkioxari

, Dollar

and Girshick

, Mask r-cnn, In The IEEE International Conference on Computer Vision (ICCV), Oct (2017).

20.

, Zhang

, Ren

and Sun

, Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, (2016), 770–778.

21.

Hirschmuller

, Stereo processing by semiglobal matching and mutual information, IEEE Transactions on pattern analysis and machine intelligence 30(2) (2007), 328–341.

22.

Kamencay

, Breznan

, Jarina

, Lukac

and Zachariasova

, Improved depth map estimation from stereo images based on hybrid method, Radioengineering 21(1) (2012).

23.

Kendall

, Gal

and Cipolla

, Multi-task learning using uncertainty to weigh losses for scene geometry and semantics, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (2018), 7482–7491.

24.

Kendall

, Martirosyan

, Dasgupta

, Henry

, Kennedy

, Bachrach

and Bry

, End-to-end learning of geometry and context for deep stereo regression. In Proceedings of the IEEE International Conference on Computer Vision, (2017), 66–75.

25.

Kirillov

, Levinkov

, Andres

, Savchynskyy

and Rother

, Instancecut: from edges to instances with multicut, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (2017), 5008–5017.

26.

L.-H.

, Qian

, Jing

, Zheng

W.-N.

and Zhou

Y.-F.

, Study on semantic image segmentation based on convolutional neural network, Journal of Intelligent & Fuzzy Systems 33(6) (2017), 3397–3404.

27.

, Chen

, Zhu

, Xie

, Huang

, Du

and Wang

, Attention-guided unified network for panoptic segmentation, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (2019), 7026–7035.

28.

, Qi

, Dai

, Ji

and Wei

, Fully convolutional instance-aware semantic segmentation, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (2017), 2359–2367.

29.

Liaw

, Wiener

, et al., Classification and regression by randomforest. R news, 2(3) (2002), 18–22.

30.

Lin

T.-Y.

, Maire

, Belongie

, Hays

, Perona

, Ramanan

, Dollár

and Zitnick

C.L.

, Microsoft coco: Common objects in context, In European conference on computer vision, pages 740–755. Springer, (2014).

31.

Liu

, Zhang

, Shao

and Yang

, Builtin depth semantic coupled encoding for scene parsing, vehicle detection and road segmentation, IEEE Transactions on Intelligent Transportation Systems, (2021), 5520–5534.

32.

Liu

, Jia

, Fidler

and Urtasun

, Sgn: Sequential grouping networks for instance segmentation, In Proceedings of the IEEE International Conference on Computer Vision (2017), 3496–3504.

33.

Long

, Shelhamer

and Darrell

, Fully convolutional networks for semantic segmentation, In Proceedings of the IEEE conference on computer vision and pattern recognition, (2015), 3431–3440.

34.

Luo

, Xu

J.-H.

and Zhang

S.-Y.

, Decompose image into meaningful regions based on contour detector and watershed algorithm, Journal of Intelligent & Fuzzy Systems 32(6) (2017), 4259–4271.

35.

Luo

, Schwing

A.G.

and Urtasun

, Efficient deep learning for stereo matching, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (2016), 5695–5703.

36.

Luvizon

D.C.

, Tabia

and Picard

, Human pose regression by combining indirect part detection and contextual information, Computers & Graphics 85 (2019), 15–22.

37.

Pang

, Sun

, Ren

J.S.J.

, Yang

and Yan

, Cascade residual learning:Atwo-stage convolutional neural network for stereo matching, In Proceedings of the IEEE International Conference on Computer Vision, (2017), 887–895.

38.

Paszke

, Gross

, Chintala

, Chanan

, Yang

, DeVito

, Lin

, Desmaison

, Antiga

and Lerer

, Automatic differentiation in pytorch. In Advances in neural information processing systems Workshop, (2017).

39.

Qadir

H.A.

, Shin

, Solhusvik

, Bergsland

, Aabakken

and Balasingham

, Polyp detection and segmentation using mask r-cnn: Does a deeper feature extractor cnn always perform better? In 2019 13th International Symposium on Medical Information and Communication Technology (ISMICT), 1–6. IEEE, (2019).

40.

Ramirez

P.Z.

, Poggi

, Tosi

, Mattoccia

and Di Stefano

, Geometry meets semantics for semi-supervised monocular depth estimation, In Asian Conference on Computer Vision, 298–313. Springer, (2018).

41.

Ren

and Zemel

R.S.

, End-to-end instance segmentation with recurrent attention, In Proceedings of the IEEE conference on computer vision and pattern recognition, (2017), 6656–6664.

42.

Ren

, He

, Girshick

and Sun

, Faster r-cnn: Towards real-time object detection with region proposal networks, In Advances in neural information processing systems, (2015), 91–99.

43.

Ruder

, An overview of gradient descent optimization algorithms, arXiv preprint arXiv:1609.04747, (2016).

44.

Shang

, Wu

, Meng

and Xu

, Instance segmentation by learning deep feature in embedding space, In 2019 IEEE International Conference on Image Processing (ICIP), pages 2444–2448. IEEE, (2019).

45.

Szegedy

, Liu

, Jia

, Sermanet

, Reed

, Anguelov

, Erhan

, Vanhoucke

and Rabinovich

, Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, (2015), 1–9.

46.

Uhrig

, Cordts

, Franke

and Brox

, Pixel-level encoding and depth layering for instance-level semantic labeling, In German Conference on Pattern Recognition, 14–25. Springer, (2016).

47.

Wang

, Wang

, Shuai

, Zuo

, Liu

, Chan

K.L.

and Wang

, Joint learning of convolutional neural networks and temporally constrained metrics for tracklet association, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, (2016), 1–8.

48.

Xie

, Girshick

, Dollár

, Tu

and He

, Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, (2017), 1492–1500.

49.

Xiong

, Liao

, Zhao

, Hu

, Bai

, Yumer

and Urtasun

, Upsnet: A unified panoptic segmentation network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (2019), 8818–8826.

50.

, Ranftl

and Koltun

, Accurate optical flow via direct cost volume processing, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (2017), 1289–1297.

51.

Zagoruyko

and Komodakis

, Learning to compare image patches via convolutional neural networks, In Proceedings of the IEEE conference on computer vision and pattern recognition (2015), 4353–4361.

52.

Zhang

, Chen

, Zhang

, Wang

, Lu

and Yang

, When visual disparity generation meets semantic segmentation: A mutual encouragement approach, IEEE Transactions on Intelligent Transportation Systems 22(3) (2020), 1853–1867.

53.

Zhang

, Zhang

, Lu

, Shao

and Yang

, Targettargeted domain adaptation for unsupervised semantic segmentation, In International Conference on Robotics and Automation, 13560–13566. IEEE, (2021).

54.

Zhang

, Fidler

and Urtasun

, Instance-level segmentation for autonomous driving with deep densely connected mrfs, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (2016), 669–677.

55.

Zhang

, Schwing

A.G.

, Fidler

and Urtasun

, Monocular object instance segmentation and depth ordering with cnns, In Proceedings of the IEEE International Conference on Computer Vision, (2015), 2614–2622.

56.

Zhao

, Shi

, Qi

, Wang

and Jia

, Pyramid scene parsing network, In Proceedings of the IEEE conference on computer vision and pattern recognition, (2017), 2881–2890.

57.

Zhou

, Zhang

, Long

, Shao

and Yang

, Depth embedded recurrent predictive parsing network for video scenes, IEEE Transactions on Intelligent Transportation Systems 20(12) (2019), 4643–4654.

Depth-embedded instance segmentation network for urban scene parsing

Abstract

Keywords

1 Introduction

2 Related work

3 Proposed approach

3.2 Depth generating

3.4 Module of detection and segmentation

3.5 Objective function

4.1 Comparison with baselines and state-of-the-art methods

4.2.1 Depth generation module

Table 2 Performance with or without the depth cost-volume, where GDC represents for Generate Depth Cost-volume Method GDC AP AP50 DISNet (Mask R-CNN) False 26.6 51.4 DISNet (Mask R-CNN) True 32.7 60.4 DISNet (UPSNet) False 33.4 61.4 DISNet (UPSNet) True 37.2 65.5

Table 3 Comparison of accuracy using different hyper-parameters in the loss function λ d λ m AP AP50 0.3 0.7 26.3 51.0 0.4 0.6 26.6 51.4 0.5 0.5 26.4 51.0 0.6 0.4 26.1 50.6 0.7 0.3 25.8 50.2

Table 4 Comparison of accuracy using different backbone networks Backbone AP AP50 ResNet-50 [20] 26.4 51.0 ResNet-101 [20] 26.6 51.4 ResNeXt [48] 26.8 51.8

Acknowledgment

References

Table 2
Performance with or without the depth cost-volume, where GDC represents for Generate Depth Cost-volume

Method GDC AP AP₅₀

DISNet (Mask R-CNN) False 26.6 51.4

DISNet (Mask R-CNN) True 32.7 60.4

DISNet (UPSNet) False 33.4 61.4

DISNet (UPSNet) True 37.2 65.5

Table 3
Comparison of accuracy using different hyper-parameters in the loss function

λ_d λ_m AP AP₅₀

0.3 0.7 26.3 51.0

0.4 0.6 26.6 51.4

0.5 0.5 26.4 51.0

0.6 0.4 26.1 50.6

0.7 0.3 25.8 50.2

Table 4
Comparison of accuracy using different backbone networks

Backbone AP AP₅₀

ResNet-50 [20] 26.4 51.0

ResNet-101 [20] 26.6 51.4

ResNeXt [48] 26.8 51.8