DeepGCNs-Att: Point cloud semantic segmentation with contextual point representations

2.1 Pointwise MLP methods

PointNet firstly applies deep learning model directly on point cloud data. However, this model only has one max-pooling layer to integrate all sampling points’ features, leading to a weak ability to extract local information. VoxelNet [35] improves the single max-pooling layer and proposes Voxel Feature Encoding(VFE), which divides the point cloud into equally spaced three-dimensional voxels and convertes a set of points in each voxel into unified features through the VFE layer. Inspired by CNN, PointNet++ uses local structure through hierarchical feature extraction and proposes two multi-scale grouping and multi-resolution grouping strategies to ensure better feature extraction. In [36], the absolute coordinates of each point and the relative coordinates of its neighbors are used to represent the point cloud, and Group Shuffle Attention is used to obtain the relationship between points. Then, a Gumbel Subset Sampling layer with displacement invariability, differentiability and end-to-end trainability is used to train hierarchical features. PointWeb [37] is based on PointNet++, which utilizes Adaptive Feature Adjustment and the relationship between local neighbors to improve the features of points. [38] is proposed to use Structural Relational Network to learn the structural relationship characteristics between different local features.

2.2 Graph-based methods

Super Points Graph (SPG) [19] is proposed to represent the point cloud as the collection of internal connections of simple shapes, which reduces the input point cloud’s size, and finally uses PointNet to get the classification of each point. DGCNN proposes the EdgeConv [34], which integrates the information of local neighbors. Each layer builds a new K-NN graph to learn the global shape attributes gradually, and captures the potential global similar features after multiple iterations. Inspired by deep CNN, DeepGCNs proposes dilated convolutions to realize deep GCN and proposes GCNs with residual connections [16] inspired by ResNet. [39] regards point clouds as a collection of connected simple shapes and superpoints, and uses directed graphs to obtain structural and environmental information. [40] proposes a supervised framework to oversegment a point cloud into pure superpoints. In order to better obtain local geometric relations in high-dimensional space, [191] proposes Network PyramNet based on Graph Embedding Module(GEM) and Pyramid Attention Network(PAN). GEM module expresses point cloud as directed acyclic graph, and uses covariance matrix to replace Euclidean distance when constructing similarity matrix. In the PAN module, four convolution kernels of different sizes are used to extract features.

2.3 Attention-based methods

Attention modules are used to solve the problem that long-distance information will be misdetected, and grasp the key points from a large amount of data without losing features. The work [2] uses self-attention in machine translation for the first time so that each word has global semantic information and captures the long-distance dependency between features. [5] proposes the attention-based score refinement(ASR) module, which concatenates scores of neighboring points with learned weights of attention modules. [13] proposes a Dual Attention Network (DANet) to integrate global and local features, including the Position Attention Module and Channel Attention Module. The results of two attention models are fused through an addition operation, and finally, the convolutional layer outputs the classification of each point. In order to better obtain the spatial distribution of point cloud, [42] proposes the Local Spatial Aware(LSA) layer to learn the weight of spatial perception. [43] proposes the attention-based Score Refinement(ASR) module to conduct post-processing of segmentation results and modify the initial segmentation results by pooling.

Traditional convolutional neural networks are not suitable for extracting point cloud features, which are spatially disordered. So ResGCNs utilizes GCNs to represent the point cloud as an undirected graph g (ν, ɛ), where ν and ɛ represent the set of n nodes and e edges of each layer respectively. Each layer of graph convolution performs the following update and aggregation operations,

g l + 1 = F ( g l , w l ) = ρ ( φ ( g l , w l agg ) , w l update )(1)

where F () is a fixed function that performs convolution operations in each l - th layer, g _l and g_l+1 represent the input and output of each layer, φ and ρ are the aggregation function and update function.The aggregation function aggregates the features of the neighborhood of each vertex in the point cloud graph, and then uses the update function to update the representation of each node through a nonlinear transformation to learn new features, h ν l + 1 = φ ( h ν l , ρ ( { h u l | u l ∈ N ( v l ) } , h ν l , w ρ ) , w φ )(2)

where h _{ν l}  ∈ R ^D represents the features of graph’s vertex ν in the l - th layer, D is the dimension of the feature and h _{ν l+1} is the feature vector in the (l + 1) - th layer. N (v _l ) is the set of neighbor vertices within a certain distance in the l - th layer. w _ρ parametrizes the vertices h _{u l} . w _φ contains the learnable parameters of the aggregation functions. In the process of neural network training, the algorithm only updates the features of the vertices of the point cloud graph, and does not update the features of the edges, which makes the vertices only learn the features of their neighbors. Nevertheless, when vertices that are far away have similar features, the traditional GCN algorithm is likely to lose these key information during training, but ResGCNs subtract the vertex h _{ν l} from its neighbors h _{u l} to distinguish different features and use the max-pool for feature aggregation: ρ ̇ = max ( h u l - h ν l | u l ∈ N ( v l ) )(3)

where N (v _l ) is the neighbor set of the vertex h _{ν l} . With the increase in the number of network layers, dilated convolution can help deep GCNs expand the field of view of the convolution kernel without increasing the amount of calculation, so that each vertex can learn the features of farther vertices, so as to obtain global feature information. As shown in Fig. 2, our GCN Attention Backbone set each node to search for the number of neighboring points k = 16, each layer of ResGCN has f = 64 hidden layers or filters, and the dilation convolution parameter d increases layer by layer. Then, ResGCNs utilize MLP as the update function with the batch normalization to speed up gradient descent and use ReLu as the activation function to achieve nonlinear transformation. Since the algorithm using ResGCN as the GCN Backbone Block, it can be stacked in multiple layers through the residual connection, which solves the vanishing gradient and over smothing problem of the graph convolutional neural networks.

OPEN IN VIEWER

Fig. 2

Examples of the ResGCN model which utilize residual connections to alleviate the vanishing gradient problem.

At the prediction block layer, our algorithm uses the fusion of global features and local features to semantically classify the category of each point in the point cloud. As shown in Fig. 1, the prediction block first uses MLP to reduce the dimensionality of the high-dimensional features of the vertices, then uses the attention module to extract the global features, and finally obtains the semantic classification results of each point.

3.2.2 Attention prediction

Compared with the traditional method of directly using softmax on the results of the network output layer to obtain the classification of each vertex, our algorithm uses the dual attention module of the network output layer to enhance the global context information of each vertex. We compared the results of the prediction part with DeepGCNs and found that our model has fewer calculation parameters, which means that the model inference has a faster prediction speed. And we increase the output vertex information of the graph to get better classification results.

OPEN IN VIEWER

Fig. 3

The architecture of our proposed Prediction Block, which stacks MLP and dual attention module to operate feature fusion and global feature aggregation.The input feature is the N*1024 feature vector obtained by ResGCN feature extraction of N three-dimensional points in the point cloud.

Channel-wise Attention is similar in form to Spatial-wise Attention. The channel’s high-dimensional feature map in each layer’s output of ResGCN can be regarded as a map of classes, making the self-attention module utilizes the interdependence between channels to learn global feature representation. Given the feature map H ∈ R^C×N of the MLP output layer, the Channel-wise Attention module takes these feature vectors as input. Different from Spatial-wise Attention, Channel-wise Attention directly operates matrix multiplication on the transposition of H _i and H _j , and then utilizes Softmax to obtain the channel attention map V _ji : V ji = soft max i ( H j , H i )(6)

where V _ji is the impact exerted by each vertex h _{i v} in i - th channel on the vertex h _{j v} in j - th channel. The output of this module is H ̈ ∈ R C × N , which can aggregate the global feature. Then the algorithm multiplies the transpose of V _ji and H _i as a matrix, and adds the result to H _j to get H ̈ : H ̈ = ∑ i = 1 N ( V ji H i ) + H j(7) where H ̇ is the weighted sum of the dependent relationship between the channel’s high-dimensional feature map and its original feature, which represents each vertex contains the local features learned by ResGCNs and the features between the distant vertexs. Finally, the algorithm adds H ̇ and H ̈ to obtain the final classification result through nonlinear transformation. So far, the model we proposed has made full use of the rich local features in the GCN backbone block, considering the spatial and channel correlation between short-distance and long-distance vertex features and the interdependence between feature maps, which makes our algorithm can perform point cloud semantic classification more accurately in complex three-dimensional scenes.

4.1 Experiment setting

4.1.4 Training and testing process

Firstly, we trained and tested the ResGCN-Att-7 and ResGCN-Att-14 models on area 5 of the S3DIS dataset. The training curves of ResGCN-Att-7 and ResGCN-Att-14 are shown in the Fig. 5. The final accuarcies of two models are both about 0.84. When the number of epochs is about 40, the test accuracy of the model begin to oscillate, indicating that the model has converged and it will be globally optimal. So we tested all models from epoch 40 to 60 and find the final best cloud segmentation model.

OPEN IN VIEWER

Fig. 5

Visualized results of semantic segmentation.

4.1.7 Comparison with state-of-the-art networks

In order to test the overall performance of the model on the entire S3DIS data set, we performed 6-fold cross-validation, which are compared with the state-of-art models on area 5 and over 6 fold respectively. The results of OA and mIoU are shown in Table 3. In Area5, the mIoU of our model is 2.48%higher than that of ResGCN-14, which is even very similar to the results of ResGCN-28, and surpasses other existing network models. In terms of over 6 fold, our model achieves accuracy close to the original network model and surpasses the existing network model on mIoU, but has fewer parameters and faster computational speed.

OPEN IN VIEWER

Table 3

IoU segmentation results of each category of the S3DIS data set in Area5 and 6-fold

Test Area	Method	OA	mIoU
Area 5	PointNet	-	41.09
	SEGCloud	-	48.92
	RSNet	-	51.93
	PointNet++	83.43	51.98
	ResGCN-14	-	49.90
	ResGCN-28	-	52.49
	ResGCN-Att-14(Ours)	83.96	52.38
6-fold	PointNet	78.5	47.6
	MS+CU	79.2	47.8
	G+RCU	81.1	49.7
	SGPN	47.6	50.4
	A-SCN	81.6	52.7
	DGCNN	84.3	56.1
	PointNet++	-	53.2
	ResGCN-28	85.9	60.0
	ResGCN-Att-14(Ours)	83.96	57.7

We summarized each category’s IoU in Table 4. It can be seen that our proposed model has a similar IoU result to ResGCN-28 in most categories, with only a small decline, which is due to the decrease in the number of GCN layers. We replaced the fusion module of the original network layer with the dual attention module, which made the prediction module have fewer parameters. With the reduction of the GCN Backbone Block’s layers, the network model can still achieve relatively good results in semantic segmentation. Besides, in terms of area5 and over 6 fold, the semantic segmentation performance of our model is better than the state-of-art model in most categories.

OPEN IN VIEWER

Table 4

IoU segmentation results of each category of the S3DIS data set in Area5

Test Area	Method	ceiling	floor	wall	beam	column	window	door	table	chair	sofa	bookcase	board	clutter
Area 5	PointNet	88.8	97.33	69.8	0.05	3.92	46.26	10.76	52.61	58.93	40.28	5.85	26.38	33.22
	SEGCloud	90.06	96.05	69.86	0.00	18.37	38.35	23.12	75.89	70.40	58.42	40.88	12.96	41.0
	RSNet	93.34	98.36	79.18	0.00	15.75	45.37	50.10	65.52	67.87	22.45	52.45	41.02	43.64
	ResGCN-Att-14(Ours)	91.44	97.62	75.85	0.00	15.38	50.47	42.48	67.64	75.39	14.46	51.32	56.66	42.23
6fold	PointNet	88.0	97.3	69.3	42.4	23.1	47.5	51.6	42.0	54.1	9.6	38.2	29.4	35.2
	MS+CU	88.6	95.8	67.3	36.9	24.9	48.6	52.3	51.9	45.1	10.6	36.8	24.7	37.5
	G+RCU	90.3	92.1	67.9	44.7	24.2	52.3	51.2	58.1	47.4	6.9	39.0	30.0	41.9
	PointNet++	90.2	91.7	73.1	42.7	21.2	49.7	42.3	62.7	59.0	19.6	45.8	36.7	51.6
	3DRNN+CF	92.9	93.8	73.1	42.5	25.9	47.6	59.2	60.4	66.7	24.8	57.0	36.7	51.6
	ResGCN-28	93.1	95.3	78.2	33.9	37.4	56.1	68.2	64.9	61.0	34.6	51.5	51.1	54.4
	ResGCN-Att-14(Ours)	93.0	92.3	76.9	32.7	33.1	56.6	67.0	61.6	63.5	21.9	51.3	47.0	53.3

As shown in Fig. 4, the classification results of our ResGCN-Att-14 are almost identical to that of ResGCN-28. Only a few parts of the point cloud misdetected since the algorithm adopted the dual attention module, which enables the model to learn global context information autonomously in the prediction part. So we utilized fewer GCN layers to obtain classification results similar to 28 layers.

OPEN IN VIEWER

Fig. 4

The training process of ResGCN-Att-7 and ResGCN-Att-14.

We conducted ablation experiments on our proposed model using the method of control variables, which change the parameters of dilation, neighbors, depth, width, fixed k-NN, and connection respectively to verify the influence of these parameters on the model results, as shown in Table 5. Effect of dilation. Results in Table 5 show that dilated graph convolutions account for a 2.46%improvement in mean IoU, justified primarily by the expansion of the network’s receptive field. We found that if the stochastic dilation is increased from 0.2 to 0.4, it will indeed increase the mIoU of the model slightly, but if it continues to increase, a stable gain cannot be obtained. On the contrary, with the increase of the stochastic dilation parameter, the result of the model decreases rapidly. Interestingly, our results in Table 5 also indicate that dilation especially helps deep networks when combined with residual graph connections. If there is no such connection, the performance will actually decrease as the graphics convolution expands. The reason may be that these changing neighbors cause a “worse” gradient, which further hinders convergence when the residual graph connection is not used. Effect of neighbors. Results in Table 5 show that a larger number of neighbors helps in general. As the number of neighbors is decreased by a factor of 2 and 4, the performance drops by 2.53%and 3.25%respectively. However, if the network capacity is large enough, a large number of neighbors will only lead to improved performance. This becomes obvious when we increase the number of neighbors to 2 times and reduce the number of filters to 2 times. Effect of depth. Results in Table 5 show that increasing the number of layers can improve network performance, but the premise is to use residual graph connection and expansion graph convolution, as shown in Table 5. Effect of width. Results in Table 5 show that increasing the number of filters will result in performance improvements similar to increasing the number of layers. Generally speaking, higher network capacity can learn the nuances needed to succeed in extreme situations. Effect of fixed k-NN. Results in Table 5 show that if there is fixed k-NN, the result of the model will decrease, but if the dynamic k-NN is used, more computational cost is required. Effect of connection. Results in Table 5 show that residual graph connections play an essential role in training deeper networks, as they tend to result in more stable gradients. When the residual graph connections between layers are removed, performance dramatically degrades(-13.04%mIoU). As the depth of the network increases, skipping connections becomes the key to convergence.