Neighbor-Aware Information Fusion for Point Cloud Classification and Segmentation

Abstract

In recent years, with the development of technologies such as computer vision, machine learning, and deep learning, as well as the popularity of large-scale data collection devices, 3D point cloud processing has become increasingly important. 3D point cloud processing can be widely used in fields such as object recognition, robot navigation, building information modeling (BIM), and urban planning. With more and more 3D point cloud data acquired, it has become a challenge for present 3D point cloud processing models to accurately and efficiently process this data. To improve the accuracy of point cloud classification and segmentation tasks, this study proposes an improved point cloud classification and segmentation model based on neighborhood aware information fusion. The model includes a Fusion Neighbor Information Feature Enhancement (FNIFE) module, which connects points in the local neighborhood and obtains the features of the current point through the feature relationships between the points in the neighborhood. By enhancing the feature expression of the point, it reduces the feature loss caused by the feature extraction operation and improves the accuracy of point cloud classification. Additionally, the model includes a Reverse Transmission of Point Features (RToPF) module, in which interpolation parameters are adjusted to ensure that the enhanced feature information can be effectively transmitted, thereby improving the accuracy and computing speed of the model. Finally, to further improve classification accuracy further, a module containing the X-Conv operator is utilized in the model to replace the max-pooling in the original network and reduce the feature loss generated during feature extraction. Comparative experiments are conducted on ModelNet40, ShapeNet, S3DIS datasets and ScanNet datasets. The experimental results show that the overall accuracy of proposed model reaches 92.4%. The average accuracy reaches 90.2% in the point cloud classification task, and the average intersection ratio reaches 84.5% in the point cloud segmentation task, achieving superior performance in classification and segmentation tasks compared with the state-of-the-art models.

Keywords

Point cloud processing feature extraction PointNet++deep learning semantic segmentation

1 Introduction

Deep learning techniques have made significant progress in image recognition tasks, such as image classification and semantic segmentation (Caron et al., 2024; Roy et al., 2023). Recently, in addition to 2D image recognition, there is an increasing interest in 3D image recognition for applications in autonomous driving (Li et al., 2023), augmented reality (Fang et al., 2023), and robotics (Bahl et al., 2023), in which LiDAR and RGB-D cameras are used as 3D sensors. The large-scale high-resolution 3D data obtained from such sensors present new challenges for deep learning techniques to accomplish image recognition tasks. For example, 1. Irregularity and disorder of point clouds 2. Difficulty in shape and structure recognition 3. Complexity of large-scale data points.

The task of classification and segmentation of 3D point clouds has been extensively studied in recent years. A 3D point cloud is a collection of unordered points in space, which is different from the sequential nature of pixels in 2D images. Therefore, traditional deep learning-based methods cannot be directly applied to 3D point clouds. To apply deep learning to 3D point cloud data, the data must be preprocessed, such as constructing a 2D projection of the point cloud data using a traditional convolutional neural network (Xiao et al., 2023) or voxelizing using a 3D volumetric neural network (Zhang et al., 2022). However, these methods lead to the loss of original information and are subject to certain limitations (Kermarrec et al., 2022; Ren et al., 2023; Zhao et al., 2023). In 2017, Qi et al. proposed PointNet (Qi et al., 2017a) model based on deep learning and PointNet++ (Qi et al., 2017b) that can be directly used to process 3D point cloud data. These methods greatly improve the accuracy and speed of point cloud processing.

However, existing point cloud feature extraction methods (such as PointNet++) mainly divide local areas by sampling and ignore the internal details of the area by feature summation, which may lead to performance degradation at the scene boundary. To solve this problem, this study proposes a new model that makes full use of the feature information of local areas to enhance the feature expression of point clouds. The model first extracts point cloud features through a neural network and further refines the local features to improve the recognition accuracy of semantic segmentation and classification tasks. In addition, in order to reduce the disorder, aggregation effect and information loss caused by maximum pooling during feature propagation, we optimize the feature propagation and extraction methods to maximize the utilization of feature information and improve classification and segmentation accuracy.

To validate whether the utilization of neighborhood information can improve the accuracy of point cloud classification and segmentation, PointNet++ network is selected as the basis for improvement. The reason for this is that it is a classic network in point cloud processing, and its network structure is relatively simple, making it easy to modify and insert new modules. Actually, many networks have been developed based on PointNet++ and its predecessor PointNet, such as PointCNN (Li et al., 2018): PointCNN is an improvement of PointNet, which introduces convolution operations to process point cloud data. Unlike traditional convolutions on regular grids, PointCNN performs convolution by defining a local neighborhood structure, thereby more effectively capturing local features and improving classification and segmentation performance. PointWeb (Zhao et al., 2019) introduces graph neural networks (GNNs) based on PointNet, representing point clouds as graph structures where points are nodes and edges represent relationships between points. GNNs are used to learn and aggregate features to more effectively capture global context information. KPConv (Thomas et al., 2019) introduces adaptive kernel points based on PointNet to implement convolution operations. Its convolution kernel can adjust the size and shape according to the distribution of point clouds to adapt to point cloud data of different shapes and densities. PointSIFT (Jiang et al., 2018) introduces the Scale Invariant Feature Transform (SIFT) based on PointNet and extends it to point cloud data. Its multi-scale feature learning and aggregation mechanism enables it to process point clouds of different scales and rotation transformations. PointRNN (Fan & Yang, 2019) is an improvement on recurrent neural networks (RNNs) in point cloud processing. It treats point clouds as sequences of points and uses RNNs to model temporal dependencies, thereby extracting spatiotemporal features and improving the performance of classification and segmentation tasks. PointCapsNet (Sun et al., 2021) is a capsule network optimized for point cloud data that can capture spatial relationships and pose information. Compared with traditional convolutional networks, PointCapsNet uses the characteristics of capsule networks for feature extraction and pose estimation, thereby improving point cloud classification and segmentation performance.

Existing improvement methods do not consider the mutual influence of point features in the neighborhood. Based on PointNet++, this paper adds a feature enhancement module that fuses neighborhood information after point cloud sampling and grouping, improves the accuracy of point cloud recognition by enhancing the feature information between point clouds in the local area, and adjusts the interpolation parameters used in the reverse migration process of point features to ensure the effective migration of point features. PointNet++ uses max pooling for feature aggregation, but the maximum value screening leads to information loss, which may cause large errors in point cloud classification and segmentation tasks. To this end, this study introduces the X-Conv operator (Li et al., 2018) to expand the convolution with a regular grid to alleviate the information loss caused by disordered points, feature aggregation and max pooling. This model is tested on ModelNet40 (Wu et al., 2015), ShapeNet (Chang et al., 2015), S3DIS (Armeni et al., 2016) and ScanNet (Dai et al., 2017) datasets. Experiments show that its performance is better than existing methods and verifies the effectiveness of domain information in point cloud tasks. The main contributions of this study are as follows:

(1)
Propose a feature enhancement module that integrates neighborhood information, and utilizes the interaction of point cloud features in local areas to enhance the point feature representation capability.
(2)
Optimize interpolation parameters during feature backpropagation to ensure the effective transmission of rich feature information and improve the semantic segmentation performance of the network.
(3)
Improve and apply the X-Conv operator to replace the maximum pooling, reduce feature loss, achieve efficient feature aggregation, and enhance point cloud recognition capabilities.

The remainder of the paper is organized as follows: Section II of this paper presents the progress of work related to the field of point cloud classification and segmentation. The proposed network model is described in detail in Section III, and comparative experiments and analysis are performed in Section IV. Finally, the conclusion is provided in Section V.
2 Related Work

2.1 Point Cloud Processing Methods Based on Traditional Techniques

For point cloud data with 3D coordinates, a simple representation method is to voxelize the data within a 3D grid structure (Zhang et al., 2022). However, as most voxels are empty, the point cloud representation is inefficient. OctNet can be used to alleviate the problem of the sparsity of voxel data (Riegler et al., 2017). However, this method may incur high memory usage due to the deep neural networks used. In addition, as voxels are discrete representations of space, this method still requires a high-resolution grid to maintain a certain feature representation capability, which consumes a large amount of memory (Lu & Rasmussen, 2012).

Another common 3D representation method is the multi-view implementation (Xiao et al., 2023), whereby point data are projected to various specific image planes in 3D space to form 2D images (Li et al., 2020). In this way, point cloud data can be processed with conventional 2D convolutions. However, this method ignores the inherent geometric relationship among the 3D points and the choice of the image plane seriously affects the results. Additionally, any regions that are occluded cannot be processed.

2.2 Point Cloud Processing Based on Deep Learning

Neighborhood feature pooling is a common deep learning method in point cloud processing. It reduces computational complexity by aggregating neighborhood features to the center point (Shu et al., 2022). Qi et al. (2017b) proposed PointNet to directly process point cloud data and introduced the T-Net spatial transformer network to align point clouds to optimize classification and segmentation. However, PointNet only extracts global features and ignores local features. To this end, Qi et al. (2017b) introduced PointNet++ to compensate for this defect by dividing local regions, but still did not consider the relationship between points. Jiang et al. (2018) designed the PointSIFT module to extract local features through multi-directional encoding to form a multi-scale representation, improve the 3D shape modeling capability, and enhance the performance of the PointNet series of models. Li et al. (2018) proposed PointCNN to reorder point clouds through X-Conv operations to make them suitable for traditional convolution. The PointWeb proposed by Zhao et al. (2019) uses the adaptive feature adjustment (AFA) module to dynamically optimize the features of each point according to its local area, further improving the feature expression capability.

Inspired by the performance of graph convolution in 2D image processing, Wang et al. (2019b) proposed DGCNN to extract local information by calculating the edge features between points and their neighborhoods. However, DGCNN has difficulty in recognizing sparse point clouds, which affects its effectiveness. Inspired by the attention mechanism, Wen et al. (2024) proposed a point cloud semantic segmentation network using local feature fusion and multi-layer attention mechanism, which improved the segmentation accuracy by utilizing the local geometric features and contextual information of the point cloud. Wang et al. (2019a) proposed Graph Attention Convolution, which captures the structural features of point clouds through adaptive convolution kernels to avoid feature pollution. In response to the influence of point cloud noise, Guo et al. proposed Point Cloud Transformer, which performs well in point cloud classification and segmentation tasks (Guo et al., 2021). Zhao et al. further applied Transformer networks to point cloud processing and built a high-performance model that surpassed traditional methods based on graph networks and sparse convolutions (Zhao et al., 2021). In recent years, in order to solve the problem of attention localization in point cloud classification and segmentation, Jing et al. proposed a graph-based attention-guided network AGNet (Jing et al., 2022) to extract local features through topological structure. Gao et al. designed a spatial deep attention network SDANet (Gao et al., 2022), which embeds a local deep attention mechanism in the MLP layer to learn neighborhood geometric relationships and capture local structures in combination with global features. Tang et al. (2022) proposed a segmentation method for scene boundaries, and evaluated the segmentation performance by creating boundary indicators and boundary learning (CBL). However, this method focuses too much on boundaries and ignores the internal features of point clouds, which needs to be improved (Tang et al., 2022)

3 Proposed Model

The proposed model is based on the PointNet++ architecture, which is a hierarchical network composed of multiple abstract layers. The improvement of proposed model mainly focuses on the set abstraction layer. Firstly, after sampling and grouping, a new X-conv module based on the existing X-conv (Li et al., 2018) is constructed. This module is used to train the X-transform matrix, which is then multiplied with the features extracted by the MLP to obtain the transformed feature matrix. Secondly, the FNIFE module is added to the X-conv module to enhance the feature relationships between points in the local neighborhood. The FNIFE module modifies the feature values of points while keeping their dimensions unchanged. Lastly, since the convolution operation is primarily used for feature aggregation, it is added to the FNIFE module to progressively integrate neighborhood information for feature aggregation. To improve the performance of the network model without increasing memory usage, the X-conv module is only used in the first set abstraction layer. The overall network framework for point cloud recognition tasks is shown in Figure 1.

3.1 Feature Enhancement Module Fusing Neighborhood Information

This section describes how to extract local point cloud features more effectively. The focus is mainly on the interaction between points in each local neighborhood, aiming to enhance the contextual information among all point pairs within the local region to improve feature expression. For a given 3D point cloud, the centroid is selected by sampling points from the farthest points. Then, a ball query (or k-nearest neighbors) is used to find the neighboring points around each centroid. For the local neighborhood containing k points, F is used to represent the point feature set, i.e., $F = F_{1}, F_{2}, \dots, F_{K}$ , where $F_{i} \in R^{C}$ and $i = 1, \dots, K$ . Here C represents the number of channels of each point feature. The goal is to obtain a representative feature $F_{o u t} \in R^{C_{O U T}}$ where $C_{O U T}$ is the number of channels of each output feature. In addition, the point feature dimension of the feature enhancement module that has been fused with neighborhood information remains unchanged, but the feature value will be adjusted and changed.

As shown in Figure 2(a), other methods, such as PointNet++, utilize local context information, but only using max pooling may not be able to aggregate the information in the local area well. However, our proposed method focuses on features. For aggregation, we focus on the interactions between points in each local neighborhood. By obtaining the contextual information between all point pairs, the representation ability is improved, as shown in Figure 2(b).

The schematic diagram of the FNIFE module is shown in Figure 3. In a given region with a feature set $F = F_{1}, F_{2}, \dots, F_{K}$ , the FNIFE module applies influence on each feature point, enabling it to learn the features of other points within the neighborhood. This establishes correlations between the various feature points within the neighborhood, thereby enhancing the point features in F by learning the contextual information of the local neighborhood.

Figure 1.

Overall Network Framework.

Figure 2.

Feature Extraction Network Diagram: (A) PointNet++ (B) FDIFE Module in our Method. The Yellow Dots Represent the Features of Integrated Regional Information, and the Red Dots are the Features of Integrated Regional Information.

Figure 3.

Schematic Diagram of FNIFE Module.

Here, $Δ$ is defined as the sum of the influences of each point feature learned from F. It is obtained as the product of the feature value difference between two points and the reciprocal of the distance between the two points.

Δ = \sum_{j - 1}^{K} \frac{1}{D_{i j}} (F_{j} - F_{i}) i \neq j

(1)

where

D_{i j}

is the Euclidean distance between the two points and

F_{j} - F_{i}

is the difference of point eigenvectors. Note that equation (1) only holds when

i \neq j

. When

i = j

Δ

is 0, which indicates that a feature has no influence on itself.

Thus, the enhanced feature $F_{i}^{'}$ for the feature $F_{i}$ is obtained,

F_{i}^{'} = F_{i} + Δ

(2)

PointNet++ uses a multilayer perceptron (MLP) and max-pooling to obtain representative features. However, this process does not consider the mutual connection of feature information between points in the local neighborhood.

3.2 Improved Point Cloud Recognition Algorithm Based on PointNet++

3.2.1 Adjust the Interpolation Parameters

In the semantic segmentation task, it is necessary to obtain the point features of all original points. Therefore, a combination of interpolation and skip connections is used to achieve reverse transfer of features. As shown in Figure 4, in the feature propagation layer, point features are propagated from point $N_{l}$ to point $N_{l - 1}$ , where $N_{l}$ , $N_{l - 1}$ are the input and output of the set abstraction layer l, respectively. Feature propagation is achieved by inserting the eigenvalue F of point $N_{l}$ at the coordinates of point $N_{l - 1}$ . Among various interpolation options, a weighted average distance based on k-nearest neighbors is used, as described in equation (3). In PointNet++, the parameter k is set to 3. A larger value of k can improve the completeness of features during backpropagation, but it increases the parameter number of the network. In practice, our experimental platform cannot support a high value of k. Therefore, after repeated validation, we selected the maximum value supported by the platform, which is k = 5. This ensures that more feature points are involved in the interpolation process and guarantees that the transmitted point features are sufficiently rich to improve the performance of the semantic segmentation network. Finally, the interpolation feature on the $N_{l - 1}$ point is connected with the point feature of the same level in the set abstraction. The process is repeated until the features are propagated to the original set of points.

f^{j} (x) = \frac{\sum_{i}^{k} w_{i} (x) f_{i}^{j}}{\sum_{i = 1}^{k} w_{i} (x)}

(3)

where

w_{i} (x) = \frac{1}{d {(x_{j} x_{i})}^{p}}, j = 1, 2, \dots, C

f_{i}^{j}

is the eigenvalue of the i point in the j channel.

Figure 4.

Semantic Segmentation Network.

3.2.2 Introduce X-Conv Module Instead of Max-Pooling

The feature aggregation method used in PointNet++ is max-pooling, which selects the maximum value of each dimension feature as the final global feature output. This leads to a high degree of information loss, which has negative effects on point cloud classification and segmentation tasks. Therefore, other methods should replace max-pooling to reduce the loss of information during the feature aggregation stage. Obviously, convolution is the best choice. For data that can be represented in regular domains, such as images, the convolution operator can extract features very efficiently, which is the key to the success of CNNs in various tasks. However, for irregular and disordered point clouds, it is difficult to use the convolution operator directly to take advantage of the spatial correlation of local point cloud data, resulting in incorrect convolution results.

Therefore, the issue of the irregularity and disorder of the point cloud should be addressed at first. The X-Conv module can effectively solve this issue. The X-Conv module consists of two parts: X-Conv and convolution operation. Among them, the X-Conv is similar to the spatial transformation network (STN) (Jaderberg et al., 2015). A multilayer perceptron for K input points coordinates $(P_{1}, P_{2}, \dots, P_{K})$ is used to learn the $K \times K$ X-transform matrix $X = M L P (P_{1}, P_{2}, \dots, P_{K})$ , which is then utilized to weight and rearrange the input features. Finally, the typical convolution is applied to this transformed feature. This process is referred to as the X-Conv operator.

$F_{1} = {(p_{1}, i, f_{1}, i) : i = 1, 2, \dots, N_{1}}$ is converted to $F_{2} = {(P_{2}, i, f_{2}, i) : f_{2}, i \in R^{C_{2}}, i = 1, 2, \dots, N_{2}}$ using X-Conv module (which is the convolution with the X-transform), where $p_{1}$ , $P_{2}$ is the D-dimensional coordinate of the point, and f is the feature corresponding to the point. $P_{2}$ is the point sampled from $p_{1}$ using the farthest point sampling method. Similar to CNN, $N_{2} < N_{1}$ , $C_{2} > C_{1}$ is also required here. With the layer-by-layer mapping, the number of points will become less and less, and the features of each point will also become more and more.

X-Conv is similar to regular convolution in that it only acts on a local area. For a point p in F, the set of points in the neighborhood of p is denoted as N. Thus, the input of each X-Conv to p is $S = {(p_{i}, f_{i}) : p_{i} \in N}$ (note that S is an unordered set here). Without loss of generality, S can be expressed as a $K \times D$ position matrix $P = (P_{1}, P_{2}, \dots, P_{K})$ , and a $K \times C_{1}$ feature matrix $F = (f_{1}, f_{2}, \dots, f_{K})$ . Therefore, there are $K \times (C_{1} + C_{δ}) \times C_{2}$ parameters of X-Conv ( $C_{δ}$ is the dimension of the high-dimensional space to which p is mapped).

4 Experiments and Results

In this section, after the datasets and experiment environments are described, the results of point cloud classification, component semantic segmentation, and scene semantic segmentation are provided. Then, to demonstrate the effect of the FNIFE module proposed in this study, T-SNE (Maaten & Hinton, 2008) (implemented using MATLAB) visualization is performed on the input and output features. After the impact of each module on network performance is analyzed, the model stability test is performed.

4.1 Datasets and Experiment Environments

To verify the effectiveness of the model proposed in this study, experiments are conducted on four widely used datasets, including ModelNet40 dataset (Wu et al., 2015) for shape classification experiment, ShapeNet dataset (Chang et al., 2015) for part segmentation experiment, S3DIS dataset (Armeni et al., 2016) and ScanNet dataset (Dai et al., 2017) for scene semantic segmentation experiment.

ModelNet40 dataset contains 12311 computer-assisted-design (CAD) models (mainly man-made) in 40 categories, in which 9843 models are used for training and 2468 models used for testing (Wu et al., 2015).

ShapeNet dataset contains 1200 shapes in 50 categories. Each category contains 24 shapes, most of which are organic with various poses, such as horses and cats (Chang et al., 2015). The training set contains 864 samples and the test set contains 336 samples. Five-fold cross-validation is used to achieve the segmentation accuracy.

S3DIS contains 6 different types of large-scale indoor scenes, distributed in three buildings, with 13 semantic categories (Armeni et al., 2016). The dataset only includes 3D point cloud data. Each point cloud contains XYZ coordinate information and RGB color information, as well as the category of each point. The experiment divides each room of the S3DIS dataset into grids with different sizes, randomly collects 4096 sample points in each grid, and traverses the entire room in steps of 0.5 m.

ScanNet dataset contains 1513 room scans reconstructed from RGB-D frames. The dataset is divided into 1201 scenarios for training and 312 scenarios for validation (Dai et al., 2017).

The experiments are performed using TensorFlow 1.4.0 with GTX 1080Ti.

4.2 Point Cloud Classification

Classification experiments are performed on point cloud data in ModelNet40 dataset. The results given by the proposed model are compared with the results of the state-of-the-art (SOTA) models (Table 1).

Table 1.
Classiﬁcation Results on ModelNet40 Dataset.

Model Input MA (%) (Qi et al., 2017a) OA(%) (Qi et al., 2017a)

PointNet (Qi et al., 2017a) Point 86.0 89.2

PointNet++ (Qi et al., 2017b) Point - 90.7

PointCNN (Li et al., 2018) Point 88.1 92.2

DGCNN (Wang et al., 2019b) Point 89.5 91.9

SRN-PointNet++ (Wen et al., 2024) Point - 91.5

MRFGAT (Li et al., 2021) Point 90.1 92.5

DGANet (Wan et al., 2021) Point 89.4 92.3

Ours Point 90.2 92.4

Model	Input	MA (%) (Qi et al., 2017a)	OA(%) (Qi et al., 2017a)
PointNet (Qi et al., 2017a)	Point	86.0	89.2
PointNet++ (Qi et al., 2017b)	Point	-	90.7
PointCNN (Li et al., 2018)	Point	88.1	92.2
DGCNN (Wang et al., 2019b)	Point	89.5	91.9
SRN-PointNet++ (Wen et al., 2024)	Point	-	91.5
MRFGAT (Li et al., 2021)	Point	90.1	92.5
DGANet (Wan et al., 2021)	Point	89.4	92.3
Ours	Point	90.2	92.4

MA represents mean per-class accuracy, which is the ratio of the number of correct classifications to the number of objects in a class (Qi et al., 2017a). OA denotes the overall accuracy, which is the ratio of the number of overall correct classifications to that of overall objects (Qi et al., 2017a).

In Table 1, the best-performing results are in bold, and the second-best results are underlined (the same notes for Tables 2, and 3). It can be observed from the experimental results that proposed model has achieved the best results in MA, and the overall accuracy of the proposed method is only 0.1% behind the first place, which is basically the same as MRFGAT in overall performance. The experimental results show that compared with PointNet++, proposed model achieves a 1.7% improvement for OA, which proves that the FNIFE module proposed in this paper is effective in improving the feature extraction capability.

Table 2.

Component Segmentation Experimental Results on ShapeNet Dataset.

Class	PointNet	PointNet++	PointCNN	DGCNN	DGANet	Ours
	(Qi et al., 2017a)	(Qi et al., 2017b)	(Li et al., 2018)	(Wang et al., 2019b)	(Wan et al., 2021)
Mean	80.4	81.9	84.6	82.3	82.7	84.5
Airplane	83.4	82.4	84.11	84.2	84.6	84.7
Bag	78.7	79.0	86.47	83.7	85.7	86.5
Car	74.9	77.3	80.83	77.1	78.5	81.1
Cap	82.5	87.7	86.04	84.4	87.8	87.2
Chair	89.6	90.8	90.62	90.9	91.0	91.4
Ear phone	73.0	71.8	79.70	78.5	77.3	78.6
Knife	85.9	85.9	88.44	87.3	87.9	88.2
Guitar	91.5	91.0	92.32	91.5	91.2	91.8
Lamp	80.8	83.7	85.31	82.9	82.4	85.9
Laptop	95.3	95.3	96.11	96.0	95.8	95.7
Motor	65.2	71.6	77.20	67.0	67.8	74.8
Mug	93.0	94.1	95.28	93.3	94.2	94.3
Pistol	81.2	81.3	84.21	82.6	81.1	84.0
Rocket	57.9	58.7	64.23	59.7	59.7	65.4
Skateboard	72.8	76.4	80.00	75.5	75.7	79.3
Table	80.6	82.6	82.99	82.0	82.0	83.1

Table 3.

Performance Comparison Under the Entire S3DIS Dataset.

Model	mIoU (Qi et al., 2017b)	OA (Qi et al., 2017a)
PointNet (Qi et al., 2017a)	47.71	78.62
PointNet++ (Qi et al., 2017b)	48.92	84.38
PointCNN (Li et al., 2018)	62.74	-
DGCNN (Wang et al., 2019b)	56.1	84.1
GACNet (Wang et al., 2019a)	62.85	87.79
PointSIFT (Jiang et al., 2018)	70.23	88.72
PointWeb (Zhao et al., 2019)	60.28	86.97
Ours	65.31	89.42

4.3 Component Semantic Segmentation

The component semantic segmentation performance of the proposed model is evaluated under the assumption that the category labels of each shape are known. The task of component semantic segmentation is to predict the part labels of each point. The normal vector of each point is used to describe the underlying shape. IoU (Intersection over Union) is used as an evaluation metric, and the average of IoU for all part categories is calculated (Qi et al., 2017b). A comparison between the proposed model and the SOA models is presented in Table 2.

The results in Table 2 show that the proposed model achieves the second-best mean IoU (mIoU) (Qi et al., 2017b) of all categories, with the highest performance in 7 of the 16 categories of the entire dataset. Figure 5 shows the visualization results. It can be observed that the various components of each category are clearly segmented, which demonstrates that proposed model is effective in component semantic segmentation.

Figure 5.

Visualization of Part Segmentation results.

4.4 Scene Semantic Segmentation

Six models are first trained using a six-fold cross-validation approach for six different large indoor scenes in S3DIS dataset, and then predictive segmentation is performed on each test set. Based on the semantic segmentation results for each region, the mIoU is computed for 13 semantic categories over the whole dataset. The results are shown in Table 3. The results show that the model proposed in this study outperforms other SOA models in terms of OA and leads the second place by 0.7%. It ranks the second in mIoU, behind PointSIFT. The results show that proposed model can effectively improve the accuracy of point cloud segmentation.

The results in Table 3 shows that the model proposed in this study outperforms other SOA models in terms of OA and leads the second place by 0.7%. It ranks the second in mIoU, behind PointSIFT. The results show that proposed model can effectively improve the accuracy of point cloud segmentation.

As shown in Figure 6, four samples from the segmentation experiment are selected for analysis. For categories with clear boundaries, such as floors, tables, walls, and chairs, the segmentation results given by the proposed model are not much different from the real segmentation results. For categories with similar features, such as whiteboards, windows, and walls, proposed model can roughly segment the object boundaries.

Figure 6.

Visualization of Point Cloud Scene Segmentation Results.

In addition, we also conducted scene semantic segmentation experiments on the ScanNet dataset to verify the effectiveness of our proposed model. Figure 7 shows the comparison results of our proposed model with PointNet and PointNet++ models. The accuracy of our proposed models is superior to the other two models, with a 0.5% lead over the PointNet++algorithm and a 9.9% higher accuracy than the PointNet algorithm. The visualization results of the experiment are shown in Figure 8. The segmentation results output by our proposed model are not significantly different from the label segmentation results, which can effectively improve the accuracy of point cloud segmentation.

Figure 7.

Performance Comparison on the ScanNet Dataset.

Figure 8.

Visualization of Scene Semantic Segmentation Results in ScanNet Dataset.

4.5 Choice of Interpolation k

The feature interpolation method is used in the feature reverse transfer module of semantic segmentation. The size of the interpolation parameter k will affect the final segmentation result. Therefore, in order to determine the most suitable k value, we conducted a detailed comparative experiment and obtained the performance of the network model in semantic segmentation under different k values. The experimental results are shown in Table 4. It can be seen that when k = 5, the performance is better than when k = 3 in the original text. In addition, it can be found that the larger the value of k, the better. When k = 7, the mIoU value is reduced. Therefore, k = 5 is selected as the interpolation parameter value in the experiment.

Table 4.
Experimental Results Table for Different Values of k (unit: %).

K 3 5 7

Ours Model 65.16 65.52 74.75

4.6 Feature Visualization

To understand the function of the feature enhancement module that integrates neighborhood information more intuitively, the input and output features of T-SNE are visualized. Ten categories of data in S3DIS dataset are selected. For each category, 100 points are randomly selected from the entire region, and the feature distribution is plotted. These features are visualized in Figure 9 regardless of whether they have been processed by the feature enhancement module or not. From left to right, this figure shows the input feature distribution map, the baseline output feature distribution map for the framework without the FNIFE module, and the output feature distribution map for the framework with the FNIFE module.

Figure 9.

Feature Visualization.

Compared with the baseline network, which uses the MLP to independently process local area features, the FNIFE module can be used to generate a more compact and representative feature representation. The points of different categories are accurately identified. The visualization results show that the FNIFE module enhances the recognition ability of proposed model.

4.7 Ablation Study

To demonstrate the impacts of the FNIFE module, RToPF module, and X-Conv module on the overall model performance, separate tests on each module are conducted. Tables 5, Tables 6, Tables 7 presents the results for the three representative datasets under the same configuration (Shape classification of ModelNet40 does not involve feature reverse transfer module, so the results of RToPF isn’t presented in Table 5).

Table 5.
Ablation Experiment Results on ModelNet40.

Model FNIFE RToPF X-Conv (Li et al., 2018) OA (Qi et al., 2017a)

PointNet++ (Qi et al., 2017b) × × × 90.7

√ × × 91.6

× × √ 92.0

√ × √ 92.4

Model	FNIFE	RToPF	X-Conv (Li et al., 2018)	OA (Qi et al., 2017a)
PointNet++ (Qi et al., 2017b)	×	×	×	90.7
√	×	×	91.6
×	×	√	92.0
√	×	√	92.4

Table 6.

Ablation Experiment Results on ShapeNet.

Model	FNIFE	RToPF	X-Conv (Li et al., 2018)	mIou (Qi et al., 2017b)
PointNet++ (Qi et al., 2017b)	×	×	×	81.9
	√	×	×	83.6
	×	√	×	82.2
	×	×	√	83.7
	√	√	√	84.5

Table 7.

Ablation Experiment Results on S3DIS.

Model	FNIFE	RToPF	X-Conv (Li et al., 2018)	mIou (Qi et al., 2017b)	OA (Qi et al., 2017a)
PointNet++ (Qi et al., 2017b)	×	×	×	57.57	84.38
	√	×	×	62.93	88.89
	×	√	×	59.38	86.21
	×	×	√	58.75	85.12
	√	√	√	65.31	89.42

Regardless of whether the task is shape classification, component semantic segmentation, or scene semantic segmentation, the addition of each module improves the model performance. Particularly, FNIFE module plays an important role in feature extraction, greatly improving the model's extraction performance. The X-Conv module and the RToPF module make relatively small improvements to the overall network performance.

4.8 Model Stability Test

This section examines the performance stability of the proposed model. By gradually reducing the density of the input point cloud, different output results are obtained to verify the impact of missing points on the performance of the proposed model. Compared with PointNet++ and PointNet, its performance is observed to decline. Different numbers of samples from the ModelNet40 dataset, including 1024, 512, 256 and 128 input samples, are used in the experiment. The experimental results are shown in Figure 10(a). As the number of input samples decreases, the accuracy of the three models decreases. When the input samples are reduced to half of the original, the accuracy of PointNet++ and our algorithm model decreases less. When the number of input samples is reduced to 1/4 of the original, the decrease in accuracy is roughly linear. With only 128 input samples, the performance of PointNet decreases significantly, while the accuracy of the proposed model remains high at 88.72%. As the number of samples decreases, the performance difference between the proposed model and PointNet++ or PointNet gradually increases. Overall, the proposed model achieves better performance at all sampling densities. This shows that the proposed model is robust to the number of input samples. This is because the proposed model fully considers the neighborhood information of the point and enhances the local feature information. In addition, we also verify the impact of noisy data on the model performance. Specifically, Gaussian noise is added to each point independently to test the stability of the model, as shown in Figure 10(b). The proposed model still performs well under the influence of noisy data.

Figure 10.

Model Stability Test.

4.9 Computational Efficiency Analysis

Table 8 summarizes the comparison of PointNet, PointNet++ and the proposed model in terms of time and space cost. The proposed model is improved based on PointNet++. Its model is smaller than the PointNet algorithm and larger than the PointNet++ algorithm. The forward transmission time is the longest. This is due to the addition of the relevant modules in this article, which increases the number of model parameters and complexity. However, in comparative experiments on three data sets, our method is better than these two models (ModelNet40 data set:Our 92.4, PointNet++ 90.7, PointNet 89.2 ; ShapeNet dataset:Our84.5,PointNet++ 81.9, PointNet 80.4; S3DIS dataset: Our 89.42, PointNet++ 84.38, PointNet 78.62) This is enough to prove that the improvement method we proposed based on PointNet++ is reasonable and effective in gaining benefits. In addition, although our method sacrifices a certain amount of computational efficiency, it gains better feature expression ability and classification and segmentation accuracy.

Table 8.
Comparative Analysis of Time and Spatial Complexity of Multiple Models.

Model PointNet PointNet++ Proposed model

Model size(MB) 40 12.6 18.5

Forward transmission(ms) 28.3 163.2 217.8

Model	PointNet	PointNet++	Proposed model
Model size(MB)	40	12.6	18.5
Forward transmission(ms)	28.3	163.2	217.8

5 Conclusion

In order to improve the performance of point cloud classification and segmentation tasks, this paper proposes a point cloud classification and segmentation model based on neighborhood aware information fusion to reduce feature loss and improve recognition performance. The model consists of three core modules: feature enhancement module, which enriches feature expression by building associations between neighborhood points; RToPF module, which optimizes feature transmission and improves information retention; X-Conv convolution operator, which reduces feature loss and improves network recognition performance. Ablation experiments verify the contribution of these modules to model performance. Comparative tests on four benchmark datasets show that the proposed model exhibits higher classification and segmentation accuracy.

However, from the perspective of the model's evaluation values, there is still a lot of room for improvement. Future research work can be deepened from the following aspects. 1. Continue to explore more effective functional relationships to improve the feature extraction of fusion domain information; 2. Related modules need lightweight design; 3. Explore a variety of optimization strategies, such as model pruning, quantization, and efficient reasoning frameworks to reduce computational overhead; 4. Explore the introduction of attention mechanisms or Transformers to achieve more efficient feature extraction. Ultimately, the performance of the model in more complex 3D scenes will be further improved.

Footnotes

Acknowledgments

This work was partially supported by the National Natural Science Foundation of China (Nos. 61871258). The authors thank the editors and anonymous reviewers for their helpful comments and suggestions.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the the National Natural Science Foundation of China, (grant number 61871258).

Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Data Availability Statement

The data used in this study are public and can be downloaded from their own website. The project's code is as follow: .

References

Armeni

Sener

Zamir

A. R.

Jiang

Brilakis

Fischer

Savarese

(2016). 3d Semantic parsing of large-scale indoor spaces. Proceedings of the IEEE conference on computer vision and pattern recognition, 1534–1543.

Bahl

Mendonca

Chen

, et al. (2023). Affordances from human videos as a versatile representation for robotics[C]. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 13778–13790.

Caron

Houlsby

Schmid

(2024). Location-aware self-supervised transformers for semantic segmentation[C]. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 117–127.

Chang

A. X.

Funkhouser

Guibas

Hanrahan

Huang

(2015). Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03012.

Dai

Chang

A. X.

Savva

, et al. (2017). Scannet: Richly-annotated 3d reconstructions of indoor scenes[C]. Proceedings of the IEEE conference on computer vision and pattern recognition, 5828–5839.

Fan

Yang

(2019). PointRNN: Point recurrent neural network for moving point cloud processing. arXiv preprint arXiv:1910.08287.

Fang

Chen

Zhang

, et al. (2023). Head-mounted display augmented reality in manufacturing: A systematic review[J]. Robotics and Computer-Integrated Manufacturing, 83(C), 102567. https://doi.org/10.1016/j.rcim.2023.102567

Gao

Lan

Wang

(2022). SDANet: Spatial deep attention-based for point cloud classification and segmentation. Machine Learning, 111(4), 1327–1348. https://doi.org/10.1007/s10994-022-06148-1

Guo

M. H.

Cai

J. X.

Liu

Z. N.

T. J.

Martin

R. R.

S. M.

(2021). Pct: Point cloud transformer. Computational Visual Media, 7(2), 187–199. https://doi.org/10.1007/s41095-021-0229-5

10.

Jaderberg

Simonyan

Zisserman

(2015). Spatial transformer networks. Advances in Neural Information Processing Systems, 28, 2017–2025. https://arxiv.org/abs/1506.02025

11.

Jiang

Zhao

(2018). Pointsift: A sift-like network module for 3d point cloud semantic segmentation. arXiv preprint arXiv:1807.00652.

12.

Jing

Zhang

Chen

Wang

(2022). AGNet: An attention-based graph network for point cloud classification and segmentation. Remote Sensing, 14(4), 1036. https://doi.org/10.3390/rs14041036

13.

Kermarrec

Yang

Czerwonka-Schröder

(2022). Classification of terrestrial laser scanner point clouds: A comparison of methods for landslide monitoring from mathematical surface approximation[J]. Remote Sensing, 14(20), 5099. https://doi.org/10.3390/rs14205099

14.

Dai

Han

, et al. (2023). Mseg3d: Multi-modal 3d semantic segmentation for autonomous driving[C]. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 21694–21704.

15.

Zhu

, et al. (2020). End-to-end learning local multi-view descriptors for 3d point clouds[C]. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 1919–1928.

16.

X. A.

Wang

L. Y.

(2021). Multiscale receptive fields graph attention network for point cloud classification. Complexity, 1–9. https://doi.org/10.1155/2021/8832081

17.

Sun

Chen

(2018). PointCNN: Convolution on Χ-transformed points. Proceedings of the 32nd International Conference on Neural Information Processing Systems, 828–838 .

18.

Rasmussen

(2012). Simplified markov random fields for efficient semantic labeling of 3D point clouds. Proceedings of 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2690–2697.

19.

Maaten

Hinton

(2008). Visualizing data using t-SNE. Journal of Machine Learning Research, 9(11), 2579–2605.

20.

C. R.

Guibas

L. J.

(2017a). Pointnet: Deep learning on point sets for 3d classification and segmentation. Proceedings of the IEEE conference on computer vision and pattern recognition, 652–660 .

21.

C. R.

Guibas

L. J.

(2017b). Pointnet++: Deep hierarchical feature learning on point sets in a metric space. Advances in neural information processing systems, 5105–5114.

22.

Ren

Guo

Wei

Guo

(2023). MFFNet: Multimodal feature fusion network for point cloud semantic segmentation. The Visual Computer, 39, 1–13. https://doi.org/10.1007/s00371-023-02907-w

23.

Riegler

Osman Ulusoy

Geiger

(2017). Octnet: Learning deep 3d representations at high resolutions. Proceedings of the IEEE conference on computer vision and pattern recognition, 3577–3586.

24.

Roy

S. K.

Deria

Shah

, et al. (2023). Spectral–spatial morphological attention transformer for hyperspectral image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 61, 1–15. https://doi.org/10.1109/TGRS.2023.3245678

25.

Shu

Sui

Zhao

, et al. (2022). Point clouds classification and segmentation based on local fusion network[C]. 2022 5th International Conference on Pattern Recognition and Artificial Intelligence (PRAI), IEEE, 460–464.

26.

Sun

Tagliasacchi

Deng

Sabour

Yazdani

Hinton

G. E.

K. M.

(2021). Canonical capsules: Self-supervised capsules in canonical pose. Advances in Neural Information Processing Systems, 34, 24993–25005.

27.

Tang

Zhan

Chen

, et al. (2022). Contrastive boundary learning for point cloud segmentation[C]. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 8489–8499.

28.

Thomas

C. R.

Deschaud

J. E.

Marcotegui

Goulette

Guibas

L. J.

(2019). Kpconv: Flexible and deformable convolution for point clouds. Proceedings of the IEEE/CVF international conference on computer vision, 6411–6420.

29.

Wan

Xie

Zeng

Yuan

Qiu

(2021). DGANet: A dilated graph attention-based network for local feature extraction on 3D point clouds. Remote Sensing, 13(17), 3484. https://doi.org/10.3390/rs13173484

30.

Wang

Huang

Hou

Zhang

Shan

(2019a). Graph attention convolution for point cloud semantic segmentation. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 10296–10305.

31.

Wang

Sun

Liu

Sarma

S. E.

Bronstein

M. M.

Solomon

J. M.

(2019b). Dynamic graph cnn for learning on point clouds. ACM Transactions On Graphics (tog), 38(5), 1–12. https://doi.org/10.1145/3326362

32.

Wen

Zhao

, et al. (2024). Point cloud semantic segmentation based on local feature fusion and multilayer attention network[J]. IET Computer Vision, 18(3), 381–392. https://doi.org/10.1049/cvi2.12255

33.

Song

Khosla

Zhang

Tang

Xiao

(2015). 3d Shapenets: A deep representation for volumetric shapes. Proceedings of the IEEE conference on computer vision and pattern recognition, 1912–1920.

34.

Xiao

Huang

Guan

, et al. (2023). Unsupervised point cloud representation learning with deep neural networks: A survey[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(9), 11321–11339. https://doi.org/10.1109/TPAMI.2023.3245678

35.

Zhang

Liao

Ding

, et al. (2022). Bridging 2D and 3D segmentation networks for computation-efficient volumetric medical image segmentation: An empirical study of 2.5 D solutions[J]. Computerized Medical Imaging and Graphics, 99, 102088. https://doi.org/10.1016/j.compmedimag.2022.102088

36.

Zhao

Jiang

C. W.

Jia

(2019). Pointweb: Enhancing local neighborhood features for point cloud processing. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 5565–5573 .

37.

Zhao

Jiang

Jia

, et al. (2021). Point transformer[C]. Proceedings of the IEEE/CVF international conference on computer vision, 16259–16268.

38.

Zhao

, et al. (2023). Deep hybrid compression network for lidar point cloud classification and segmentation. Remote Sensing, 15(16), 4015. https://doi.org/10.3390/rs15164015

Neighbor-Aware Information Fusion for Point Cloud Classification and Segmentation

Abstract

Keywords

1 Introduction

2.1 Point Cloud Processing Methods Based on Traditional Techniques

2.2 Point Cloud Processing Based on Deep Learning

3 Proposed Model

3.1 Feature Enhancement Module Fusing Neighborhood Information

3.2.1 Adjust the Interpolation Parameters

4 Experiments and Results

4.1 Datasets and Experiment Environments

4.2 Point Cloud Classification

Table 4. Experimental Results Table for Different Values of k (unit: %). K 3 5 7 Ours Model 65.16 65.52 74.75

Table 5. Ablation Experiment Results on ModelNet40. Model FNIFE RToPF X-Conv (Li et al., 2018) OA (Qi et al., 2017a) PointNet++ (Qi et al., 2017b) × × × 90.7 √ × × 91.6 × × √ 92.0 √ × √ 92.4

Table 8. Comparative Analysis of Time and Spatial Complexity of Multiple Models. Model PointNet PointNet++ Proposed model Model size(MB) 40 12.6 18.5 Forward transmission(ms) 28.3 163.2 217.8

Footnotes

Acknowledgments

Funding

Conflicting Interests

Data Availability Statement

References

Table 4.
Experimental Results Table for Different Values of k (unit: %).

K 3 5 7

Ours Model 65.16 65.52 74.75

Table 5.
Ablation Experiment Results on ModelNet40.

Model FNIFE RToPF X-Conv (Li et al., 2018) OA (Qi et al., 2017a)

PointNet++ (Qi et al., 2017b) × × × 90.7

√ × × 91.6

× × √ 92.0

√ × √ 92.4

Table 8.
Comparative Analysis of Time and Spatial Complexity of Multiple Models.

Model PointNet PointNet++ Proposed model

Model size(MB) 40 12.6 18.5

Forward transmission(ms) 28.3 163.2 217.8