Increasing the localization accuracy of visual SLAM with semantic segmentation and motion consistency detection in dynamic scenes 1

Abstract

Visual Simultaneous Localization and Mapping (SLAM) is one of the key technologies for intelligent mobile robots. However, most of the existing SLAM algorithms have low localization accuracy in dynamic scenes. Therefore, a visual SLAM algorithm combining semantic segmentation and motion consistency detection is proposed. Firstly, the RGB images are segmented by SegNet network, the prior semantic information is established and the feature points of high-dynamic objects are removed; Secondly, motion consistency detection is carried out, the fundamental matrix is calculated by the improved Random Sample Consistency (RANSAC) algorithm, the abnormal feature points are output by the epipolar geometry method, and the feature points of low-dynamic objects are eliminated by combining the prior semantic information. Thirdly, the static feature points are used for pose estimation. Finally, the proposed algorithm is tested on the TUM dataset, the algorithm in this paper reduces the average RMSE of ORB-SLAM2 by 93.99% in highly dynamic scenes, which show that the algorithm can effectively improve the localization accuracy of the visual SLAM system in dynamic scenes.

Keywords

Simultaneous localization and mapping (SLAM)semantic segmentation motion consistency detection dynamic feature points

1 Introduction

Simultaneous Localization and Mapping (SLAM) can be described as a robot that estimates its own motion and establishes an environment map during the movement process in an unknown environment, according to the data returned by the sensors it carries. It is visual SLAM due to its visual sensor [1].

Nowadays, there are many mature solutions for visual SLAM, such as RGB-D SLAM [2] based on depth camera, LSD-SLAM [3] based on direct method and ORB-SLAM2 [4, 5] based on feature point method. ORB-SLAM2 is currently considered to be one of the best solutions based on the feature point method, and its additional use of the relocation method ensures higher localization accuracy. However, most algorithms such as ORB-SLAM2 are based on static scenes for tracking and localization, which is not suitable for dynamic scenes. The movement of dynamic objects in the scene may cause the camera to produce relative motion to the moving objects, thus making the accuracy of localization poor. Therefore, it is especially important to realize accurate localization of visual SLAM in dynamic scenes.

This paper proposes a visual SLAM algorithm based on semantic segmentation and motion consistency detection. The combination of the two methods enhances the visual SLAM system’s ability to understand the scene, and effectively eliminate dynamic feature points, and only use static feature points for pose estimation, which can effectively improve the localization accuracy of the visual SLAM system in dynamic scenes. The rest of this paper is structured as follows: the second part is related work, the third part is the system detail of the proposed algorithm, the fourth part is the experimental results, and the fifth part is the conclusion.

2 Related work

At present, the more commonly used method is to remove the dynamic region of the image in the front-end visual odometry, and it only uses the information of the static region to calculate the transformation relationship between poses [6]. Li et al. proposed a static weighting method, which utilized the weight to represent the possibility of the depth edge point being a static point, and combined the strength-assisted iterative closest point algorithm to complete the inter-frame registration, it reduced the impact of dynamic objects on the system localization accuracy [7]. Wei et al. applied the geometric constraint method to determine the sparse feature points, and then combined the color and depth information to segment the dynamic region, and eliminated the feature points in the region [8]. Dai et al. used the Delaunay [9] triangulation method to connect the feature points into multiple groups of triangles, and compared the transformation of the triangle edges between adjacent frames. If the edge transformation was too large, it was considered to contain dynamic feature points and removed the feature points. Finally, the maximum connected area of the triangle was selected as the static area [10]. Yang et al. first used the geometric constraint method to roughly filter out the dynamic feature points, and then applied Random Sample Consensus (RANSAC) sample point sampling method based on region division to calculate fundamental matrix, and combined with epipolar geometry to filtering out dynamic feature points accurately [11]. Chen et al. exploit the characteristic that the direction of optical flow of dynamic objects does not coincide with that of static objects, combine depth images to remove optical flow unrelated to camera motion, and find out the threshold of optical flow of moving objects by clustering to achieve high accuracy detection of dynamic regions [12]. Lin et al. segmented the static region by means of a cumulative map of reprojection errors and optimized the dynamic and static regions with different weights to obtain accurate poses [13].

With the development of deep learning, combining semantic information can give better results. Bescos et al. proposed DynaSLAM, which combined multi-view geometry and deep learning-based Mask R-CNN [14] semantic segmentation network to remove dynamic feature points and used static feature points for tracking and localization [15]. The system has high localization accuracy for dynamic scenes, but the real-time performance is poor. The DS-SLAM proposed by Yu et al. distributed the epipolar geometry method and semantic segmentation to two threads, which improved the calculation speed [16]. However, the calculation of the fundamental matrix in epipolar geometry contains too many exterior points, which leads to the localization of the system less accurate. Yang et al. first utilized YOLOv3 [17] to produce semantic borders, then removed the feature points in the dynamic label, reduced the number of exterior points, calculated a more stable fundamental matrix, and then used epipolar geometry to determine the dynamic feature points [18]. But in this method pixel-level semantic segmentation is not implemented, and a certain number of static feature points will be additionally removed when the feature points in the label frame are removed, so that the internal points of the fundamental matrix are reduced. Ruan et al. utilized the Mask R-CNN network to perform pixel-level semantic segmentation of images, it established prior semantic information of dynamic objects, and then combined with epipolar geometry to remove dynamic feature points [19]. AI et al. proposed the DDL-SLAM algorithm, adding a Mask R-CNN based semantic segmentation network and multi-view geometry to detect moving objects on top of ORB-SLAM2, and used static objects for pose estimation [20]. Han et al. proposed the PSP Net-SLAM system based on ORB-SLAM2, which employed optical flow and semantic segmentation to detect and eliminate dynamic points for accurate localization of visual SLAM in dynamic scenes [21]. But these methods only consider people as dynamic objects, except dynamic objects other than people may still affect the localization accuracy of the system. Huang et al. first used the Fast-SCNN network to remove potential dynamic feature points, and then added static points using motion constraints with chi-square tests [22]. Wei et al, used the BiseNet V2 network for semantic segmentation and combined with dense optical flow for target detection to identify specific dynamic objects, reject dynamic regions and construct static maps [23]. Xu et al. improved the single response matrix accuracy and then combined it with YOLOv3 network to reduce the effect of fuzzy noise to improve the localization accuracy of SLAM in dynamic scenes [24]. Hu et al. used DeepLabv3 + semantic segmentation network to segment potentially dynamic objects, then used a multi-view geometric approach to detect object motion information, and finally used an ant colony strategy to reduce detection time and improve real-time performance [25]. Ran et al. used semantic segmentation for target detection, tracking and localization of multiple objects and background reconstruction, with different moving object planes solved independently for rigid body motion and tracked and position independently, while non-planar moving objects were rejected as outliers and not mapped for 3D reconstruction [26].

In order to enable visual SLAM to obtain higher localization accuracy in dynamic scenes, a dynamic scene visual SLAM algorithm combining semantic segmentation and motion consistency detection is proposed. It identifies dynamic objects, divides them into high-dynamic objects and low-dynamic objects, and filters out dynamic feature points on dynamic objects for three times, and only uses static feature points for pose estimation. This algorithm is mainly applicable to indoor environment, so the RGB-D model is adopted. The experimental results show that the algorithm effectively improves the localization accuracy of the visual SLAM system in dynamic scenes.

3 System introduction

3.1 System frameworks

The algorithm in this paper is improved on the RGB-D model of the ORB-SLAM2 algorithm, added a semantic segmentation thread and improved the tracking thread. As shown in Fig. 1, there are total of 4 threads, which including tracking, local map, closed loop detection and semantic segmentation. The semantic segmentation thread and the tracking thread receive RGB images at the same time. The semantic segmentation thread performs pixel-level semantic segmentation on the RGB image, adds semantic labels to the segmentation results, and inputs them to the tracking thread as prior semantic information. The tracking thread adds a dynamic feature point filtering module to remove dynamic feature points and only keep static feature points for pose estimation. The other two threads are as same as ORB-SLAM2.

Fig. 1

System framework.

3.2 Semantic segmentation network

The SegNet deep learning network [27] based on the caffe [28] framework is utilized to perform pixel-level semantic segmentation of RGB images. The PASCLAL VOC [29] dataset is applied as a training model, the SegNet network trained with this model can segment 20 types of objects. The input of the semantic segmentation thread is an RGB image, and the output is a segmented image with labels. Semantic recognition can enhance the scene understanding ability of the visual SLAM system, and the output semantic segmentation images are input to the tracking thread as prior semantic information. For indoor environment, this paper takes “Person” class label as the prior semantic information of high-dynamic objects, and other indoor objects “Bottle”, “Chair”, “Table”, “Plant” and “Tv/monitor” label as the prior semantic information of low-dynamic objects. The prior semantic information is updated in real time and each frame image has its own independent prior semantic information. The current frame segmentation image only takes effect on the RGB image judgment of the current frame. After the tracking thread uses the prior semantic information and removes the dynamic feature points of the current frame image, the prior semantic information of the current frame becomes invalid, and the prior semantic information of the next frame will overwrite it.

3.3 Dynamic feature points filtering method

Figure 2 is the flow chart of the dynamic feature point filtering module. The ORB feature points are extracted and the descriptors are calculated, and the output result of the semantic segmentation thread is waited for. When the tracking thread receives the semantic segmentation image, the prior semantic information is used to detect dynamic objects’ feature points.

Fig. 2

Dynamic feature point filtering module flow chart.

People are most likely to be moving objects in an indoor environment. Because of their large size, a large number of feature points are extracted from people. However, ORB-SLAM2 is based on a static environment and cannot filter out too many dynamic features, at the same time, the bag-of-words model [30] utilized in ORB-SLAM2’s closed-loop detection relies on feature point descriptors, which may affect closed-loop detection when a person’s spatial position in the scene changes and causing the system to generate “false positives” [31]. These impact will have a great impact on the localization accuracy of visual SLAM. Therefore, the people are regarded as high-dynamic objects, and the feature points located on the people are eliminated. The “Person” class label is used as the prior information of the high-dynamic object. After the tracking thread receives the semantic segmentation image, if there is a “Person” label, it will first remove all the feature points in the label, which perform the first dynamic feature point filtering.

However, in addition to high-dynamic objects such as people, there are also other low-dynamic objects that may move, such as chairs, bottle et al. If a certain number of feature points are extracted on these objects that have moved, it will have a certain impact on the localization accuracy of visual SLAM. Therefore, the motion consistency detection is performed on the remaining feature points, and abnormal feature points are output and put them into points set P_out. After the motion consistency detection module outputs abnormal feature points, the tracking thread receives the segmentation result of the semantic segmentation thread again, and the “Bottle”, “Chair”, “Table”, “Plant” and “Tv/monitor” are regard as low-dynamic objects and set as the label to be detected, which is used as the prior semantic information of low-dynamic objects. At this time, the abnormal feature points in the point set are accessed. When a certain number of abnormal feature points fall within the semantic labels of low-dynamic object classes, all the feature points in this class of area will be eliminated, and the dynamic feature points will be filtered out for the second time.

The PASCLAL VOC dataset utilized as a training model only contains 20 types of objects, and there may be other moving objects that have not been recognized, which lacking prior semantic information of low-dynamic objects that not included in the dataset. Therefore, after the second filtering of dynamic feature points, all abnormal feature points in the point set P_out are regarded as dynamic feature points and eliminated, and the third filtering of dynamic feature points is performed, and the remaining stable and reliable static feature points are used for pose estimation.

3.4 Motion consistency detection

The motion consistency detection is performed on the remaining feature points, and the abnormal feature points are output by the method of epipolar geometry based on the assumption of epipolar constraints.

Figure 3 shows the epipolar geometric relationship between feature points on two adjacent frames of images. O₁ and O₂ are the optical centers of the camera in different poses, P (x, y, z) is the observed point in three-dimensional space, p₁ (u₁, v₁) and p₂ (u₂, v₂) are the projections of point P on plane I₁ and plane I₂, and are the feature points at the same time, which are two-dimensional pixel coordinates. The three points O₁PO₂ constitute the polar plane E_p, which intersects with the plane I₁ on the polar line l₂, and intersects with the plane I₂ on the polar line l₁. The line segment O₁O₂ intersects at the plane I₁ and the plane I₂ at the points e₁ and e₂, e₁ and e₂ are the poles.

Fig. 3

Schematic diagram of epipolar geometry.

Assuming that p₁ and p₂ are correct feature matching, let the normalized coordinates of p₁ and p₂ to be P₁ = (u₁, v₁, 1) and P₂ = (u₂, v₂, 1), then Equation (1) is satisfied ${\begin{matrix} d_{1} P_{1} = KP \\ d_{2} P_{2} = K (RP + t) \end{matrix}$ (1)

d₁ and d₂ are the depth values of the feature points p₁ and p₂, K is the camera internal parameter matrix, R and t respectively represent the rotation and translation transformation of the camera moving from point O₁ to point O₂.

Let, x₁ = K^-1P₁, x₂ = K^-1P₂we can get $x_{2} = C_{0} ({Rx}_{1} + C_{1} t)$ (2)

Where C₀ and C₁ are constants, and multiplying both sides of Equation (2) to the left by $x_{2}^{T} [t^{\land}]$ at the same time, we can get $x_{2}^{T} [t^{\land}] x_{2} = C_{0} x_{2}^{T} [t^{\land}] {Rx}_{1}$ (3)

Where [t^∧] is the antisymmetric matrix corresponding to the translation vector t, the left side of Equation (3) is equal to zero, so there is Equation (4) $x_{2}^{T} [t^{\land}] {Rx}_{1} = 0$ (4)

We can Equation (5) when put x₁ = K^-1P₁ and x₂ = K^-1P₂ into Equation (4) ${\begin{matrix} P_{2}^{T} {FP}_{1} = 0 \\ F = K^{- T} [t^{\land}] {RK}^{- 1} \end{matrix}$ (5)

Equation (5) is the polar constraint relationship between the normalized coordinates P₁ and P₂ of the pixels p₁ and p₂ on the plane I₁ and the plane I₂, indicating that P₁ and P₂ are on the plane O₁PO₂, where F is the fundamental matrix. The vector $\vec{e_{2} P_{2}}$ is the projection of the vector $\vec{O_{1} P}$ on the plane I₂. $\vec{e_{2} P_{2}}$ is on the polar line l₁ corresponding to the normalized coordinate point P₁. If the point P does not move, the normalized coordinate point p₂ will fall on the polar line l₁.

However, because of the existence of errors in practical applications, the normalized coordinate point P₂ does not necessarily fall completely on the epipolar line l₁, and the distance D from P₂ to the epipolar line l₁ should tend to zero. When D is too large, there are two situations: one is that p₁ and p₂ are incorrectly matched, and the other is that point P has moved. In the case where P moves, as shown in Fig. 3, when the camera moves from O₁ to O₂, P moves to P′, the projection at the plane I₂ is $p_{2}^{'}$ , O₁P′O₂ three points constitute a new polar plane, whose normalized coordinate $P_{2}^{'}$ is not located on the intersection line of the original polar plane E_p and the plane I₂. Therefore, it can be determined whether p₂ is an abnormal feature point that has moved, according to the distance from the normalized coordinate point P₂ to the polar line l₁.

The polar line l₁ corresponding to the normalized coordinate point P₁ is on the plane I₂, and it satisfies Equation (6) ${\begin{matrix} {\begin{matrix} a \\ b \\ c \end{matrix}} = {FP}_{1} = F {\begin{matrix} u_{1} \\ v_{1} \\ 1 \end{matrix}} \\ l_{1} : ax + by + c = 0 \end{matrix}$ (6)

Where (a, b, c) ^T represents the direction vector of the epipolar line l₁, F is the fundamental matrix, and the distance from the normalized coordinate P₁ to the epipolar line l₁ can be expressed as $D = \frac{| P_{2}^{T} {FP}_{1} |}{\sqrt{{∥ a ∥}^{2} + {∥ b ∥}^{2}}}$ (7)

The threshold ξ₀ is set, when D> ξ₀, let p₂ be the abnormal feature point and put it into the point set $P_{out} = {p_{2}^{1}, p_{2}^{2}, \dots, p_{2}^{i}, \dots}$ .

3.5 Improved RANSAC to solve fundamental matrix

The fundamental matrix F can represent the pose change relationship of two adjacent frames of images, and is the key information for judging dynamic feature points by using epipolar geometry. Let F₂ be the current frame image, F₁ be the previous frame image, and their feature point sets are $P_{F_{2}} = {p_{2}^{1}, p_{2}^{2}, \dots, p_{2}^{n}}$ and $P_{F_{1}} = {p_{1}^{1}, p_{1}^{2}, \dots, p_{1}^{m}}$ respectively, n and m are the total number of feature points after removing the people feature points from the corresponding image respectively. The matching point pair between the current frame and the previous frame through feature matching are found, and they are put into the matching point set M_p ={ (p₁p₂) ¹, (p₁p₂) ², ⋯ , (p₁p₂) ^c }, where c is the number of matching point pairs.

However, the matching point set M_p may have a certain number of mismatches and matching point pairs of dynamic feature points other than people. Too many exterior points make the fundamental matrix F solved by the traditional RANSAC [32] algorithm have a large error, which affects the subsequent determination of other dynamic feature points except for people. How to solve a stable fundamental matrix is the key to testing abnormal feature points.

The fundamental matrix is calculated by using the improved RANSAC algorithm. On the basis of the traditional RANSAC algorithm, two main improvements have been made: the first is to filter the initial data to remove incorrect matching and matching point pairs that may have large errors; the second is to use double iterations and additional threshold judgments, after the first iteration, the static feature point assumption method is applied to screen the data again, and a more accurate feature matching point pair is selected as the new data input, and a better interior point is used for the second iteration to solve a more stable fundamental matrix.

First, the matching point pairs in the matching point set M_p of the current frame F₂ and the previous frame F₁ are screened out. The threshold H₀ is set. When the Hamming distance of the matching point pair is larger than H₀, the matching point pair is eliminated. At the same time, in order to reduce the influence of pixel-level semantic segmentation error, all feature points in F₂ are traversed, and when there is a “Person” semantic label in the 3 pixels around the feature point, the matching point pair corresponding to this feature point is cleared. Then the filtered matching point set M_pre ={ (p₁p₂) ¹, (p₁p₂) ², (p₁p₂) ³, ⋯ , (p₁p₂) ^a } is obtained, where a is the total number of matched point pairs after filtering. The point set has been preliminarily screened out exterior points, and more reliable data is used for iteration. The specific iterative process is as follows:

Step 1) Eight point pairs in matching point set M_pre are select randomly, and the selected data is used to estimate the mode M₀.

Step 2) The estimated model M₀ is used to check all the point pairs in the matching point set M_pre, the point pairs are saved that satisfy the model M₀, and the number of point pairs n is recorded.

Step 3) Step 1) and step2) are repeated, when the iteration meets the requirement of times, the set with the largest number of point pairs is selected, the model parameters are output, and the initial value F₀ of the fundamental matrix is solved.

Step 4) The initial value F₀ of the fundamental matrix is substituted into the Equations (5) and (6), the static feature point assumption distance threshold d₀ is set, all the point pairs in the matching point set M_pre are traversed, when there is D< d₀, the detected point will be put the pair (p₁p₂) ⁱ into the new point set M_cur ={ (p₁p₂) ¹, (p₁p₂) ², (p₁p₂) ³, ⋯ , (p₁p₂) ^b }, where b is the total number of new matching point pairs.

Step 5) A second iteration is performed with better interior points, eight point pairs in the new point set M_cur are selected randomly, and the model M₀ is estimated again.

Step 6) The model M₀ is selected to check all the point pairs in the new point set M_cur, the data satisfying the model M₀ is saved, and the number of point pairs n is recorded.

Step 7) Step 5) and step 6) are repeated, when the number of iterations is reached, the model parameters with the largest number of point pairs are output, and the final stable fundamental matrix F is obtained.

Finally, the solved stable fundamental matrix F is put into Equations (6) and (7), Equation (7) is used to verify all matching point pairs in the matching point set M_p, and abnormal feature points are output.

4 Experimental result and analysis

The experiment was carried out on a computer with processor Intel i5-10300 H CPU @2.50 GHz, memory 16G, graphics card NVIDIA GeForce GTX1650Ti, and operating system Ubuntu18.04. In order to simulate dynamic scenes in indoor environments, the dynamic object sequence of the TUM dataset [33], which consists of a “walking” sequence and a “sitting” sequence, was selected as the image input for the experimental RGB-D data. The data of the TUM dataset is collected from the actual environment by the depth camera, and the image part is composed of a set of color maps and corresponding depth maps, and also contains the real motion trajectory of the camera. In order to verify the effectiveness and localization accuracy of the proposed algorithm, the dynamic feature point elimination experiment and the SLAM system performance test experiment were carried out respectively.

4.1 Dynamic feature elimination experiment

This experiment is used to test the ability of the proposed algorithm to eliminate feature points of dynamic objects, including high-dynamic objects such as “people” and low-dynamic objects such as “Chair”. The dynamic object sequence “walking_xyz” from the TUM dataset was used as the image input to compare the effectiveness of ORB-SLAM2 and this paper’s algorithm to eliminate dynamic feature points.

Figure 4 is the effect diagram of the dynamic feature point removal of the people, the corresponding image sequence is the 386th frame and the 571th frame, and the persons are in a walking state in these two frames of images. The ORB-SLAM2 algorithm extracted a certain number of feature points on the persons. The algorithm in this paper removed the feature points on the persons after semantic segmentation and motion consistency detection.

Fig. 4

People dynamic feature point removal effect.

Figure 5 is the rendering of the dynamic feature point culling of people and other objects. The corresponding image sequences are the 600th and 642th frames. A person in these two images move the chair and sit down, and both the person and the chair on the right side of the picture have moved. ORB-SLAM2 extracted feature points from person and moving chair. The algorithm in this paper considered the influence of other objects except people, as shown in Fig. 5(b), so the feature points on person and chair were eliminated. And the feature points on the chair that did not move on the left side of the image were preserved.

Fig. 5

The culling effect of dynamic feature points of people and other objects.

4.2 SLAM system performance test

In order to verify the localization accuracy of the proposed algorithm in dynamic scenes, a SLAM system performance test experiment was conducted. The ORB-SLAM2 algorithm, the DS-SLAM algorithm and the algorithm in this paper are compared separately. ORB-SLAM2 is considered to be one of the best and most stable SLAM solutions based on the feature point method. At the same time, DS-SLAM is one of the excellent visual SLAM solutions for dynamic environments, and it was also improved on the basis of ORB-SLAM2.

The Absolute Trajectory Error (ATE) and the Relative Pose Error (RPE) were used as the evaluation criteria for the experiments. The ATE refers to the difference between the real pose and the estimated pose, and the difference between the two sets of poses is calculated under the same timestamp, which can most intuitively reflect the accuracy of SLAM pose estimation and the accuracy of the estimated trajectory. The RPE represents the difference between the real trajectory and the estimated trajectory pose change per unit time, which reflects the rotational drift and translational drift state of the SLAM system pose estimate. The Root Mean Square Error (RMSE), mean error, median error, and standard deviation (std) were used serve as further detailed evaluation criteria.

Equation (8) is used as the calculation of the improvement value, where Q is the improvement value, φ is the experimental evaluation result being compared and θ is the evaluation result of the algorithm in this paper. $Q = (1 - \frac{θ}{φ}) \times 100$ (8)

The experiment selected the highly dynamic object sequence “walking”, and part of the low dynamic object sequence “sitting” in dynamic object sequence. The “walking” sequence contains four sub-sequences of walking_xyz (w_xyz), walking_halfsphere (w_half), walking_rpy (w_rpy) and walking_static (w_stat). The “xyz” sequence camera translates along the xyzh axis, the “halfsphere” sequence camera moves in a hemispherical shape, the “rpy” sequence camera rotates in three directions along the xyzh axis, and the “static” sequence camera basically remains stationary. In these four sub-sequences, two persons move frequently in the scene and move some objects in the scene, which is rich in dynamic information. In the experiment, the sub-sequences of sitting_halfsphere (s_half) and sitting_static (s_stat) were selected in the “sitting” sequence. In these two sub-sequences, two persons communicate while sitting on the chair, and there is a small amount of body movement and less dynamic information.

Tables 1 to 3 show the results of the comparison tests. In the highly dynamic scene sequences, it can be seen that the value of this paper’s algorithm in ATE and translational drift are lower than ORB-SLAM2 and slightly lower than DS-SLAM. The value of this paper’s algorithm in rotational drift are better than ORB-SLAM2, and the value of the other three sequences are slightly higher than DS-SLAM except for the “w_half” sequence; In the low dynamic scene sequences, the value of this paper’s algorithm in ATE, translation drift and rotation drift differ very little from ORB-SLAM2 and DS-SLAM.

Table 1

Test results of absolute trajectory error (ATE)

Sequence	ORB-SLAM2/m				DS-SLAM/m				Algorithm/m
	RMSE	mean	media	std	RMSE	mean	media	std	RMSE	mean	media	std
w_xyz	0.7077	0.5922	0.5010	0.3874	0.0367	0.0301	0.0229	0.0209	0.0335	0.0275	0.0216	0.0191
w_half	0.5899	0.5057	0.5365	0.3038	0.0304	0.0266	0.0230	0.0148	0.0285	0.0251	0.0221	0.0142
w_rpy	0.9418	0.8033	0.9323	0.4916	0.3824	0.3284	0.2526	0.1960	0.0930	0.0700	0.0503	0.0612
w_stat	0.1979	0.1667	0.1267	0.1066	0.0103	0.0091	0.0078	0.0048	0.0091	0.0078	0.0064	0.0047
s_half	0.0277	0.0244	0.0225	0.0130	0.0227	0.0202	0.0240	0.0104	0.0238	0.0213	0.0195	0.0107
s_stat	0.0086	0.0075	0.0069	0.0041	0.0079	0.0070	0.0063	0.0036	0.0073	0.0065	0.0057	0.0034

Table 2

Test results of translational drift (RPE)

Sequence	ORB-SLAM2/m				DS-SLAM/m				Algorithm/m
	RMSE	mean	media	std	RMSE	Mean	media	std	RMSE	mean	media	std
w_xyz	0.4131	0.3137	0.2337	0.2688	0.0401	0.0320	0.0264	0.0241	0.0350	0.0289	0.0241	0.0197
w_half	0.3863	0.2375	0.0635	0.3047	0.0333	0.0292	0.0264	0.0161	0.0323	0.0280	0.0247	0.0161
w_rpy	0.4018	0.2782	0.1210	0.2898	0.1470	0.1018	0.0650	0.1061	0.1013	0.0758	0.0510	0.0672
w_stat	0.1123	0.0421	0.0131	0.1041	0.0106	0.0092	0.0084	0.0052	0.0086	0.0077	0.0069	0.0039
s_half	0.0274	0.0216	0.0179	0.0169	0.0260	0.0233	0.0215	0.0117	0.0261	0.0233	0.0216	0.0116
s_stat	0.0099	0.0087	0.0081	0.0046	0.0093	0.0080	0.0070	0.0047	0.0087	0.0076	0.0069	0.0040

Table 3

Test result of rotational drift (RPE)

Sequence	ORB-SLAM2/°				DS-SLAM/°				Algorithm/°
	RMSE	mean	media	std	RMSE	mean	media	std	RMSE	mean	media	std
w_xyz	7.9655	6.0568	4.6740	5.1734	0.8402	0.6277	0.5126	0.5585	0.7572	0.5874	0.4902	0.4779
w_half	7.9920	4.9860	1.5714	6.2460	0.8641	0.7645	0.7002	0.4027	0.8684	0.7678	0.7161	0.4056
w_rpy	7.9107	5.4447	2.4731	5.7389	2.7635	1.8659	1.1227	2.0386	1.8858	1.4269	0.9372	1.2331
w_stat	1.9733	0.8089	0.3276	1.7999	0.2672	0.2376	0.2126	0.1224	0.2397	0.2145	0.2000	0.1069
s_half	0.7942	0.6918	0.6126	0.3756	0.7813	0.6918	0.6191	0.3631	0.7584	0.6629	0.5869	0.3684
s_stat	0.2887	0.2594	0.2429	0.1267	0.2867	0.2575	0.2437	0.1261	0.2866	0.2634	0.2424	0.1211

Table 4 shows the improvement values of our algorithm relative to ORB-SLAM2 and DS-SLAM, where RPE(T) represents translational drift and RPE(R) represents rotational drift. For ATE, in highly dynamic sequences, while compared to ORB-SLAM2, the average RMSE of our algorithm is reduced by 93.99% and the average std by 93.2%. While compared to DS-SLAM, the average RMSE of our algorithm was reduced by 25.58% and the average std was reduced by 20.88% for “w_rpy” sequences, which was greatly improved. In low dynamic sequences, the average RMSE of our algorithm is reduced by 14.6% and the average std is reduced by 17.38% compared to ORB-SLAM2. The average RMSE of this paper’s algorithm is reduced by 6.22% and the average std is reduced by 1.34% compared to DS-SLAM. In low dynamic sequences, where the amount of character movement in the scene is low and close to a static scene, the algorithms in this paper have reduced improvement values for ORB-SLAM2 and DS-SLAM. For RPE, the improvement values are similar to and correlated with ATE. In summary, the improvements in this paper are obvious, with some improvement in localization accuracy, reduced drift and a more stable system.

Table 4

Improvement value

Improvement (Compare to ORB-SLAM2)/%						Improvement (Compare to DS-SLAM)/%
Sequence		w_xyz	w_half	w_rpy	w_stat	s_half	s_stat	w_xyz	w_half	w_rpy	w_stat	s_half	s_stat
ATE	RMSE	95.27	95.17	90.13	95.40	14.08	15.12	8.72	6.25	75.68	11.65	4.85	7.59
	mean	95.36	95.04	91.29	95.32	12.70	13.33	8.64	5.64	78.68	14.29	–5.45	7.14
	std	95.07	93.52	87.55	93.06	17.69	17.07	8.61	4.05	68.78	2.08	–2.88	5.56
RPE (T)
	RMSE	91.53	91.64	74.79	92.34	4.74	12.12	12.72	3.00	31.09	18.87	–0.38	6.45
	mean	90.78	88.21	72.75	81.71	–7.87	12.64	9.68	4.10	25.54	16.30	0.00	5.00
	std	92.67	94.71	76.81	96.25	31.36	13.04	18.25	0.02	36.66	25.00	0.85	14.89
RPE (R)
	RMSE	90.49	89.13	76.16	87.85	4.51	0.73	9.88	–0.50	31.76	10.29	2.93	0.03
	mean	90.30	84.60	73.79	73.48	4.17	–1.54	6.42	0.43	23.52	9.72	4.17	–2.29
	std	90.76	93.50	78.51	94.06	1.91	4.41	14.43	–0.72	39.51	12.66	–1.45	3.96

The improved effect is also shown graphically. Figure 6 shows the ATE diagram of ORB-SLAM2, DS-SLAM and the algorithm in this paper. The black line is the truth trajectory of the camera, the blue line is the estimated trajectory, and the red line represents the difference between the two. In highly dynamic sequence, compared with ORB-SLAM2, the area of the red is greatly reduced, localization accuracy of this algorithm is significantly improved, and the localization accuracy in this paper is slightly higher than that of DS-SLAM. In “w_rpy” sequence, the algorithm in this paper has a great improvement compared to DS-SLAM. In low dynamic sequence, the difference between the localization accuracy of the three is small. The algorithm in this paper has a priori semantic information in motion consistency detection, reducing the influence of too many outlier points; the improved RANSAC algorithm is used to calculate a more stable fundamental matrix, which can more accurately use the polar line geometry to detect dynamic feature points; at the same time, the influence of dynamic objects other than humans is additionally considered, making the rejection of dynamic feature points more comprehensive. As a result, the algorithm in this paper provides some improvement in localization accuracy in highly dynamic sequences compared to DS-SLAM.

Fig. 6

Absolute trajectory error.

Compared with other sequences, the localization accuracy of the “w_rpy” sequence is lower, because the motion mode of “w_rpy” is rotation, and the rotation speed is too fast, which causing some images to be blurred. At the same time, the feature point descriptor has poor adaptability to rotation, and a large number of stable features match point pairs is not formed, thereby the localization accuracy is reduced.

5 Conclusion

A visual SLAM algorithm based on semantic segmentation and motion consistency detection is proposed to solve the problem of low localization accuracy in dynamic senses. The algorithm filters out the dynamic feature points for three times: First, the SegNet network is used to perform pixel-level semantic segmentation of RGB images, and the segmentation results are used as prior semantic information to define people as high-dynamic objects, the “Person” label is detected, the feature points located on the people are removed, which perform the first filtering of dynamic feature points; Then, the abnormal feature points are output through motion consistency detection, and the feature points on other low-dynamic objects except people are eliminated according to the distribution of abnormal feature points and combined with prior semantic information of low-dynamic objects, which perform the second filtering of dynamic feature points; Finally, all abnormal feature points are regarded as dynamic feature points and eliminated, which perform the third filtering of dynamic feature points. Finally, the TUM dataset is selected for testing, and highly dynamic scenes and low dynamic scenes are selected respectively. Compared with ORB-SLAM2, the average RMSE of this algorithm is reduced by 93.99%, and the average RMSE of DS-SLAM is reduced by 25.58%. The improvement in low dynamic scenes is slightly lower, and the average RMSE is 14.6% lower than ORB-SLAM2 and 6.22% lower than DS-SLAM.

The SLAM algorithm proposed in this paper uses the RGB-D model. In subsequent work, additional monocular and stereoscopic models are planned. It is also planned to apply the algorithms to real robots to enable real-time map building in different environments.

References

Quan

M.X.

, Piao

S.H.

and Li

, An overview of visual SLAM[J], CAAI Transactions on Intelligent Systems 11(6) (2016), 768–776.

Endres

, Hess

, Engelhard

et al., An evaluation of the RGB-D SLAM system[J], Proceedings IEEE International Conference on Robotics & Automation, 2012.

Engel

, Schöps

, Cremers

LSD-SLAM: Largescale direct monocular SLAM[C], European conference on computer vision, Springer, Cham, 2014:834–849.

Mur-Artal

, Montiel

J.M.M.

and Tardos

J.D.

, ORB-SLAM: A Versatile and Accurate Monocular SLAM System[J], IEEE Transactions on Robotics 31(5) (2015), 1147–1163.

Mur-Artal

, Tardós

J.D.

ORB-SLAM2: An OpenSource SLAM System for Monocular, Stereo, and RGB-D Cameras[J], IEEE Transactions on Robotics, 2017.

Wang

K.S.

, Yao

X.F.

, Huang

et al., Review of Visual SLAM in Dynamic Environment[J],, ROBOT 43(6) (2021), 18.

and Lee

, RGB-D SLAM in Dynamic Environments using Static Point Weighting[J], IEEE Robotics & Automation Letters 2(4) (2017), 2263–2270.

Wei

and Li

, Binocular Vision SLAM Algorithm Based on Dynamic Region Elimination in Dynamic Environment[J],, ROBOT 42(3) (2020), 10 .

Barber

C.B.

, Dobkin

D.P.

and Huhdanp

, Aa, The quickhull algorithm for convex hulls[J],, ACM Transactions on Mathematical Software 22(4) (1993).

10.

Dai

, Zhang

, Li

et al., RGB-D SLAM in Dynamic Environments Using Point Correlations[J],, IEEE Transactions on Pattern Analysis and Machine Intelligence PP(99) (2020), 1–1.

11.

Yang

S.Q.

, Fan

G.H.

, Bai

L.L.

et al., Geometric Constraint-Based Visual SLAM Under Dynamic Indoor Environment[J],, Computer Engineering and Applications 57(16) (2021), 203–212.

12.

, Zhou

Y.J.

and Luo

, Visual SLAM Method Based on Optical Flow and Instance Segmentation for Dynamic Scenes[J],, Acta Optica Sinica 42(14) (2022), 147–159.

13.

Lin

, Liang

X.W.

and Cai

J.Y.

, Dynamic RGB-D SLAM algorithm based on reprojection depth difference cumulative map and static probability[J],, Journal of Zhejiang University (Engineering Science) 56(06) (2022), 1062–1070.

14.

, Gkioxari

, Dollar

et al., Mask r-cnn[C], Proceedings of the IEEE international conference on computer vision, 2017:2961–2969.

15.

Bescos

, Facil

J.M.

, Civera

et al., DynaSLAM: Tracking, mapping, and inpainting in dynamic scenes[J],, IEEE Robotics and Automation Letters 3(4) (2018), 4076–4083.

16.

, Liu

X.J.

et al., DS-SLAM: A semantic visual SLAM towards dynamic environments[C], 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, 2018:1168–1174.

17.

Redmon

, Divvala

, Girshick

et al., You only look once: Unified, real-time object detection[C], Proceedings of the IEEE conference on computer vision and pattern recognition, 2016:779–788.

18.

Yang

, Fan

, Bai

et al., SGC-VSLAM: A semantic and geometric constraints VSLAM for dynamic indoor environments[J],, Sensors 20(8) (2020), 2432.

19.

Yuan

X.G.

, Guo

P.Y.

and Huang

, Semantic Visual SLAM Based on Deep Learning in Dynamic Scenes[J],, Journal of Beijing University of Technology 48(1) (2022), 8.

20.

, Rui

, Lu

et al., DDL-SLAM: A robust RGB-D SLAM in dynamic environments combined with Deep Learning[J],, IEEE Access PP(99) (2020), 1–1.

21.

Han

and Xi

, Dynamic Scene Semantics SLAM Based on Semantic Segmentation[J], IEEE Access 8 (2020), 43563–43570.

22.

Huang

S.J.

, Hu

M.J.

, Zhou

Y.S.

et al., Vehicle Visual SLAM in Dynamic Scenes Based on Semantic Segmentation and Motion Consistency Constraints[J],, Automotive Engineering 44(10) (2022), 1503–1510.

23.

Wei

S.F.

, Li

, Liu

G.Z.

et al., A Visual SLAM Algorithm Combined with Dynamic Target Detection[J],, Science of Surveying and Mapping 47(07) (2022), 93–103.

24.

X.S.

and Zeng

, Visual Simultaneous Localization and Mapping Algorithm Based on Dynamic Target Detection[J], Laser &Optoelectronics Progress 58(16) (2021), 423–430.

25.

, Zhao

, Luo

et al., Semantic SLAM Based on Improved DeepLabv3⁺ in Dynamic Scenarios[J],, IEEE Access pp(10) (2022), 21160–21168.

26.

Long

, Rauch

, Zhang

T.W.

et al., RGB-D SLAM in Indoor Planar Environments With Multiple Large Dynamic Objects[J], IEEE Robotics andAutomation Letters 2022:8209–8216.

27.

Badrinarayanan

, Kendall

, Cipolla

SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation[J], IEEE Transactions on Pattern Analysis & Machine Intelligence (2017), 1–1.

28.

Jia

, Shelhamer

, Donahue

et al., Caffe: Convolutional Architecture for Fast Feature Embedding[J], ACM, 2014.

29.

Everingham

, Eslami

, Gool

L.V.

et al., The Pascal Visual Object Classes Challenge: A Retrospective[J],, International Journal of Computer Vision 111(1) (2015), 98–136.

30.

Angeli

, Filliat

, Doncieux

et al., A fast and incremental method for loop-closure detection using bags of visual words[J], IEEE Transactions on Robotics 2008:1027–1037.

31.

Gao

, Zhang

Fourteen lectures on visual SLAM. Version 2[M], Beijing: Publishing House of Electonics Industry, 2019, 8.

32.

and An

W.F.

, Visual SLAM based on the improved RANSAC algorithm[J],, Journal of Tianjin University (Science and Technology) 53(10) (2020), 1069–1076.

33.

Sturm

, Engelhard

, Endres

et al., Abenchmark for the evaluation of RGB-D SLAM systems[J], 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2012:573–580.