Stereo camera visual odometry for moving urban environments

Abstract

This work proposes a system designed to estimate the ego-motion of a synchronized calibrated stereo camera in scenes containing a moderate number of moving objects. This is particularly useful in busy road scenes and populated urban areas. The key novelty of the proposed approach is that it estimates the motion of clusters of pixels between stereo frames, which allows it to explicitly reject clusters in motion. This is in contrast to current state-of-the-art algorithms, that tend to treat moving elements as outliers, which are removed using strategies such as RANSAC or M-estimators. Unfortunately treating moving pixels as outliers can give poor performance when the motion represents a significant portion of pixels. The proposed approach overcomes this, if the motion is due to many independently moving objects (such as people or cars). Our experiments show promising results in a variety of urban environments.

Keywords

Visual odometry stereo 3-D

1. Introduction

Computer Vision from a sequence of images is far more powerful than from just a single frame. However, in order to relate an entire sequence of images, it is important to know the location of the camera at the point of each image’s acquisition. Visual odometry is the name given to the problem of estimating the motion of the camera, between frames, based only on image data. The search for good visual odometry solutions is a popular topic in literature, especially in conjunction with the automation of cars [1, 2, 3], simultaneous localization and mapping [4] (SLAM), and unmanned aerial vehicles [5] (UAV).

This work focuses on estimating the trajectory of a synchronized stereo camera in an urban environment with moving objects. The proposed algorithm specifically seeks to estimate the motion of a free moving (six degrees of freedom) stereo camera between frames captured at a moderate frame rate (30 frames per second).

The challenge, in such an environment, is overcoming the fact that a large number of pixels, within an image, can be associated with objects that are moving in the scene. Many state-of-the-art algorithms treat moving objects as outliers, that are filtered using a strategy such as RANSAC [6] or M-estimators [7]. Unfortunately, when outliers counts exceed certain thresholds, these algorithms tend to perform poorly.

The strategy, proposed in this work, is to find several clusters of corresponding pixels between images. Then, using a direct linear transform (DLT), each of these clusters is able to yield an independent motion estimate of the camera between images. If the scene consists of some or many independently moving objects with a generally static background, it is expected that there will be a variety of motion estimates. However, those associated with background elements should be relatively consistent, while those associated with several independently moving foreground elements would generally not be. Thus, the estimated motion of the camera, in such cases, may be determined from the set of corresponding clusters of pixels that have the most agreement in terms of estimated camera motion.

A review of current literature is given in Section 2. The proposed visual odometry solution is described in Section 3. The framework used to assess the solution is given in Section 4. Results are in Section 5. This work then concludes with Section 6 and future work is described in Section 7.

2. Literature review

Visual Odometry is the problem of estimating the motion of Hthe camera between frames, based purely on image data. Typically, these estimates are computed based on the way specific features physical position evolve between images.

One popular choice of feature, is a sparse set of 2-D points that are obtained by corner detectors such as the Harris Detector [3] and the Features from Accelerated Segment Test [2] (FAST) detector. These 2D image points are typically converted to 3-D space using either the properties of calibrated stereo cameras [6-8] or a probabilistic map [4, 5] in the case of monocular systems. Once the 2-D points have been converted to 3-D, camera motion can be estimated using an algorithm such as Epnp [9], if the problem is formulated as a 3-D to 2-D mapping, or the approach of Arun et al. [10], when the problem is formulated as a 3-D to 3-D mapping. If a 3-D to 3-D mapping is chosen, in the context of depth from stereo, it is recommended that scale biases be accounted for with an appropriate technique, such as the one proposed by Dubbelman and Groen [11].

Another approach to estimating camera motion between images is to register dense depth maps using the iterative closest point [12] (ICP). This approach is typically used in conjunction with algorithms that acquire depth maps using active depth sensor cameras such as Intel ${}^{\@setsize{\scriptsize}{8pt}{\viipt}{\@viipt}\textregistered}$ Real Sense ${}^{\text{TM}}$ depth cameras.1 There are several variants of the ICP algorithm, including the projective data association variant [13] that can execute in “real-time” on a GPU, as is the case of KinectFusion [14].

It is also possible to estimate camera motion between images using gray-scale or color information directly. A typical strategy to achieve this, is to use the Lucas-Kanade inverse compositional [15], as can be seen in work by Klose et al. [16] and Forster et al. [5].

All of these approaches assume that the variation of the chosen features between images is purely due to the motion of the camera. Of course, this may not be the case, especially if there are moving elements in the scene. Unfortunately, moving objects is a rather common phenomenon, especially in urban settings. By far, the most common approach, in the state-of-the-art, to deal with these moving objects is to treat them as another source of noise [1], along with lighting variance, occlusions and correspondence mismatches. There are several strategies for dealing with such noise, RANSAC [1, 3, 6, 17] being the most popular. The Kalman filter [8] is another commonly used approach, along with M-estimators [7]. Another popular technique, that takes advantage of the iterative non-linear solvers that are typically used to refine camera motion estimates introduces weighted errors [18] to minimize the effect of outliers on the final result. The perceived problem with this approach, is that outliers are assumed to represent a small percentage of the data in most filtering schemes, however this may not be the case in typical congested and cluttered urban scenes.

The few approaches which have explicitly tackled the problem of scene motion with respect to visual odometry employ strategies revolving around basic low-level object recognition. Kitt et al. [8] uses a Support Vector Machine [19] (SVM) scheme trained to identify objects traditionally associated with motion (cars), and mask out those objects during ego motion estimation.

The main advantage of object recognition, is that it is able to identify a group of pixels within an image, and assign a set of attributes to that group. It is often assumed that a group of pixels that all corroborate a single classification, is more reliable than an approach that operates on a pixel-by-pixel basis to filter outliers. However, object recognition can be slow, often requires training data, and sometimes leads to the unnecessary filtering of pixels (a stationary car may be filtered out if cars are assumed to be moving objects). The proposed approach uses the idea of Engel et al. [4] and Yao et al. [20] to track against regions of high gradient change within images, which can be relatively common in man-made uniform environments that are generally devoid of much texture. However, this work exploits the characteristics of these regions which tend to be clusters of pixels with assigned attributes that determine whether those clusters are reliable for features tracking.

3. Proposed approach

3.1 Prerequisites

The proposed system is specifically designed to be used in conjunction with a synchronized stereo camera system as $\{C_{1},C_{2}\}$ , where $C_{1}$ is the left camera and $C_{2}$ is the right. The left camera $C_{1}$ and right camera $C_{2}$ are expected to be rigidly mounted with respect to each other. The camera system $\{C_{1},C_{2}\}$ should acquire a sequence of $m$ stereo pairs $\mathcal{I}_{i}=\{I_{1:i},I_{2:i}\}$ with $\{i\in\mathbb{Z}\ |\ 0\leqslant i<m\}$ , where $I_{1:i}$ is the $i^{\text{th}}$ left image and $I_{2:i}$ is the $i^{\text{th}}$ right image.

It is also expected that the camera system $\{C_{1},C_{2}\}$ is calibrated, where calibration is the process of determining the intrinsic matrix $\mathbf{K_{j:3\times 3}}$ (given that $j=1$ or $j=2$ ) for each of the two cameras in the stereo camera system, and extrinsic parameters $\mathbf{M_{4\times 4}}$ that maps the coordinate system of the left camera to the coordinate system of the right camera.

Here $\mathbf{K_{j:3\times 3}}$ is a camera matrix of the form:

$\displaystyle\mathbf{K_{j:3\times 3}}=\left[{\begin{array}[]{ccc}f_{j:x}&0&c_{% j:x}\\ 0&f_{j:y}&c_{j:y}\\ 0&0&1\\ \end{array}}\right]$ (1)

where $f_{j:x}$ and $f_{j:y}$ are the focal lengths of the $j^{\text{th}}$ camera in the $x$ - and $y$ -directions, and $[c_{j:x},c_{j:y}]^{\top}$ are the coordinates of the optical center of the $j^{\text{th}}$ camera in pixels.

A distortion model is also extracted for each camera, which in this work, is defined as follows:

$\displaystyle u_{u}\approx u_{d}+\hat{u}(\kappa_{1}r^{2}+\kappa_{2}r^{4})+\rho% _{1}(r^{2}+2\hat{u}){}+2\rho_{2}\hat{u}\hat{v}$ (2) $\displaystyle v_{u}\approx v_{d}+\hat{u}(\kappa_{1}r^{2}+\kappa_{2}r^{4})+\rho% _{2}(r^{2}+2\hat{v}){}+2\rho_{1}\hat{u}\hat{v}$ (3)

where $[u_{u},v_{u}]^{\top}$ and $[u_{d},v_{d}]^{\top}$ are the distortion-free and distorted coordinates within the image, $\kappa_{1}$ and $\kappa_{2}$ are the first and second order radial distortion coefficients, $\rho_{1}$ and $\rho_{2}$ are the first and second order tangential coefficients, $\hat{u}=(u_{d}-c_{x})$ , $\hat{v}=(v_{d}-c_{y})$ and $r=\sqrt{\hat{u}^{2}_{d}+\hat{v}^{2}_{d}}$ .

The extrinsic matrix is defined as

$\displaystyle\mathbf{M_{4\times 3}}=\left[{\begin{array}[]{cc}\mathbf{R_{3% \times 3}}&\mathbf{T}\\ 0_{1\times 3}&1\\ \end{array}}\right]$

where $\mathbf{R_{3\times 3}}$ is a $3\times 3$ rotation matrix and $\mathbf{T}$ is a 3 element column vector representing a translation.

Calibration may be performed using the popular Zhang calibration [21], however the experimental work featured in this article prefers Tsai’s calibration [22] following the approach of Gee et al. [23].

After calibration, each stereo pair $\mathcal{I}_{i}=\{I_{1:i},I_{2:i}\}$ has its distortion removed and is rectified to $\mathcal{I}_{r:i}=\{I_{r1:i},I_{r2:i}\}$ using the calibration parameters and the approach outlined in Fusiello et al. [24]. This also yields a rectified camera matrix $\mathbf{K_{3\times 3}}$ , with focal length $f$ and optical center $[c_{x},c_{y}]^{\top}$ , which is chosen to be the same for both left and right cameras.

It is assumed that stereo matching [25] is performed on each rectified stereo pair $\mathcal{I}_{r:i}$ yielding a disparity map $D_{\textit{disp}:i}$ for each rectified left image $\mathcal{I}_{r1:i}$ . These disparity maps can be converted to depth maps using the relation:

$\displaystyle Z_{l}=\frac{f||-\mathbf{R_{3\times 3}}^{\top}\mathbf{T}||}{d_{l}}$ (4)

where $Z_{l}$ is the depth at pixel $\mathbf{p_{l}}=[u_{l},v_{l}]^{\top}$ , $\mathbf{R_{3\times 3}}$ and $\mathbf{T}$ are the rotation matrix and translation vector from extrinsic matrix $\mathbf{M_{4\times 4}}$ , and $d_{l}$ is the disparity value from disparity map $D_{\textit{disp}:i}$ at pixel $\mathbf{p_{l}}$ . The operation $||\cdot||$ returns the magnitude of a vector. The resultant depth map $D_{i}$ correspondences to the left image $I_{r1:i}$ of the rectified stereo pair $\mathcal{I}_{r:i}$ .

After a sequence of images has been acquired, rectified and stereo matched, the proposed system attempts to estimate the trajectory of the camera by estimating the motion of the camera between consecutive rectified stereo pairs $\mathcal{I}_{r:i}$ and $\mathcal{I}_{r:i+1}$ . The remainder of this section focuses on the problem of estimating the motion of the camera between rectified stereo pair $\mathcal{I}_{r:i}$ and rectified stereo pair $\mathcal{I}_{r:i+1}$ .

3.2 The cluster feature

Given rectified stereo pairs $\mathcal{I}_{r:i}$ and $\mathcal{I}_{r:i+1}$ , the goal is to estimate camera motion in the form of matrix $\mathbf{M_{i:4\times 4}}$ , a $4\times 4$ rigid transform that maps the coordinate system of the synchronized stereo camera $\{C_{1},C_{2}\}$ when it acquired stereo pair $\mathcal{I}_{r:i}$ to the coordinate system of the same camera $\{C_{1},C_{2}\}$ when it acquired stereo pair $\mathcal{I}_{r:i+1}$ .

The first step in doing this, is to identify a set of pixels $\mathcal{P}_{i}$ in the left image $I_{r1:i}$ of stereo pair $\mathcal{I}_{r:i}$ which can then be matched to the left image $I_{r1:i+1}$ of stereo pair $\mathcal{I}_{r:i+1}$ . Based on these correspondences, the motion of the camera may be estimated between $\mathcal{I}_{r:i}$ and $\mathcal{I}_{r:i+1}$ . In the proposed system, the set of pixels $\mathcal{P}_{i}$ , is not a single entity but a set of $n$ pixel clusters $\mathcal{F}_{k}$ where $\{k\in\mathbb{Z}\ |\ 0\leqslant k<n\}$ , known in this work as cluster features. This section covers the process of finding these cluster features in $I_{r1:i}$ . The steps of the process are shown in Fig. 1.

Figure 1.

Feature detection pipeline. Feature detection starts with determining the magnitude of the gradients within the image and extracting areas with the largest gradients as candidate feature points. Depth filtering is performed to remove features around edges. Finally clustering is performed to identify each cluster feature.

Figure 2.

Candidate features. The original image is on the left, while candidate features map after a threshold has been applied is on the right (candidate features are in white).

Ideal cluster features $\mathcal{F}_{k}$ are made up of multiple neighboring pixels, in which each pixel is associated with significant gradient change. The patterns of gradient change within a cluster feature are useful for matching across images. Cluster features, should not be on object edges, which are identifiable as sharp depth changes in a depth map, since these edges are associated with occlusions that can result in poor matching across images. Clusters should be large enough to allow for the calculation of camera motion of the cluster across images, thus should have at least 7 points, as the pose estimation Eq. (10) has 7 unknowns as is given below. Note that 7 points is a minimal value, ideally clusters should be much larger than this in order to “average” out the effects of outliers.

3.2.1 Gradient detection

Given an image of size $w\times h$ , the image value at pixel $\mathbf{p_{l}}$ is given as $I(u_{l},v_{l})=c_{l}$ , where pixel $\{\mathbf{p_{l}}=[u_{l},v_{l}]^{\top}\ |\ 0\leqslant u_{l}<w\ \land\ 0% \leqslant v_{l}<h\}$ given that $p_{l}\in\mathbb{Z}^{2}$ and $\{c_{l}\in\mathbb{Z}\ |\ 0\leqslant c_{i}<256\}$ . The value $c_{l}$ is the gray-scale value associated with pixel $\mathbf{p_{l}}$ . Note that $\land$ is the logical AND operator. The gradient in the $x$ -direction may be found as:

$\displaystyle\Delta x_{l}=\frac{I(u_{l}-1,v_{l})-I(u_{l},v_{l})}{2}$ (5)

and the gradient in the $y$ -direction may be found as:

$\displaystyle\Delta y_{l}=\frac{I(u_{l},v_{l}-1)-I(u_{l},v_{l})}{2}$ (6)

which leads to an image gradient estimate of:

$\displaystyle\Delta g_{l}=\sqrt{\Delta x_{l}^{2}+\Delta y_{l}^{2}}$ (7)

The resulting map $G(u_{l},v_{l})=\Delta g_{l}$ can then be used to determine the degree of intensity change within various regions within the image. Connected regions of at least seven pixels in size, each with an intensity changes above a particular threshold is likely to be relatively unique within an image, and therefore are desirable as element to attempt to match across images.

3.2.2 Gradient thresholding

After finding the gradient map $G(u_{l},v_{l})=\Delta g_{l}$ , this map can be converted into a candidate feature map as follows:

$\displaystyle F(u_{l},v_{l})=\left\{\begin{array}[]{ll}1,&\text{if }G(u_{l},v_% {l})\geqslant\tau_{\textit{grad}}\\ 0,&\text{otherwise}\\ \end{array}\right.$ (8)

where $\tau_{\textit{grad}}$ is a provided threshold parameter. Figure 2 shows an example of such a map.

Figure 3.

Cluster features. (Left) Original Image. (Right) Cluster features overlaid on the original image. Each different cluster is marked in a different color.

3.2.3 Depth filtering

As stated before, candidate features on the physical edges of objects are undesirable. These are filtered out from the candidate feature map $F(u_{l},v_{l})$ using the depth map $D_{i}(u_{l},v_{l})=d_{l}$ where $d_{l}$ is a depth value in millimeters. The algorithm is as follows:

Filter candidate features on physical edges.[1] FilterFeatures $F$ , $D$ each pixel $[u_{l},v_{l}]^{\top}$ in $F$ $F(u_{l},v_{l})=1$ $\min=D(u_{l},v_{l})$ , $\max=D(u_{l},v_{l})$ j $=-$ 1 to 1 k $=-$ 1 to 1 $d_{l}=D(u_{l}+j,v_{l}+k)$ $\min=\min(d_{l},\min)$ $\max=\max(d_{l},\max)$ $\Delta d=\max-\min$ $\Delta d>\tau_{\textit{depth}}$ $F(u_{l},v_{l})=0$

where $\tau_{\textit{depth}}$ is the maximum allowable depth range in the local neighborhood.

3.2.4 Clustering

The final stage in the proposed feature detection is to cluster features into groups. The proposed strategy is to group features into connected regions based on 8-neighbor connectivity. It is hoped that clusters formed in this way represent areas of high gradient variation within the image. Note that since the described features in the system corresponds closely with edge features, 8-neighbor connectivity was selected in order to identify cluster groups with a diagonal orientation.

Filter candidate features on physical edges.[1] Cluster $F$ each pixel $[u_{l},v_{l}]^{\top}$ in $F$ $F(u_{l},v_{l})=1$ $\textit{cluster}[l]=l$ add $[u_{l},v_{l}]^{\top}$ to feature $\textit{changed}=\textit{true}$ changed $\textit{changed}=\textit{false}$ each pixel $[u_{l},v_{l}]^{\top}$ in featureeach pixel $[u_{j},v_{j}]^{\top}$ in $n(u_{l},v_{l})$ $(u_{j},v_{j})\in\textit{features}$ $id=\min(\textit{cluster}[i],\textit{cluster}[j])$ $\textit{clusterId}\neq\textit{cluster}[l]$ $\textit{changed}=\textit{true}$ $\textit{cluster}[l]=id$ cluster

With respect to the Algorithm 3.2.4, assume that the function $n(u_{l},v_{l})$ returns the 8-neighborhood of the pixel $\mathbf{p_{l}}=[u_{l},v_{l}]^{\top}$ . The output of Algorithm 3.2.4 is a mapping that links each feature to a cluster identified by its $i d$ . Features that map to the same $i d$ are considered to be part of the same cluster.

Figure 3 shows the original image of a scene and the identified clusters in that scene after applying the proposed algorithm. Each cluster is marked with a different color.

3.3 Cluster feature matching

The main purpose of the proposed cluster feature, is to find robust correspondences between consecutive frames, assuming a small amount of rigid motion between frames. Constructing a feature, from multiple pixels, is intended to yield more robust matching than matching single pixels based on feature descriptors or optical flow. Another advantage of the proposed cluster features, is that it consists of seven or more pixels, which means that it is possible to estimate the motion of the cluster feature across images based on its internal correspondences. This has three advantages:

•
In perfectly static scenes, all cluster feature matches should yield similar rigid transforms. Thus, outliers should be easily identifiable.
•
An inability to find a single rigid transform that adequately describes the motion of a feature cluster, can indicate non-rigid motion of the pixels that make up the cluster feature, or inaccurate depth estimates in the depth map. Either way, it identifies the feature as untrustworthy to track against, and therefore the cluster feature can be rejected. This is preferable to rejecting the cluster later on in the tracking process as an outlier.
•
When a small minority of the pixels making up the cluster feature are mismatched, they can be automatically corrected as the points making up the cluster feature are assumed to be a rigid structure (as consistent with a static scene assumption).

Given left images $I_{r1:i}$ and $I_{r1:i+1}$ of two consecutive rectified stereo pairs and assume that the set of $n$ cluster features $\mathcal{F}_{i:k}$ has been found for image $I_{r1:i}$ . The goal is to find corresponding cluster features $\mathcal{F}^{\prime}_{i:k}$ in image $I_{r1:i+1}$ . The first step in doing this, is finding corresponding estimated locations for each of the pixels that make up $\mathcal{F}_{i:k}$ in image $I_{r1:i+1}$ . This is done using the pyramidal implementation of the Lucas Kanade feature tracker [26], found in OpenCV [27]. Poor matches are removed and the remaining points are then used to estimate the camera motion $\mathbf{M_{i:k:4\times 4}}$ between the cluster feature $\mathcal{F}_{i;k}$ and the newly found feature $\mathcal{F}^{\prime}_{i:k}$ . This motion estimate is then used to estimate the actual locations of pixels that make up $\mathcal{F}^{\prime}_{i:k}$ from $\mathcal{F}_{i;k}$ .

The estimation of camera motion begins by recognizing that each cluster feature $\mathcal{F}_{i:k}$ contains a set of $n_{\textit{feat}}$ 2-D feature points $\{\mathcal{F}_{i:k}:\mathbf{f_{l}}=[u_{l},v_{l}]^{\top}\ |\ 0\leqslant l<n_{% \textit{feat}}\}$ . Using the depth map $D_{i}$ it is possible to derive corresponding set of 3-D feature points $\{\mathcal{P}:\mathbf{P_{l}}=[X_{l},Y_{l},Z_{l}]^{\top}\ |\ 0\leqslant l<n_{% \textit{feat}}\}$ using the relation:

$\displaystyle\mathbf{P_{l}}=\frac{Z_{l}}{f}\left[\begin{array}[]{c}u_{l}-c_{x}% \\ v_{l}-c_{y}\\ f\\ \end{array}\right]$ (9)

Assume that the set $\{\mathcal{F}^{\prime}_{k}:\mathbf{f^{\prime}_{l}}=[u^{\prime}_{l},v^{\prime}_% {l}]^{\top}\ |\ 0\leqslant l<n_{\textit{corr}}\}$ of corresponding 2-D pixels has been found in image $I_{r1:i+1}$ . It can then be inferred that these 2-D feature points also correspond to the 3-D $\{\mathcal{P}:\mathbf{P_{l}}=[X_{l},Y_{l},Z_{l}]^{\top}\ |\ 0\leqslant l<n_{% \textit{feat}}\}$ . Given that there are at least seven such points, this sets up a Perspective-n-Point (PnP) pose estimation problem that can be solved using a direct linear transform (DLT). Note that it is assumed that distortion has been removed from $I_{r1:i}$ and $I_{r1:i+1}$ .

Deriving the DLT can follow a similar approach to that of Tsai calibration [22]. Thus, the goal is to solve the system of equations given by:

$\displaystyle\mathbf{U_{M\times 1}}=\mathbf{A_{M\times 7}}\mathbf{L_{7\times 1}}$ (10)

where $U_{M\times 1}$ is a vector of all $n_{\textit{corr}}$ coordinates of value $u_{l}-c_{x}$ . $\mathbf{A_{m\times 7}}$ is a stack of $n_{\textit{corr}}$ rows, with each row representing known quantities as follows:

$\displaystyle\mathbf{A_{row}}=\left[{\begin{array}[]{c}y_{l}X_{l}\\ y_{l}Y_{l}\\ y_{l}Z_{l}\\ y_{l}\\ -x_{l}X_{l}\\ -x_{l}Y_{l}\\ -x_{l}Z_{l}\\ \end{array}}\right]^{\top}$ (11)

given that $x_{l}=u_{l}-c_{x}$ and $y_{l}=v_{l}-c_{y}$ . The matrix $\mathbf{L_{7\times 1}}$ represents a vector of unknowns as follows:

$\displaystyle\mathbf{L_{7\times 1}}=\frac{1}{t_{y}}[r_{1}\,r_{2}\,r_{3}\,t_{x}% \,r_{4}\,r_{5}\,r_{6}]^{\top}$ (12)

where $r_{i}$ are the first two rows of the rotation matrix $\mathbf{R_{3\times 3}}$ and $t_{x}$ and $t_{y}$ are the components of the translation vector $\mathbf{T}$ . The combination of $\mathbf{R_{3\times 3}}$ and $\mathbf{T}$ is the rigid transform between the two views that is being calculated.

Figure 4.
Matched features. Image $I_{r1:i}$ (left) and $I_{r1:i+1}$ (right) with cluster features in color. Note that the locations of cluster features in $I_{r1:i+1}$ have been derived by transforming $\mathcal{P}$ by the pose determined during feature matching and then projected onto the new image plane of $I_{r1:i+1}$ .

As with Tsai calibration, the magnitude of $t_{y}$ may be determined using $\sqrt{r_{4}^{2}+r_{5}^{2}+r_{6}^{2}}=1$ and the sign of $t_{y}$ can be determined with substitution. The final row of the rotation matrix can be found using the cross product and $t_{z}$ can be found using the Moore-Penrose pseudo inverse [28].

After camera motion matrix $\mathbf{M_{i:k:4\times 4}}$ , that maps cluster feature $\mathcal{F}_{i:k}$ to cluster feature $\mathcal{F}^{\prime}_{i:k}$ , has been found, it is possible to find the estimated pixel locations $\mathbf{f^{\prime}_{l}}$ of $\mathcal{F}^{\prime}_{i:k}$ that map to the pixels $\mathbf{f_{l}}$ of $\mathcal{F}_{i:k}$ . To do this, the corresponding 3-D locations $\mathbf{P_{l}}$ of $\mathbf{f^{\prime}_{l}}$ are first transformed to the coordinate space of $\mathcal{F}^{\prime}_{i:k}$ . This is achieved as $\mathbf{\widetilde{P}^{\prime}_{l}}=\mathbf{M_{i:k:4\times 4}}\mathbf{% \widetilde{P}_{l}}$ where $\mathbf{\widetilde{P}^{\prime}_{l}}$ is the 4-D homogeneous form of 3-D $\mathbf{P^{\prime}_{l}}$ and $\mathbf{\widetilde{P}_{l}}$ is the 4-D homogeneous form of 3-D $\mathbf{P_{l}}$ . These points can then be projected to 2-D as $\mathbf{\widetilde{f}^{\prime}_{l:{est}}}=\mathbf{K_{3\times 3}}\mathbf{% \widetilde{P}_{l}}$ where $\mathbf{\widetilde{f}^{\prime}_{l:{est}}}$ is the 3-D homogeneous form of 2-D $\mathbf{f^{\prime}_{l:{est}}}$ .

The degree to which the camera motion estimate of the cluster feature between images accounts for the motion of its pixels can be estimated using an error function that finds the average difference between the corresponding points $\mathbf{f^{\prime}_{l}}$ found through optical flow and those found using the camera motion transform $\mathbf{f^{\prime}_{l:est}}$ as follows:

$\displaystyle E_{\textit{motion}}=\frac{1}{n_{\textit{corr}}}\sum_{l}^{n_{% \textit{corr}}}{\left\|\mathbf{f^{\prime}_{l}}-\mathbf{f^{\prime}_{l:est}}% \right\|}$ (13)

If the value of $E_{\textit{motion}}$ exceeds threshold $\tau_{\textit{match}}$ then the cluster features $\mathcal{F}_{i:k}$ and the corresponding $\mathcal{F}^{\prime}_{i:k}$ are rejected due to a lack of consistency in the pixel data indicating motion within the cluster.
3.4 Pose estimation

The goal of the above section was to estimate the camera motion encoded as matrix $\mathbf{M_{i:4\times 4}}$ between the consecutive rectified stereo frames $\mathcal{I}_{r:i}$ and $\mathcal{I}_{r:i+1}$ .

So far, a feature consisting of clusters of pixels with high gradient variance has been introduced, and a means of finding corresponding clusters in other images has been explained. The process of matching these cluster features has produced a set of $n$ clusters feature motion estimates of the form $\mathbf{M_{i:k:4\times 4}}$ , one for each cluster feature in the image, ignoring the fact that some may have been filtered out due to poor matching.

The goal of this section is to refine this set of estimates into a single estimate. The challenge is, of course, that there is expected motion between the image $\mathcal{I}_{r:i}$ and the image $\mathcal{I}_{r:i+1}$ , which would affect some of the clusters, and therefore these needs to be filtered out. The strategy used to separate the moving clusters from the stationary clusters is to apply the heuristic that the motion estimate, with the most consensus amongst the set of estimates, is most likely the true one. This is based on the assumption that the scene contains a set of moving objects with a background that comprises of significantly more pixels than any individual object in the scene.

The main strategy, used to find the motion with the most consensus, is to group motion estimates based on their similarity to each other, and find the group with the most members. In order to do this, a variant of the RANSAC [29] algorithm is used. Firstly, each motion estimate is converted from a rotation-translation matrix to its associated Lie algebra $\mathfrak{se}(3)$ , which is a six elements vector of the form $\mathbf{\xi_{i}}=[\omega_{i:1},\omega_{i:2},\omega_{i:3},t_{i:x},t_{i:y},t_{i:% z}]^{\top}$ . Two motion estimates of this form may be compared by finding the Euclidean norm of the difference between the two vectors. Two motion estimates may be considered “similar”, if their Euclidean difference is less than $\tau_{\textit{pose}}$ . The RANSAC style algorithm selects a cluster feature $\mathcal{F}_{i:k}$ from the set of $n$ cluster features at random. It then computes the number of motion estimates within the entire set that are “similar” to the selected one. This is repeated a small number times (see [29] for an explanation on determining the appropriate number of iterations), and the group $\mathcal{F}_{\max}$ with the largest number of members is retained as the set representing the most likely estimate of camera motion.

The final pose $\mathbf{M_{i:3\times 4}}$ is determined from the group $\mathcal{F}_{\max}$ , using the Levenberg-Marquardt algorithm [30] to minimize the $E_{\textit{motion}}$ estimate across the entire group.

Figure 5.

Pose estimate. The original image at time $t$ (left) and an image at time $t+1$ warped to match the image at time $t$ (right). The black pixels are missing depth values or are out of the image plane of image $t+1$ .

4. Experimental methodology

In this work, a new feature for ego-motion tracking was introduced, called the cluster feature. The goal of the experimentation proposed below is to verify that there is a significant advantage of using the proposed cluster feature, in the context of moving urban environments, over other state-of-the-art algorithms. This section introduces the equipment and evaluation metric used. The results of the experiments are then covered in Section 5.

4.1 Equipment

The experiments performed in this section were conducted using two GoPro Hero 3+ Black Edition camera with in-built hardware synchronization. These cameras were housed in a waterproof stereo box manufactured by GoPro (see Fig. 6) with a 33 mm baseline. The cameras were set to capture images with size 1720 $\times$ 1080 pixels, on the wide-angle setting of video mode at 30 frames per second.

Figure 6.

GoPro camera. The GoPro Hero 3+ camera in the waterproof stereo case used within this section.

Experiments were executing on an Intel ${}^{\@setsize{\scriptsize}{8pt}{\viipt}{\@viipt}\textregistered}$ Core ${}^{\text{TM}}$ i7-5500U Processor with 16 GB RAM.

Figure 7.

Image database. Selected stills from each of the twelve video sequences used for the presented experiments. Note that this database is expected to be available at http://www.ivs.auckland.ac.nz/ivs-applications/home.php from early 2019.

4.2 Parameters

The proposed algorithm requires three provided threshold parameters for noise filtering. These three parameters are $\tau_{\textit{grad}}$ defined in Section 3.2.2 for isolating large gradients, $\tau_{\textit{depth}}$ defined in Section 3.2.3 indicating a reliable depth range for features, and $\tau_{\textit{match}}$ defines in Section 3.3 for the specification of acceptable error tolerance of pose estimation. For the purpose of experimentation, these parameters were set as follows:

•
$\tau_{\textit{grad}}$ : While it was found experimentally that setting this value to a reasonably large number (say 100 for the given 8-bit image) worked reasonably well across a wide set of images, it needs to be acknowledged that a proper theoretical basis for choosing this value is required for robust performance. It was found that using the thresholds from Adaptive Thresholding Canny [31] was one such way to determine $\tau_{\textit{grad}}$ based on actual image content. Assuming a video sequence in a relatively constant environment, it was found that $\tau_{\textit{grad}}$ could be reasonably reliably determined from the first few frames and then reused across the rest.
•
$\tau_{\textit{depth}}$ : A key issue with depth acquired from binocular stereo is that its resolution gets more coarse as one moves further from the camera. Since it was decided, in this work, that an appropriate measure of accuracy should be in millimeters, measurements with a resolution greater than 1 mm should be filtered out. This is the purpose of $\tau_{\textit{depth}}$ . The value $\tau_{\textit{depth}}$ is therefore determined from the calibration parameters of the binocular stereo cameras used in the system.
•
$\tau_{\textit{match}}$ : Estimating the motion of a cluster is reliant on the accuracy of feature location and matching. Unfortunately this is never perfect, therefore a mechanism is needed to identify situations when the system has failed. One way of doing this is to determine how well the estimated motion matches the measurements, since it is difficult to fit a consistent model to a set of bad measurements. The same can be said for a cluster that does not have a single consistent motion (lets say the cluster spans across two scene objects with different motions). As accuracy is measured in pixels and a good motion model should have a sub-pixel accuracy, $\tau_{\textit{match}}$ was set to 1.

4.3 Algorithms

There are two state-of-the-art algorithms representing the general variety ego-motion estimation algorithms available: Feature-based and dense pose estimation.

•
Feature-based algorithms: A common approach to estimating camera motion is to use feature point matching. The basic idea is that two frames are provided to the algorithm with a significant amount of overlap. Our implementation finds corresponding feature points using the Fast Feature detector [32] and Lucas Kanade optical flow [26]. The correspondences are then converted into 3-D points (using provided depth maps) leading to a pose estimation by alignment using iterative least-squares based on Levenberg Marquardt. Feature point outliers are filtered using a RANSAC based derivation of the fundamental matrix [33]. Many elements of this implementation mirror state-of-the-art implementations such as FOVIS [34].
•
Dense pose estimation: Steinbrücker et al. [35] provide a fast accurate implementation of a dense pose estimation algorithm from RGB-D cameras [35]. While this algorithm was strictly developed for active depth sensor cameras such as Microsoft’s Kinect, this algorithm was found to work reasonably well on depth-maps derived from stereo matching. It is included in this experimentation as dense pose estimation and is quite popular at the moment, with several published works claiming that dense strategies give superior performance over feature-based approaches [36].

4.4 Databases

In order to test robustness to scene movement, a collection of twelve video sequences were captured. Each of these video sequences featured moving elements. Care was taken to select video sequences from a variety of scenes including indoor scenes, outdoor scenes, scenes with many man-made structures and scenes that consisted of natural objects (such as trees, plants, landscape etc.). Stills from each of the videos are shown in Fig. 7.

It should be noted that there already exists several available databases of video sequences, for the purpose of validating visual odometry algorithms. However the authors of this work were particularly interested in scenes with various types of motion, which is typically not the focus in such videos. Therefore it was decided to create a new database of videos specifically for this work. This database is expected to available is expected to be available at http://www.ivs. auckland.ac.nz/ivs-applications/home.php from early 2019, with future additions as research into this area continues.

4.5 Assessement

Experiments were generally performed by processing the captured video sequences with the algorithms. The main measure of accuracy used was photometric error. Photometric error measures the quality of motion estimate $\mathbf{M_{i:4\times 4}}$ , based on how effective it is at warping image $I_{r1:i}$ to approximate the next frame in the sequence at $I_{r1:i+1}$ in conjunction with depth map $D_{i}$ .

Intuitively, the smaller the photometric error, the more accurate the pose estimate. Naturally, there are other factors as well including lighting changes, movement in the scene and depth map accuracy. However, the approach should still be useful when comparing the results of different algorithms using the same data. Indeed, an algorithm that is able to consistently produce smaller photometric errors than another on the same data is clearly more accurate, despite the fact that it may not achieve “perfect” photometric scores.

The derivation of photometric error may be formulated starting with image $I_{r1:i}$ and $I_{r1:i+1}$ in a sequence with known depth map $D_{i}$ associated with $I_{r1:i}$ . If the pose between $I_{r1:i}$ and $I_{r1:i+1}$ is estimated to be $\mathbf{M_{i:4\times 4}}$ , the photometric error between these two images may be calculated as:

$\displaystyle E_{p}=\sum_{\Omega}\|I_{r1:i}(u_{l},v_{l},1){}-I_{r1:i+1}(% \mathbf{K_{3\times 3}}\mathbf{M_{i:3\times 4}}\mathbf{P_{l}})\|^{2}$ (14)

where $\Omega$ represents the set of all the pixels within the image and $\mathbf{M_{i:3\times 4}}$ is derived from $\mathbf{M_{i:4\times 4}}$ by removing the last row. The value $P_{l}$ is the 3-D location of a pixel within the image $I_{r1:i}$ found using the depth map $D_{i}$ , the 2-D pixel location $\mathbf{p_{l}}=[u_{l},v_{l}]^{\top}$ and Eq. (9).

5. Results

5.1 Database sequence photometric errors

The first set of experiments were performed against scenes with moving elements to test the claim that the proposed algorithm was more robust in scenes with moving elements that other algorithms. The results are shown in Table 1.

Table 1
Photometric error (avg $\pm$ std) in moving scenes

Video #	RGB-D [35]	Features [32, 26]	Proposed
1	9.1 $\pm$ 16.2	11.2 $\pm$ 20.6	8.4 $\pm$ 15.3
2	20.5 $\pm$ 27.5	12.6 $\pm$ 13.1	11.9 $\pm$ 11.7
3	28.1 $\pm$ 33.7	22.2 $\pm$ 28.1	20.1 $\pm$ 26.6
4	17.6 $\pm$ 27.5	15.1 $\pm$ 25.8	14.7 $\pm$ 26.0
5	23.47 $\pm$ 32.7	18.9 $\pm$ 30.6	15.4 $\pm$ 27.6
6	20.6 $\pm$ 36.2	19.1 $\pm$ 34.4	18.7 $\pm$ 35.3
7	25.7 $\pm$ 37.4	21.7 $\pm$ 34.8	10.1 $\pm$ 20.4
8	18.7 $\pm$ 23.1	26.8 $\pm$ 34.5	17.9 $\pm$ 26.1
9	27.5 $\pm$ 37.5	11.8 $\pm$ 19.5	8.7 $\pm$ 12.7
10	45.5 $\pm$ 51.1	25.4 $\pm$ 34.4	14.8 $\pm$ 22.5
11	43.2 $\pm$ 42.1	24.6 $\pm$ 31.1	12.8 $\pm$ 20.1
12	3.5 $\pm$ 7.6	5.3 $\pm$ 11.1	3.6 $\pm$ 7.8

The results show that the proposed algorithm was the top performing algorithm in all videos except for the last one. The last video featured a mostly static indoor scene with two slow-moving human subjects – footage that is well suited to the direct matching algorithm, and thus it was the top performer in this sequence. However the direct matching algorithm appeared to be the most susceptible to motion in the images, while the feature-based approach was generally more robust, while still deteriorating significantly with increased level of motion in the images. The proposed algorithm was significantly better in videos 8 (forward motion scene with a women walking directly in front of the camera), 10 (large amount of moving foliage), 11 (a close-up of a bushes) which were the videos with the most motion.

5.2 Timings

Timings per frame are provided per the tested algorithms in Table 2. While the proposed algorithm is slower than the RGB-D algorithms, its timing is similar to the feature tracking approach. Note that these timings only include the time taken to perform visual odometry, and do not include the preprocessing of the images (i.e. calibration, image acquistion, distortion removal, rectification, stereo matching, and disparity-to-depth conversion). It should be noted that the same preprocessing is required for each of the three presented algorithms.

Table 2
Average time per ego-motion estimation strategy

Algorithm	Time in milliseconds
RGB-D tracking	160 $\pm$ 13
Feature tracking	421 $\pm$ 52
Proposed	501 $\pm$ 68

5.3 Road walk experiment

A road walk experiment of about 200 m was performed along a segment of Auckland, New Zealand’s Karangahape Road. Experiments were conducted on a Friday afternoon, where Karangahape Road is a fairly busy street in the Auckland CBD. The experiment was conducted in order to test the proposed system in conditions of moving traffic and pedestrians.

Figure 8.

Karangahape walk. Selected stills from the walk down Karangahape Road showing the moving traffic and pedestrians that were present in the scene.

Figure 8 shows some of the moving elements that were present in the scene. The sequence of 4628 calibrated stereo frames were given as input to the proposed algorithm, which estimated the trajectory of the person walking with the camera. The superimposed top-view of the path is shown in Fig. 9.

Table 3

Details of the road walk experiment

Distance	216.55 m (google maps)
	205.19 m (proposed system)
Frames	4628
Camera	GoPro Hero 3+ Stereo Black Edition
Photometric error	10.3 $\pm$ 19.8
Walk time	$\pm$ 2 minutes 30 seconds
Processing time	38 minutes 29 seconds

Figure 9.

Karangahape path. The path (dotted line) acquired using the proposed system superimposed on a map of the walk acquired from Google Maps.

Figure 10.

Roadside statue. An image taken from a 300 stereo frame sequence of a statue sitting on the side of Karangahape Road.

Figure 11.

Model top view. A top view of the 3-D reconstruction of the camera. The path of the camera, estimated by the proposed algorithm, is shown as a red dotted line.

Figure 12.

Model front and back views. A front view (top) and a back view (bottom) of the generated model is shown.

Table 3 shows the details of the road walk experiments. Note that the time of 38 minutes 29 seconds is significantly longer than the actual walk time of 2 minutes 30 seconds. The main reason for this is that the algorithm currently performs at a frame every half-a-second while the camera was capturing at 30 frames per seconds. However, as seen in Section 5.2, our implementation is only slightly slower than the feature point approach. It should also be noted that the algorithm is executed on the CPU without much optimization and without parallelization. However clearly, if the goal if this algorithm is to support real-time applications such as driver-less car navigation, an important area of future work is to establish whether the algorithm can be sped up significantly without a significant loss in accuracy.

5.4 Scene reconstruction

As a final test, it is important to verify that the proposed strategy for Visual Odometry is able to support 3-D reconstruction. While the primary focus of the work presented in this paper has been to track a calibrated stereo camera in a moving scene, moving objects still pose a problem in a 3-D reconstruction pipeline after odometry, since it is hard to merge a sequence of point clouds containing moving objects. For this reason, it was decided to focus the 3-D reconstruction test on a relatively static scene. Figure 10 is a frame taken from a 300 stereo frames sequence of a roadside statue. The images were acquired using the GoPro Hero 3+ Black Edition camera.

The proposed algorithm was used to estimate the trajectory of the stereo camera as it captured the 3-D reconstruction. Stereo frames were then merged together within a Truncated Signed Distance Function [37]. The marching cubes algorithm [38] was used to extract the final mesh. The top view of the resulting 3-D model is shown in Fig. 11 and the front and back views are shown in Fig. 12.

6. Conclusions

The aim of this work was to improve ego-motion estimation. The proposed approach was to introduce a new, more robust feature which was named the “cluster feature”.

The cluster feature is composed of multiple feature points which lead to its improved robustness. This composition also allows to correct small mismatch errors and to estimate its match correctness with respect to both its internal consistency (the consistency of the matches of the features that make up the cluster feature) and external consistency (the degree to which this cluster feature agrees with other cluster features within the image). Finally, the cluster feature combines feature based “reprojection error” metric with “photometric error” metric and yields a compromise between the two approaches.

Experimental results demonstrated that the proposed feature is capable of performing visual odometry in scenes with moderate amounts of movement. In this realm, the proposed algorithm outperforms the other state-of-the-art approaches tested.

7. Future work

The approach in this work shows promise as a solution to the problem of determining the motion of a camera from images in scenes with a variety of moving objects. However there is clearly a lot more research to be done before the proposed solution is a viable solution for modern applications, such as augmented reality and real-time 3D reconstruction algorithms. Speed is naturally the foremost concern, and already the authors are working on ways to speed up algorithms using parallelism, SIMD and GPU technologies. There is also some work going into a faster cluster feature matching algorithm that is non-iterative. Another project is using machine learning to establish whether clustering and matching using models acquired through machine learning is a viable fast solution.

The assessment of presented in this work is at the level of a proof of concept, using fairly good quality cameras without much motion blur or illumination artifacts and a single metric with respect to accuracy. A much more thorough investigation needs to be undertaken with a much larger database of images captured with a variety of different camera types and conditions. The algorithm needs to be assessed with several different types of evaluation metrics in both the image plane (in pixels) and also in the scene domain (in metric units such as millimeters) such as Hausdorff distance [39]. Such a study could answer research questions about the robustness of the proposed algorithm in difficult conditions and establish the types of situations that favor the proposed algorithm and those that don’t. Also the effect of calibration quality can be assessed on the algorithm, along with the effect of various types of distortion models and noise models.

A final area of interest is the 3D reconstruction of moving scenes. The 3D reconstruction presented in Section 5.4 was chosen to be static because merging moving elements is difficult. However there are several ideas as to how this problems can be avoided or overcome. One such idea is to reconstruct only static elements and leave out moving elements, this would entail generating a 3D reconstruction of a street by ignoring the pedestrians and vehicles for example. Another idea is to capture scenes with known moving objects and use motion models to predict and deform point clouds so that motion across frames may be reconciled.

Footnotes

https://www.intel.com/content/www/us/en/architecture-and-technology/realsense-overview.html.

References

Golban

Istvan

Nedevschi

. Stereo based visual odometry in difficult traffic scenes. In: Intelligent Vehicles Symposium (IV), 2012 IEEE. IEEE; 2012. pp. 736–741.

Howard

. Real-time stereo visual odometry for autonomous ground vehicles. In: Intelligent Robots and Systems, 2008. IROS 2008. IEEE/RSJ International Conference on. IEEE; 2008. pp. 3946–3952.

García-García

Sotelo

Parra

Fernández

Naranjo

Gavilán

. 3D visual odometry for road vehicles. Journal of Intelligent and Robotic Systems. 2008; 51(1): 113–134.

Engel

Sturm

Cremers

. Semi-dense visual odometry for a monocular camera. In: Computer Vision (ICCV), 2013 IEEE International Conference on. IEEE; 2013. pp. 1449–1456.

Forster

Pizzoli

Scaramuzza

. SVO: Fast semi-direct monocular visual odometry. In: Robotics and Automation (ICRA), 2014 IEEE International Conference on. IEEE; 2014. pp. 15–22.

Kitt

Geiger

Lategahn

. Visual odometry based on stereo image sequences with ransac-based outlier rejection scheme. In: Intelligent Vehicles Symposium (IV), 2010 IEEE. IEEE; 2010. pp. 486–492.

Comport

Malis

Rives

. Accurate quadrifocal tracking for robust 3d visual odometry. In: Robotics and Automation, 2007 IEEE International Conference on. IEEE; 2007. pp. 40–45.

Kitt

Moosmann

Stiller

. Moving on to dynamic environments: Visual odometry using feature classification. In: Intelligent Robots and Systems (IROS), 2010 IEEE/RSJ International Conference on. IEEE; 2010. pp. 5551–5556.

Lepetit

Moreno-Noguer

Fua

. Epnp: An accurate o (n) solution to the pnp problem. International Journal of Computer Vision. 2009; 81(2): 155.

10.

Arun

Huang

Blostein

. Least-squares fitting of two 3-D point sets. IEEE Transactions on Pattern Analysis and Machine Intelligence. 1987; 1(5): 698–700.

11.

Dubbelman

Groen

. Bias reduction for stereo based motion estimation with applications to large scale visual odometry. In: Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on. IEEE; 2009. pp. 2222–2229.

12.

Besl

McKay

. Method for registration of 3-D shapes. In: Sensor Fusion IV: Control Paradigms and Data Structures. vol. 1611. International Society for Optics and Photonics; 1992. pp. 586–607.

13.

Rusinkiewicz

Levoy

. Efficient variants of the ICP algorithm. In: 3-D Digital Imaging and Modeling, 2001. Proceedings. Third International Conference on. IEEE; 2001. pp. 145–152.

14.

Izadi

Kim

Hilliges

Molyneaux

Newcombe

Kohli

, et al. KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera. In: Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology. ACM; 2011. pp. 559–568.

15.

Baker

Matthews

. Lucas-kanade 20 years on: A unifying framework. International Journal of Computer Vision. 2004; 56(3): 221–255.

16.

Klose

Heise

Knoll

. Efficient compositional approaches for real-time robust direct visual odometry from RGB-D data. In: Intelligent Robots and Systems (IROS), 2013 IEEE/RSJ International Conference on. IEEE; 2013. pp. 1100–1106.

17.

Liu

Zhang

. Robust stereo visual odometry using improved ransac-based methods for mobile robot localization. Sensors. 2017; 17(10): 2339.

18.

Kerl

Sturm

Cremers

. Robust odometry estimation for RGB-D cameras. In: Robotics and Automation (ICRA), 2013 IEEE International Conference on. IEEE; 2013. pp. 3748–3754.

19.

Suykens

Vandewalle

. Least squares support vector machine classifiers. Neural Processing Letters. 1999; 9(3): 293–300.

20.

Yao

Zhang

Song

Zhang

. Robust RGB-D visual odometry based on edges and points. Robotics and Autonomous Systems. 2018; 107: 209–220.

21.

Zhang

. A flexible new technique for camera calibration. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2000; 22(11): 1330–1334.

22.

Tsai

. A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses. IEEE Journal on Robotics and Automation. 1987; 3(4): 323–344.

23.

Gee

Delmas

Stones-Havas

Sinclair

Van Der Mark

, et al. Tsai camera calibration enhanced. In: Machine Vision Applications (MVA), 2015 14th IAPR International Conference on. IEEE; 2015. pp. 435–438.

24.

Fusiello

Trucco

Verri

. A compact algorithm for rectification of stereo pairs. Machine Vision and Applications. 2000; 12(1): 16–22.

25.

Nguyen

Chan

Delmas

Gimel’farb

. Symmetric dynamic programming stereo using block matching guidance. In: Image and Vision Computing New Zealand (IVCNZ), 2013 28th International Conference of. IEEE; 2013. pp. 88–93.

26.

Bouguet

. Pyramidal implementation of the affine lucas kanade feature tracker description of the algorithm. Intel Corporation. 2001; 5(1-10): 4.

27.

Bradski

Kaehler

. Learning OpenCV: Computer vision with the OpenCV library. “O’Reilly Media, Inc.”; 2008.

28.

Golan

. Moore-penrose pseudoinverses. In: The Linear Algebra a Beginning Graduate Student Ought to Know. Springer; 2012. pp. 441–452.

29.

Fischler

Bolles

. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. In: Readings in computer vision. Elsevier; 1987. pp. 726–740.

30.

Moré

. The Levenberg-Marquardt algorithm: Implementation and theory. In: Numerical analysis. Springer; 1978. pp. 105–116.

31.

Jie

Ning

. An improved adaptive threshold canny edge detection algorithm. In: 2012 International Conference on Computer Science and Electronics Engineering. IEEE; 2012. pp. 164–168.

32.

Rosten

Drummond

. Fusing points and lines for high performance tracking. In: Computer Vision, 2005. ICCV 2005. Tenth IEEE International Conference on. vol. 2. IEEE; 2005. pp. 1508–1515.

33.

Fathy

Hussein

Tolba

. Fundamental matrix estimation: A study of error criteria. Pattern Recognition Letters. 2011; 32(2): 383–391.

34.

Huang

Bachrach

Henry

Krainin

Maturana

Fox

, et al. Visual odometry and mapping for autonomous flight using an RGB-D camera. In: Robotics Research. Springer; 2017. pp. 235–252.

35.

Steinbrücker

Sturm

Cremers

. Real-time visual odometry from dense RGB-D images. In: Computer Vision Workshops (ICCV Workshops), 2011 IEEE International Conference on. IEEE; 2011. pp. 719–722.

36.

Newcombe

Izadi

Hilliges

Molyneaux

Kim

Davison

, et al. KinectFusion: Real-time dense surface mapping and tracking. In: Mixed and Augmented Reality (ISMAR), 2011 10th IEEE International Symposium on. IEEE; 2011. pp. 127–136.

37.

Curless

Levoy

. A volumetric method for building complex models from range images. In: Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques. ACM; 1996. pp. 303–312.

38.

Lorensen

Cline

. Marching cubes: A high resolution 3D surface construction algorithm. In: ACM Siggraph Computer Graphics. vol. 21. ACM; 1987. pp. 163–169.

39.

Zhang

Han

Zou

Chen

. An efficient approach to directly compute the exact Hausdorff distance for 3D point sets. Integrated Computer-Aided Engineering. 2017; 24(3): 261–277.

Stereo camera visual odometry for moving urban environments

Abstract

Keywords

1. Introduction

2. Literature review

3. Proposed approach

3.1 Prerequisites

3.2.4 Clustering

3.3 Cluster feature matching

4.1 Equipment

4.5 Assessement

5.1 Database sequence photometric errors

Table 1 Photometric error (avg ± std) in moving scenes

Table 2 Average time per ego-motion estimation strategy

6. Conclusions

7. Future work

Footnotes

References

Table 1
Photometric error (avg $\pm$ std) in moving scenes

Table 2
Average time per ego-motion estimation strategy