Lidar and radar fusion for real-time road-objects detection and tracking

Abstract

In this paper, based on the fusion of Lidar and Radar measurement data, a real-time road-Object Detection and Tracking (LR_ODT) method for autonomous driving is proposed. The lidar and radar devices are installed on the ego car, and a customized Unscented Kalman Filter (UKF) is used for their data fusion. Lidars are accurate in determining objects’ positions but significantly less accurate on measuring their velocities. However, Radars are more accurate on measuring objects velocities but less accurate on determining their positions as they have a lower spatial resolution. Therefore, the merits of both sensors are combined using the proposed fusion approach to provide both pose and velocity data for objects moving in roads precisely. The Grid-Based Density-Based Spatial Clustering of Applications with Noise (GB-DBSCAN) clustering algorithm is used to detect objects and estimate their centroids from the lidar and radar raw data. Then, the estimation of the object’s velocity as well as determining its corresponding geometrical shape is performed by the RANdom SAmple Consensus (RANSAC) algorithm. The proposed technique is implemented using the high-performance language C $++$ and utilizes highly optimized math and optimization libraries for best real-time performance. The performance of the UKF fusion is compared to that of the Extended Kalman Filter fusion (EKF) showing its superiority. Simulation studies have been carried out to evaluate the performance of the LR_ODT for tracking bicycles, cars, and pedestrians.

Keywords

Sensor fusion Kalman filter object detection object tracking ADAS autonomous driving

1. Introduction

Improving safety, lowering road accidents, boosting energy efficiency, enhancing comfort, and enriching driving-experience are the most important driving forces behind equipping present-day cars with Advanced Driving Assistance Systems (ADAS) [1, 2]. Many ADAS functions represent incremental steps toward a hypothetical future of safe fully autonomous cars [3, 4, 5, 6, 7, 8, 9, 10, 11, 12].

A critical component of the various ADAS features that are also highly required in autonomous cars is the recognition and accurate assessment of the surroundings [5, 6, 7, 8]. This component depends on data observed from sensors mounted on the ego car [9]. If there is an object close by, it is of interest to know where that object is, what the object’s velocity is, and if the object can be described by a plain geometric shape [12]. Lidar and radar are ones of the sought-after sensors for exploiting in ADAS and autonomous-car features [13]. A lidar always returns many concentrated detection points (point-cloud) that describe each detected object [14, 15]. Likewise, a radar often returns multiple detections per target but not as dense as a lidar [16]. This means that it is necessary to group detections originating from the same target, i.e. to cluster the detections, to obtain information about the surroundings [16, 17].

The ego car equipped with a lidar and radar receives a collection of raw data of sensors measurements that include information of detected road objects. Then, the proposed LR_ODT method employs a two-step approach to find and identify these road objects within the received data. The first step is to coarse cluster the lidar/radar raw data separately to detect objects within using the Grid-Based Density-Based Spatial Clustering of Applications with Noise (GB-DBSCAN) algorithm [18]. Accordingly, each object is then represented with its core point (centroid). The second step is the estimation of the object’s cluster velocity as well as determining its corresponding geometrical shape, which is performed using the RANdom SAmple Consensus (RANSAC) iterative algorithm [19].

As mentioned before, it is mandatory to have continuous, precise, and accurate velocity and position information about the objects surrounding the ego car. In this paper, this is accomplished by combining data from lidar and radar sensors.

Recent lidars have a large range (up to 200 m) and a wide field of view, and can thus track objects even at big distances (necessary at high speeds) as well as in curves (i.e. very accurate in position measurement) [13]. Their main drawback is that they completely lack dynamic information about the detected objects (velocity measurement). Radar sensors, on the other hand, have a relatively narrow field of view and reduced angular resolution (less accurate in position measurement), but use the Doppler effect to directly provide velocity information. The fusion of the data from both sensors can thus benefit from the combination of their merits [20].

Accordingly, sensor fusion of lidar and radar that combines the strengths of both sensor types is a logical step. This step has been investigated earlier in the literature, with promising prospects in the automotive industry [21].

As an early endeavor, Gohring et al. [20] apply lidar-radar fusion for the application of car following on highways based on the Kalman Filter (KF) [22]. To test the performance of their fusion technique, the authors have formed the ground truth by computing the mean square errors of relative distances and velocities in a highway-tracking scenario using a least-squares polynomial approximation of sensors data.

Moreover, to improve the perceived model of the environment, Chavez-Garcia et al. [23] include the objects’ classification from multiple sensors (lidar, radar, and camera) detections as a key component of the object’s representation and the perception process that is based on a framework derived from evidence theory. The fusion approach is tested using real data from different driving scenarios and focusing on four objects of interest: pedestrian, bike, car, and truck.

For tracking multiple objects, Rangesh et al. [24] propose a modular framework capable of accepting object proposals from different sensor modalities (cameras and lidars) and fuse them. The approach is tested on real-world highway data, showing its effectiveness to track objects through entire maneuvers around the ego-vehicle.

For obstacle detection, Hajri et al. [25] employ the global nearest neighbor standard filter (GNN) on the fused lidar/radar sensors data for associating new measurements with the underlying observed objects. The benefits of data fusion comparing with the use of a single sensor are illustrated through several tracking scenarios (on a highway and a bend).

The emphasis of this paper is on the data fusion between lidar and radar for road-objects’ tracking. The proposed technique is the acronym: a Lidar/Radar-based road-Object Detection and Tracking technique (LR_ODT). According to the presented state of the art, two different general fusion methods can be distinguished and both were applied for lidar and radar: Kalman filter and evidence theory. However, approaches that apply any of these methods agree on the main steps. Meanwhile, the technique that has been adopted in this work is based on using Kalman filters. Despite the previously mentioned works [26], the literature still clearly lacks the investigation of employing the UKF [27, 28] for lidar/radar fusion while applied for road-object tracking for autonomous driving. Therefore, this paper will mainly focus on the tailoring and implementation of UKF for tracking various road objects. The UKF design will be validated by tracking three road-objects: car, bicycle, and pedestrian. A quantitative comparison between the performance of the UKF versus that of the EKF [29, 30], as well as a quantitative comparison between the performance of the LR_ODT with and without the employment of lidar/radar fusion, are carried out.

The contribution of this paper can be enumerated as follows:

1.
Tailoring the UKF as well as the EKF algorithms to fuse multiple radars and lidars data to achieve more accurate pose data for moving objects around the ego car, proving that the UKF-based method has better performance but the EKF-based method is less computationally demanding.
2.
Employing a high-order-generic-object-motion model (5 state variables that suits the most common road-objects: car, bicycle, and pedestrian) in the development of the UKF and EKF to generate more accurate estimates and improve the overall performance.
3.
Carrying out a quantitative comparative study between the sensor fusion performance using EKF and UKF using the same use cases.
4.
Evaluating the gain of fusion by testing the UKF on three different cases (lidar $+$ radar, lidar only, and radar only).
5.
Evaluating the real-time performance of both the EKF and UKF on a moderate computational platform.
6.
Employing the GB-DBSCAN clustering algorithm to detect potential objects from the lidar/radar raw data, and finding their centroids.
7.
Employing the RANSAC algorithm and object proposals for bicycle, car, and pedestrian for estimation of the detected object’s cluster velocity as well as determining its corresponding geometrical shape.

2. Kalman filter overview

The Kalman filter [22] is a system of equations working together to form a predictor-update cyclic optimal estimator that minimizes the estimated error covariance. The KF estimates the state $x\in R^{n}$ given the measurement $z\in R^{m}$ of a discrete-time controlled process that is modeled by the following set of linear stochastic difference equations:

$\displaystyle x_{k}=Fx_{k-1}+Bu_{k}+{\nu}_{k}$ $\displaystyle z_{k}=Hx_{k}+{\omega}_{k}$ $\displaystyle{\nu}_{k}\sim\mathcal{N}(0,Q_{k})$ $\displaystyle{\omega}_{k}\sim\mathcal{N}(0,R_{k})$ (1)

where $F$ is the process state transition model, $B$ is the control-input model, $u_{k}$ is the control input, $H$ is the measurement model, ${\nu}_{k}$ is the process white noise which is the Gaussian distribution ( $\mathcal{N}$ ) with zero mean and covariance matrix $Q_{k}$ , and ${\omega}_{k}$ is the measurement white noise which is the Gaussian distribution ( $\mathcal{N}$ ) with zero mean and covariance matrix $R_{k}$ .

The KF estimation process works in two steps:

The predication step: estimates the next state as follows:

$\displaystyle{\hat{x}}_{k}=Fx_{k-1}+Bu_{k}$ $\displaystyle{\hat{P}}_{k}=FP_{k-1}F^{\rm T}+Q_{k}$ (2)

The measurement update step: works as follows

$\displaystyle{\hat{y}}_{k}=z_{k}-H{\hat{x}}_{k}$ $\displaystyle S_{k}=H{\hat{P}}_{k}H+R_{k}$ $\displaystyle K_{k}={\hat{P}}_{k}H^{\rm T}S^{-1}_{k}$ $\displaystyle x_{k}={\hat{x}}_{k}+K_{k}{\hat{y}}_{k}$ $\displaystyle P_{k}=(I-KH){\hat{P}}_{k}$ (3)

where $P_{k}$ is the KF process estimate covariance, $K_{k}$ is the KF gain, and $S_{k}$ is the measurement covariance matrix.

However, the above equations are only limited to linear processes, and accordingly, it is not suitable to the radar measurement process which is inherently nonlinear. Therefore, the extended Kalman filter [29] is introduced. The EKF estimation process is represented by Eq. (2) instead of Eq. (2) as follows:

$\displaystyle{\hat{x}}_{k}=f(x_{k-1},u_{k})+{\nu}_{k}$ $\displaystyle{\hat{z}}_{k}=h(x_{k-1})+{\omega}_{k}$ (4)

where $f(\cdot)$ and $h(\cdot)$ are nonlinear functions and can be linearized around an arbitrary operating point $\mu$ using the truncated Taylor series expansion as follows:

$\displaystyle f(x)=f(\mu)+\frac{\partial f(\mu)}{\partial x}(x-\mu)$ $\displaystyle h(x)=h(\mu)+\frac{\partial f(\mu)}{\partial x}(x-\mu)$ (5)

The derivatives of $f(\cdot)$ and $h(\cdot)$ with respect to $x$ are called Jacobians. The $F_{j}$ and $H_{j}$ Jacobians are calculated as in Eq. (2) while taking the form of matrices of orders $n\times n$ and $m\times n$ respectively. These matrices contain all the partial derivatives with respect to each state variable.

$\displaystyle F_{j}=\frac{\partial f(\mu)}{\partial x}$ $\displaystyle H_{j}=\frac{\partial h(\mu)}{\partial x}$ (6)

Accordingly, the EKF process is represented as well by the prediction and update Eqs (1) and (2) respectively after replacing $F$ with $f(x_{k-1},u_{k})$ , $H$ with $h(x_{k-1})$ in Eq. (1), and $F$ with $F_{j}$ , $H$ with $H_{j}$ in Eq. (2).

Due to the EKF is using only the first-order derivative in the linearization process and ignoring the higher-order terms, errors are accumulated in the state and covariance estimation. The unscented Kalman filter is introduced [27] to overcome this limitation. The UKF is a derivative-free alternative to EKF that uses a deterministic sampling approach. The UKF utilizes as well the predict-update two-step process, however, they are now augmented with further steps like generation and prediction of sigma points as shown in Fig. 1.

Figure 1.

UKF roadmap.

In the UKF process, the state Gaussian distribution is represented using a minimal set of carefully chosen sample points, called sigma points. $n_{x}=2n+1$ sigma points are selected based on the following rule:

$\displaystyle X_{k}=\Big{[}x_{k}x_{k}+\sqrt{(\lambda+n_{x})P_{k}}x_{k}-\sqrt{(% \lambda+n_{x})P_{k}}\Big{]}$ (7)

where $X_{k}$ is the sigma-point matrix which includes $n_{x}$ sigma-point vectors, $\lambda$ is a design parameter that determines the spread of the generated sigma points and usually takes the form $\lambda=3-n_{x}$ .

In the sigma-point prediction step, each generated sigma point is inserted in the UKF nonlinear process model given in Eq. (8) to produce the matrix of the predicted (estimated) sigma points, which has an $n\times n_{x}$ dimension.

$\displaystyle{\hat{X}}_{k+1}=f(X_{k},{\nu}_{k})$ (8)

In the next step, the predicted state-mean and covariance matrices are calculated from the predicted sigma points as given in Eq. (9):

$\displaystyle{\hat{x}}_{k+1}=\sum^{n_{x}}_{i=0}{w_{i}{\hat{X}}_{k+1,i}}$ (9) $\displaystyle{\hat{P}}_{k+1}=\sum^{2n_{x}}_{i=0}w_{i}({\hat{X}}_{k+1,i}-{\hat{% x}}_{k+1})({\hat{X}}_{k+1,i}-{\hat{x}}_{k+1})^{\rm T}$

where $w_{i}^{\prime}s$ are the sigma-point weights that are used here to invert the spreading of the sigma points. These weights are calculated as shown in Eq. (2):

$\displaystyle w_{i}=\frac{\lambda}{\lambda+n_{x}},∼{}i=0$ $\displaystyle w_{i}=\frac{1}{2(\lambda+n_{x})},∼{}i=1\ldots n_{x}$ (10)

In the measurement prediction step, each generated sigma point is inserted in the UKF nonlinear measurement model given in Eq. (11) to produce the matrix of the predicted measurement sigma points, which has an $n\times n_{x}$ dimension.

$\displaystyle{\hat{Z}}_{k+1}=h({\hat{X}}_{k+1})$ (11)

In the next step, the predicted measurement-mean-and-covariance matrices are calculated from the predicted sigma points as well as the measurement noise covariance matrix $R$ as given in Eq. (2):

$\displaystyle{\hat{z}}_{k+1}=\sum^{n_{x}}_{i=0}{w_{i}{\hat{Z}}_{k+1,i}}$ $\displaystyle S_{k+1}=\sum^{2n_{x}}_{i=0}{w_{i}({\hat{Z}}_{k+1,i}-{\hat{z}}_{k% +1})}({\hat{Z}}_{k+1,i}-{\hat{z}}_{k+1})^{\rm T}+R$ $\displaystyle R=E\{{\omega}_{k}\cdot{\omega}^{\rm T}_{k}\}$ (12)

where $w_{i}^{\prime}s$ are the sigma-point weights that are determined using Eq. (2).

The final step is the UKF state update, where the UKF gain matrix ( $K$ ) is calculated as in Eq. (2) using the calculated cross-correlation matrix ( $T$ ) between the sigma points in the state space and the measurement space. The gain is used to update the UKF state vector ( $x$ ) as well as the state covariance matrix ( $P$ ).

$\displaystyle T_{k+1}=\sum^{2n_{x}}_{i=0}w_{i}({\hat{X}}_{k+1,i}-{\hat{x}}_{k+% 1})({\hat{Z}}_{k+1,i}-{\hat{z}}_{k+1})^{\rm T}$ $\displaystyle K_{k+1}=T_{k+1}S^{-1}_{k+1}$ $\displaystyle x_{k+1}={\hat{x}}_{k+1}+K_{k+1}({\hat{z}}_{k+1}-z_{k+1})$ $\displaystyle P_{k+1}={\hat{P}}_{k+1}-K_{k+1}S_{k+1}K^{\rm T}_{k+1}$ (13)

3. The moving object model

The state of the moving object [31] is determined by the five variables grouped into the state vector $x$ shown in Eq. (14), where $p_{x}$ , and $p_{y}$ are the object position in the $x$ and $y$ -axis respectively as shown in Fig. 2, $v$ is the magnitude of object velocity derived from its $x$ and $y$ components, $v_{x}$ and $v_{y}$ respectively. $\psi$ is the yaw angle (object orientation) and $\dot{\psi}$ is rate of change of the object-yaw angle.

$\displaystyle x=\left[\begin{array}[]{@{}c@{}}p_{x}\\ p_{y}\\ v\\ \psi\\ \dot{\psi}\end{array}\right],v=\sqrt{v^{2}_{x}+v^{2}_{y}},\psi=\tan^{-1}\frac{% v_{y}}{v_{x}}$ (14)

The nonlinear $x_{k+1}=f(x_{k},{\nu}_{k})$ difference equation that describes the motion model of the object is derived based on the state vector $x$ and presented in Eqs (15) and (16).

$\displaystyle x_{k+1}=x_{k}+$ (15) $\displaystyle\qquad∼{}\left[\begin{array}[]{@{}c@{}}{\displaystyle\frac{v_{k}}% {\dot{\psi}_{k}}}(\sin(\psi_{k}+\dot{\psi}_{k}\Delta t)-\sin(\psi_{k}))\\ {\displaystyle\frac{v_{k}}{\dot{\psi}_{k}}}(-\cos(\psi_{k}+\dot{\psi}_{k}% \Delta t)+\cos(\psi_{k}))\\ 0\\ \Delta t\\ 0\end{array}\right]+{\nu}_{k}$ $\displaystyle{\nu}_{k}=\left[\begin{array}[]{@{}c@{}}{\displaystyle\frac{1}{2}% }{(\Delta t)}^{2}\cos({\psi}_{k})\cdot{\nu}_{a,k}\\ {\displaystyle\frac{1}{2}}{(\Delta t)}^{2}\sin({\psi}_{k})\cdot{\nu}_{a,k}\\ \Delta t\cdot{\nu}_{a,k}\\ {\displaystyle\frac{1}{2}}{(\Delta t)}^{2}\cdot{\nu}_{\ddot{\psi,k}}\\ \Delta t\cdot{\nu}_{\ddot{\psi,k}}\end{array}\right]$ (16) $\displaystyle\Delta t=t_{k+1}-t_{k}$ $\displaystyle{\nu}_{a,k}\sim\mathcal{N}(0,{\sigma}^{2}_{a})$ $\displaystyle{\nu}_{\ddot{\psi},k}\sim\mathcal{N}(0,{\sigma}^{2}_{\ddot{\psi}})$ (17)

where $\Delta t$ is the time difference between two consecutive samples, $\ddot{\psi}$ is the yaw acceleration, $a$ is the longitudinal acceleration, ${\nu}_{a,k}$ is the longitudinal acceleration noise at sample $k$ with a standard deviation ${\sigma}^{2}_{a}$ , ${\nu}_{\ddot{\psi},k}$ is the yaw acceleration noise at sample $k$ with a standard deviation ${\sigma}^{2}_{\ddot{\psi}}$ .

Figure 2.

An object motion model.

If the $\dot{\psi}$ is zero, and to avoid dividing by zero in Eq. (15), the following approximation is used to calculate the prediction of $p_{x}$ , and $p_{y}$ :

$\displaystyle p_{x_{k+1}}=p_{x_{k}}+v_{k}\cos({\psi}_{k})\Delta t$ $\displaystyle p_{y_{k+1}}=p_{y_{k}}+v_{k}\sin({\psi}_{k})\Delta t$ (18)

Figure 3.

Lidar and radar data fusion using EKF.

4. Sensor fusion using EKF

Figure 3 presents the lidar and radar data fusion technique employing the EKF. The received sensor raw data (either lidar or radar) is getting processed before being supplied to the EKF.

The processing is performed using clustering and association algorithms. DBSCAN [32] is an unsupervised clustering algorithm that groups together data points if the density of the points is high enough. It requires two parameters to determine the density. The first parameter is $\varepsilon$ describing the radial distance from a point $p$ , that is being evaluated. The second parameter is minPts, which is the least number of detections that have to be within a distance $\varepsilon$ from $p$ , including $p$ itself, to form a cluster. By choosing $\varepsilon$ and minPts, it is then possible to decide the necessary density for a group of points to form a cluster, however, these fixed parameters not convenient if various types and topologies of road objects need to be detected. As an improvement to this algorithm, GB-DBSCAN is introduced [13]. It works in the same manner as DBSCAN but does not have fixed parameters. Instead, a polar grid is created according to the radial and angular resolution of the sensor. Instead of looking at a circular search area with a fixed radius, GB-DBSCAN is able to use a more dynamic, elliptic, search area.

While GB-DBSCAN is used for coarse clustering, the RANSAC [19] is used to fine-tune the clustering and associate geometrical shape proposals to potential coarse clusters. The output of the RANSAC is a fine-tuned object centroid ( $p_{x}$ and $p_{y}$ ) and its type (bicycle, car or pedestrian).

The processed lidar measurement vector includes the moving object centroid position ( $p_{x}$ and $p_{y}$ ) in cartesian coordinates, while the radar measurement vector includes the moving object centroid position ( $\rho$ , $\varphi$ ) and radian velocity ( $\dot{\rho}$ ) in polar coordinates as represented by Eq. (19). The mapping function that specifies how lidar Cartesian coordinates got mapped to the radar polar coordinates are given as well in Eq. (20).

$\displaystyle z_{\text{lidar}}=\left(\begin{array}[]{@{}c@{}}p_{x}\\ p_{y}\end{array}\right),∼{}∼{}z_{\text{radar}}=\left(\begin{array}[]{@{}c@{}}% \rho\\ \varphi\\ \dot{\rho}\end{array}\right)$ (19) $\displaystyle h(x)=\left(\begin{array}[]{@{}c@{}}\rho\\ \varphi\\ \dot{\rho}\end{array}\right)=\left(\begin{array}[]{@{}c@{}}\sqrt{p^{2}_{x}+p^{% 2}_{y}}\\ \arctan\left({\displaystyle\frac{p_{y}}{p_{x}}}\right)\\ {\displaystyle\frac{p_{x}v_{x}+p_{y}v_{y}}{\sqrt{p^{2}_{x}+p^{2}_{y}}}}\end{% array}\right)$ (20) $\displaystyle p_{x}=\rho\cos(\varphi),∼{}p_{y}=\rho\sin(\varphi)$ (21)

The EKF state nonlinear model is the moving object model that is described in detail in Section III and mathematically represented by Eqs (15)–(3). The Jacobian of this state model ( $F_{j}$ ) is presented in Eq. (22), while the associated state covariance matrix ( $Q$ ) is given by Eq. (23).

$\displaystyle F_{j}=$ (22) $\displaystyle\left[\begin{array}[]{@{∼}ccc@{}}∼{}1&0&{\displaystyle\frac{1}{% \dot{\psi}}}(-\sin(\psi)+\sin(\Delta t\dot{\psi}+\psi))\\ ∼{}0&1&{\displaystyle\frac{1}{\dot{\psi}}}(\cos(\psi)-\cos(\Delta t\dot{\psi}+% \psi))\\ ∼{}0&0&0\\ ∼{}0&0&1\\ ∼{}0&0&0\\ \end{array}\right.$ $\displaystyle\quad∼{}\left.\begin{array}[]{@{}ccccc@{}}{\displaystyle\frac{v}{% \dot{\psi}}}(-\cos(\psi)+\cos(\Delta t\dot{\psi}+\psi))&\\ {\displaystyle\frac{v}{\dot{\psi}}}(-\sin(\psi)+\sin(\Delta t\dot{\psi}+\psi))% &\\ 1&\\ 0&\\ 0&\\ \end{array}\right.$ $\displaystyle\left.\begin{array}[]{@{}c@{}}{\displaystyle\frac{v\Delta t}{\dot% {\psi}}}\cos(\Delta t\dot{\psi}\!+\!\psi)\!-\!{\displaystyle\frac{v}{{\dot{% \psi}}^{2}}}(-\sin(\psi)\!+\!\sin(\Delta t\dot{\psi}\!+\!\psi))\\ {\displaystyle\frac{v\Delta t}{\dot{\psi}}}\sin(\Delta t\dot{\psi}\!+\!\psi)\!% -\!{\displaystyle\frac{v}{{\dot{\psi}}^{2}}}(\cos(\psi)\!-\!\cos(\Delta t\dot{% \psi}\!+\!\psi))\\ \Delta t\\ 0\\ 1\\ \end{array}\right]$

$\displaystyle Q=$ (23) $\displaystyle\left[\begin{array}[]{@{}ccccc@{}}{\displaystyle\frac{\Delta t^{4% }}{4}}{\sigma}^{2}_{a_{x}}&0&{\displaystyle\frac{\Delta t^{3}}{2}}{\sigma}^{2}% _{a_{x}}&0&0\\ 0&{\displaystyle\frac{\Delta t^{4}}{4}}{\sigma}^{2}_{a_{y}}&{\displaystyle% \frac{\Delta t^{3}}{2}}{\sigma}^{2}_{a_{y}}&0&0\\ {\displaystyle\frac{\Delta t^{3}}{2}}{\sigma}^{2}_{a_{x}}&{\displaystyle\frac{% \Delta t^{3}}{2}}{\sigma}^{2}_{a_{y}}&\Delta t^{2}{\sigma}^{2}_{a}&0&0\\ 0&0&0&\Delta t^{2}{\sigma}^{2}_{\psi}&0\\ 0&0&0&0&\Delta t^{2}{\sigma}^{2}_{\dot{\psi}}\end{array}\right]$

Equation (4) presents the lidar measurement model ( $H_{\text{lidar}}$ ) as well as the measurement noise covariance matrix ( $R_{\text{lidar}}$ ) based on the state vector in Eq. (14). These matrices are required for the update step of the EKF that is represented by Eq. (2).

$\displaystyle H_{\text{lidar}}=\left[\begin{array}[]{@{}ccccc@{}}1&0&0&0&0\\ 0&1&0&0&0\end{array}\right]$ $\displaystyle R_{\text{lidar}}=E[\omega\cdot{\omega}^{\rm T}]=\left[\begin{% array}[]{@{}cc@{}}{\sigma}^{2}_{p_{x}}&0\\ 0&{\sigma}^{2}_{p_{y}}\end{array}\right]$ (24)

where ${\sigma}_{p_{x}}$ and ${\sigma}_{p_{y}}$ are the noise standard deviations for the object $x$ and $y$ positions respectively.

Figure 4.

Lidar and radar data fusion using UKF.

Additionally, Eq. (25) shows the derived Jacobian matrix ( $H_{j_{\text{radar}}}$ ) which represents the radar measurement model that will be used to estimate the measurement vector that is given in Eq. (19). The estimated measurement will be used in the update step of the EKF as illustrated in Eq. (2). The measurement noise covariance matrix ( $R_{\text{radar}}$ ) is derived and shown in Eq. (26). This matrix is required as well for the update step of the EKF.

$\displaystyle H_{j_{\text{radar}}}=$ (25) $\displaystyle\left[\begin{array}[]{@{}ccccc@{}}{\displaystyle\frac{p_{x}}{% \sqrt{p^{2}_{x}+p^{2}_{y}}}}&{\displaystyle\frac{p_{y}}{\sqrt{p^{2}_{x}+p^{2}_% {y}}}}&\\ -{\displaystyle\frac{p_{y}}{p^{2}_{x}+p^{2}_{y}}}&{\displaystyle\frac{p_{x}}{p% ^{2}_{x}+p^{2}_{y}}}&\\ \scalebox{0.85}{${\displaystyle\frac{p_{y}v(p_{y}\cos(\psi)\!-\!p_{x}\sin(\psi% ))}{{(p^{2}_{x}+p^{2}_{y})}^{3/2}}}$}&\scalebox{0.85}{${\displaystyle\frac{p_{% x}v(p_{x}\sin(\psi)\!-\!p_{y}\cos(\psi))}{{(p^{2}_{x}+p^{2}_{y})}^{3/2}}}$}&% \end{array}\right.$ $\displaystyle\left.\begin{array}[]{@{}ccccc@{}}0&0&0\\ 0&0&0\\ \scalebox{0.95}{${\displaystyle\frac{p_{x}\cos(\psi)\!+\!p_{y}\sin(\psi)}{% \sqrt{p^{2}_{x}\!+\!p^{2}_{y}}}}$}&\scalebox{0.95}{${\displaystyle\frac{v(p_{y% }\cos(\psi)\!-\!p_{x}\sin(\psi))}{\sqrt{p^{2}_{x}\!+\!p^{2}_{y}}}}$}&0\\ \end{array}\right]$

$\displaystyle R_{\text{radar}}=\left[\begin{array}[]{@{}ccc@{}}{\sigma}^{2}_{% \rho}&0&0\\ 0&{\sigma}^{2}_{\varphi}&0\\ 0&0&{\sigma}^{2}_{\dot{\rho}}\end{array}\right]$ (26)

where ${\sigma}_{\rho}$ is the noise standard deviation of the object radial distance, ${\sigma}_{\varphi}$ is the noise standard deviation of the object heading (bearing), ${\sigma}_{\dot{\rho}}$ is the noise standard deviation of the object yaw rate.

As per the above presentation, each sensor has its own prediction update scheme, however, both sensors share the same state prediction scheme. The belief about the object’s position and velocity is updated asynchronously each time the measurement is received regardless of the source sensor.

Both distinct update schemes are getting updated by any received measurement data (either from lidar or from radar). The state vector ( $x$ ) is getting updated after receiving a lidar measurement vector ( $z_{\text{lidar}}$ ) in Eq. (19) by inserting ( $F_{j},Q,H_{\text{lidar}}$ , and $R_{\text{lidar}})$ in Eq. (23). Likewise, the state vector ( $x$ ) is getting updated after receiving a radar measurement vector ( $z_{\text{radar}}$ ) in Eq. (19) by inserting ( $F_{j},Q,H_{j_{\text{radar}}}$ , and $R_{\text{radar}})$ in Eq. (23). Accordingly, the state vector ( $x$ ) is the product of the fusion between lidar and radar measurement data.

5. Sensor fusion using UKF

Figure 4 presents the lidar and radar data fusion technique employing the UKF. After computing the elapsed time between consecutive sensor reading ( $\Delta t$ ), the sigma points ( $X_{k}$ ) are generated using Eq. (7), and then a next-time-step prediction for sigma points ( ${\hat{X}}_{k+1}$ ) is carried out using Eq. (8) while employing the moving object nonlinear motion model given in Eq. (15). The resulted predicted sigma points are then used to compute the state mean ( ${\hat{x}}_{k+1}$ ) and covariance ( ${\hat{P}}_{k+1}$ ) matrices using Eq. (9).

Then the fusion technique thus branches into two directions based on the source of the last sensor data measurement. If the source is a radar, and employing the nonlinear radar measurement model (Eq. (20)), the predicted measurement sigma points ( ${\hat{Z}}_{k+1}$ ) are calculated from the predicted state sigma points ( ${\hat{X}}_{k+1}$ ) using Eq. (11). Then, the predicted measurements ( ${\hat{z}}_{k+1}$ ) and their covariance matrix ( $S_{k+1}$ ) are calculated based on Eq. (2) using the measurement noise covariance matrix $R_{\text{radar}}$ given in Eq. (26). Then, ${\hat{x}}_{k+1}$ and ${\hat{z}}_{k+1}$ and are used to compute the cross-correlation matrix ( $T_{k+1}$ ) between the sigma points in the state space ( ${\hat{X}}_{k+1}$ ) and the measurement space ( ${\hat{Z}}_{k+1}$ ) as in Eq. (2). Based on this cross-correlation matrix, the Kalman filter gain ( $K_{k+1}$ ) is then calculated and used to compute the updated object’s state vector ( $x_{k+1}$ ) and covariance matrix ( $P_{k+1}$ ) as shown by Eq. (2).

If the measurement data source is a lidar, and employing the linear lidar measurement model ( $H_{\text{lidar}}$ ) shown in Eq. (23), the predicted measurement sigma points ( ${\hat{Z}}_{k+1}$ ) is directly calculated from ( ${\hat{X}}_{k+1}$ ). Then, the predicted measurements ( ${\hat{z}}_{k+1}$ ) and their covariance matrix ( $S_{k+1}$ ) are calculated based on Eq. (2) using the measurement noise covariance matrix $R_{\text{lidar}}$ given in Eq. (4). Then, ${\hat{x}}_{k+1}$ and ${\hat{z}}_{k+1}$ and are used to compute the cross-correlation matrix ( $T_{k+1}$ ) between the sigma points in the state space ( ${\hat{X}}_{k+1}$ ) and the measurement space ( ${\hat{Z}}_{k+1}$ ) as in Eq. (2). Based on this cross-correlation matrix, the Kalman filter gain ( $K_{k+1}$ ) is then calculated and used to compute the updated object’s state vector ( $x_{k+1}$ ) and covariance matrix ( $P_{k+1}$ ) as shown by Eq. (2).

6. Implementation of Kalman filters

Both Kalman filters (EKF and UKF) are implemented using the high-performance language GCC C $++$ [33] on Ubuntu Linux operating system [34]. This combination is fitting for the required real-time performance [30]. A C $++$ numerical solver, matrix and vector operations package “Eigen” [36] is used to numerically calculate the Jacobeans, object model and effectively performing the predict and update steps.

In the following sections, several important matters that have a crucial effect on the Kalman filters design process will be highlighted and discussed.

6.1 Noise parameters setting

The object motion model described by Eqs (14)–(3) includes several noise parameters that need to be carefully set. To set the two process noise parameters as an example: the longitudinal acceleration noise standard deviation ${\sigma}_{a}$ and the yaw acceleration noise ${\sigma}_{\ddot{\psi}}$ , one has to approximate the expected top acceleration road objects can exhibit both longitudinal and angular as an initial guess, then fine-tune these values through trial-and-error iterations. After, exhaustive tuning process, the most appropriate valuation for ${\sigma}_{a}$ and ${\sigma}_{\ddot{\psi}}$ are found to be 3 m/sec ${}^{2}$ and 0.6 rad/sec ${}^{2}$ . Similarly, the other parameters can be set and accordingly, Table 1 presents the fine-tuned parameters for both EKF and UKF.

${\sigma}_{a}$ is set to 3 m/sec ${}^{2}$ because $\{-6,6\}$ m/sec ${}^{2}$ is the expected boundary of the longitudinal acceleration (e.g. road vehicles) as statistically 95% of the time the acceleration will stay within the $\{-2{\sigma}_{a},2{\sigma}_{a}\}$ range. The same principle applies to ${\sigma}_{\ddot{\psi}}$ where the appraisal of how fast a vehicle can accelerate or decelerate while completing a circular path is guiding the selection of the set value of 0.6 rad/sec ${}^{2}$ .

Table 1
The Kalman filters noise parameters

Parameter	EKF	UKF
${\sigma}_{a}$ m/sec ${}^{2}$	3.0	1.0
${\sigma}_{a_{x}}$ m/sec ${}^{2}$	3.0	–
${\sigma}_{a_{y}}$ m/sec ${}^{2}$	3.0	–
${\sigma}_{\ddot{\psi}}$ rad/sec ${}^{2}$	0.6	0.6
${\sigma}_{\dot{\psi}}$ rad/sec	0.06	0.06
${\sigma}_{p_{x}}$ (lidar) m	0.15	0.15
${\sigma}_{p_{y}}$ (lidar) m	0.15	0.15
${\sigma}_{\rho}$ (radar) m	0.3	0.3
${\sigma}_{\varphi}$ (radar) rad	0.03	0.03
${\sigma}_{\dot{\rho}}$ (radar) m/sec	0.3	0.3

6.2 Consistency measures for Kalman filters

The KF design is considered consistent if the estimation error ( ${\hat{y}}_{k}$ in Eq. (2)) is unbiased, i.e. has zero-mean, and that the actual mean square error of the filter matches the filter-calculated state covariance. As a measure of filter consistency, the time-average Normalized Innovation Squared (NIS) [37] can be used to finetune the noise parameters. The NIS follows a Chi-squared distribution ( $X^{2}$ ) and based on the number of the measurement-vector dimension; is used to assess whether KF’s noise assumptions are consistent with the realized measurements. The metric, described by Eq. (27), is used to calculate the NIS value at each sample $k$ and then averaging these values ( $\textit{NIS}_{\text{Average}}$ ) over a moving window of measurements of length $N$ .

$\displaystyle\textit{NIS}_{k}={(z_{k+1}-{\hat{z}}_{k})}^{\rm T}S^{-1}_{k}(z_{k% +1}-{\hat{z}}_{k})$ (27) $\displaystyle\textit{NIS}_{\text{Average}}=\frac{1}{N}\sum^{k=N}_{k=1}{\textit% {NIS}_{k}}$ (28)

The radar’s measurement is a three-dimensional vector (three degrees of freedom), therefore, for a consistent EKF or UKF design the values of $\textit{NIS}_{k}$ should be less than “7.815” in 95% of the time-steps. Likewise, the lidar’s measurement is a two-dimensional vector (two degrees of freedom), therefore, for a consistent EKF or UKF design the values of $\textit{NIS}_{k}$ should be less than “5.991” in 95% of the time steps.

In both sensors, the $\textit{NIS}_{\text{Average}}$ ’s value should not be too low as it means the process uncertainty is highly overestimated, or too high as it means the process uncertainty is highly underestimated. It actually should be around the middle or the 2/3 ${}^{\text{rd}}$ of their corresponding threshold ranges.

6.3 Initialization of Kalman filters

The proper initialization of the Kalman filter is very crucial to its subsequent performance [38]. The main initialized variables are the estimate state vector ( $x$ ) and its estimate covariance matrix ( $P$ ).

The first two terms of the state vector $x$ given by Eq. (14) are $p_{x}$ and $p_{y}$ which are simply initialized using the first received raw sensor measurement. For the other three terms of the state vector, intuition augmented with some trial-and-error is used to initialize these variables as listed in Table 2.

The state covariance matrix is initialized as a diagonal matrix that contains the covariance of each variable estimate (Eq. (29)). The initialization logic works as follows: little or almost no correlation among the state variables (independent variables) is assumed, therefore, the off-diagonal terms (covariances between variables) are initialized to zeros. Each diagonal term represents the variance (confidence) of each state element estimate as shown in Eq. (29). The variance of each element is initialized depends on the a priori information about this element. Since the first two elements of the state vector ( $p_{x}$ and $p_{y}$ ) are initialized using the first raw reading of the sensors, then both ${\sigma}^{2}_{{\hat{p}}_{x}}$ and ${\sigma}^{2}_{{\hat{p}}_{y}}$ are set to small values. However, little a priori information is known about the other three terms ( $v,\psi,\dot{\psi}$ ), therefore, they have been initialized to large values as listed in Table 2. Note that radar velocity measurement ( $\dot{\rho}$ ) cannot directly be used to initialize the state vector velocity (object velocity $v$ ) as they are not the same.

$\displaystyle P=\left[\begin{array}[]{@{}ccccc@{}}{\sigma}^{2}_{{\hat{p}}_{x}}% &0&0&0&0\\ 0&{\sigma}^{2}_{{\hat{p}}_{y}}&0&0&0\\ 0&0&{\sigma}^{2}_{\hat{v}}&0&0\\ 0&0&0&{\sigma}^{2}_{\hat{\psi}}&0\\ 0&0&0&0&{\sigma}^{2}_{\hat{\dot{\psi}}}\end{array}\right]$ (29)

Table 2

Initialization of Kalman filters states

Parameter	EKF	UKF
$p_{x}$ m	1 ${}^{\text{st}}$ raw $x$ -reading	1 ${}^{\text{st}}$ raw $x$ -reading
$p_{y}$ m	1 ${}^{\text{st}}$ raw $y$ -reading	1 ${}^{\text{st}}$ raw $y$ -reading
$v$ m/sec	0.0	0.0
$\psi$ rad	0.0	0.0
$\dot{\psi}$ rad/sec	0.0	0.0
${\sigma}_{{\hat{p}}_{x}}$ m	1.0	1.0
${\sigma}_{{\hat{p}}_{y}}$ m	1.0	1.0
${\sigma}_{\hat{v}}$ m/sec	$\sqrt{1000}$	$\sqrt{1000}$
${\sigma}_{\hat{\psi}}$ rad	$\sqrt{1000}$	$\sqrt{1000}$
${\sigma}_{\hat{\dot{\psi}}}$ m/sec ${}^{2}$	$\sqrt{1000}$	$\sqrt{1000}$

6.4 Performance measures for Kalman filters

To check the performance of the KF, in terms of how far the estimated results from the true results (ground truth). There are many evaluation metrics [39], but perhaps the most common one is the Root Mean Squared Error (RMSE) given in Eq. (30). The metric is calculated over a moving window of measurements of length $N$ [40]. $x^{\text{est}}_{k}$ is the estimated state vector of the KF given in Eq. (2), and $x^{\text{true}}_{k}$ is the true state vector supplied by the simulator or given as training data during the KF design phase.

$\displaystyle\textit{RMSE}=\sqrt{\frac{1}{N}\sum^{k=N}_{k=1}{{(x^{\text{est}}_% {k}-x^{\text{true}}_{k})}^{2}}}$ (30)

Table 3

Performance evaluation of Kalman filters

Test case	State variable	RMSE EKF	RMSE UKF	Percentage improvement
Bicycle	$p_{x}$	0.0959	0.0648	$-$ 32.43%
	$p_{y}$	0.0931	0.0809	$-$ 13.10%
	$v_{x}$	0.2953	0.1452	$-$ 50.83%
	$v_{y}$	0.3750	0.1592	$-$ 57.55%
	$\psi$	0.0728	0.0392	$-$ 46.15%
Car	$p_{x}$	0.1946	0.1857	$-$ 4.57%
	$p_{y}$	0.1903	0.1899	$-$ 0.21%
	$v_{x}$	0.5190	0.4745	$-$ 8.57%
	$v_{y}$	0.8111	0.5075	$-$ 37.43%
	$\psi$	0.4037	0.2580	$-$ 36.09%
Pedestrian	$p_{x}$	0.0758	0.0652	$-$ 13.98%
	$p_{y}$	0.0842	0.0605	$-$ 28.15%
	$v_{x}$	0.6323	0.5332	$-$ 15.67%
	$v_{y}$	0.5807	0.5442	$-$ 6.29%
	$\psi$	0.2301	0.2075	$-$ 9.82%

Figure 5.

Test track for a moving bicycle.

7. Testing and evaluation results

Extensive trials-and-errors attempts are used to tune the many hyper-parameters of the LR_ODT. However, to be more consistent and accurate, numerical Key Performance Indicators (KPIs) are constructed and coded as in Eqs (27) and (30) to evaluate the performance of the fusion technique under the given set of hyper-parameters.

Several test tracks have been used to evaluate the performance of the LR_ODT under different sets of hyper-parameters in an iterative tuning process. Examples of these test tracks are shown in Figs 5–7. These tracks are representing three different moving objects, bicycle, car, and pedestrian respectively, to emulate various motion profiles and velocities.

Table 4
Consistency evaluation of Kalman filters

	NIS average	NIS min	NIS max	NIS above 95% threshold
Radar	2.8054	0.05806	11.124	3.6%
Lidar	2.7903	0.00116	14.749	1.6%
Lidar $+$ Radar	2.2797	0.00116	14.749	2.2%

Figure 6.

Test track for a moving car.

Figure 7.

Test track for a walking pedestrian.

Table 3 presents the testing results of the LR_ODT fusion algorithm that uses the UKF and compares it with that uses the EKF. The performance evaluation is carried out on the three test tracks. The RMSE KPI (Eq. (30)) is used to compare both UKF and EKF performances on the five state variables: $p_{x}$ , $p_{y}$ , $v_{x}$ , $v_{y}$ , and $\psi$ . The KPI is comparing each estimated state variable to its ground-truth value and finding the error. The lower the value of the KPI the better the performance. Moreover, The bicycle tracking results using the UKF are depicted in the form of $x$ - $y$ position (shown in Fig. 8), longitudinal velocity (shown in Fig. 9), yaw angle (shown in Fig. 11), and yaw rate (shown in Fig. 12). The bicycle tracking results using the EKF are reported in the third column of Table 3 and the estimated longitudinal velocity profile is depicted in Fig. 10.

Table 5

Sensor fusion evaluation of the UKF (Bicycle Track)

	Lidar $+$ Radar	Lidar only	Radar only
RMSE – $p_{x}$	0.0648	0.1612	0.2031
RMSE – $p_{y}$	0.0809	0.1464	0.2539
RMSE – $v_{x}$	0.1452	0.2082	0.1971
RMSE – $v_{y}$	0.1592	0.2129	0.1871
RMSE – $\psi$	0.0392	0.0540	0.0480
NIS – average	2.2797	1.6941	2.6576
NIS – min	0.0012	0.04874	0.11309
NIS – max	14.749	12.997	12.183
NIS $>$ 95% threshold	2.2%	3.2%	5.2%

Figure 8.

Performance of the UKF position estimation on the bicycle track.

Figure 9.

UKF-velocity-estimation performance on the bicycle track.

The UKF design-consistency indicator (NIS) results are reported in Table 3. The table reports values for the NIS by taking into consideration the estimated measurements by lidar alone, the estimated measurements by radar alone, and after combining both lidar and radar measurements. The values reported in the 5 ${}^{\text{th}}$ column of the table shows a very consistent filter design. All the values are significantly lower than 5%.

Table 6

Sensor fusion execution time of EKF and UKF

Phase	EKF – $\mu$ sec	UKF – $\mu$ sec
Predict	4.827	27.32
Update	16.40	14.20
Total	21.23	41.52

Table 7

LR_ODT execution time for single object

	EKF	UKF
State estimation execution time	0.637 msec	1.246 msec
Clustering and object association	0.427 msec	0.835 msec
Control code overhead – 20%	0.213 msec	0.416 msec
Total – 30 fps	1.276 msec	2.496 msec

Figure 10.

EKF-velocity-estimation performance on the bicycle track.

To assess the significance of the fusion between lidar and radar in tracking. The UKF is tested in one time with measurements from lidar alone, and another time with measurements from radar alone. The results reported in Table 5 show how fusion makes the difference and substantially improves accuracy. The estimation of all state variables is spectacularly improved. For example, the RMSE of $x$ -position ( $p_{x}$ ) estimation is reduced by 60% compared to “lidar-alone” and 60% compared to “radar-alone” estimations. Moreover, the RMSE of $x$ -velocity ( $v_{x}$ ) estimation is reduced by 30% compared to “lidar-alone” and 26% compared to “radar-alone” estimations. The NIS values are calculated as well for “lidar-alone” and “radar-alone” cases to test the consistency of the UKF in their cases. The reported values show that fusion significantly improves the consistency. The NIS values that exceed the 95%-threshold have been reduced by 31% compared to the “lidar-alone” and 38.5% compared to the “radar-alone” ones.

Comparing UKF performance with that of the EKF, Table 3 details the results. It is obvious that UKF outperforms EKF at all velocity and motion profiles. The accuracy of all the estimated states is sustainably higher using the UKF. States that got affected more with non-linearities in the object model (e.g. $v_{x}$ , $v_{y}$ , and $\psi$ ) have seen more accuracy improvement. Comparing the velocity profiles in both Figs 9 and 10 gives some insight into the achieved improvement.

Figure 11.

UKF-yaw-angle estimation performance on the bicycle track.

Figure 12.

UKF-yaw-rate estimation performance on the bicycle track.

Figure 13.

Lidar NIS values for UKF on the bicycle track.

Figure 14.

Radar NIS values for UKF on the bicycle track.

The LR_ODT proved to be well enough fast in execution to be used in real-time. Using an Intel Core i5 with 1.6 GHz and 8 GB RAM which is a very moderate computational platform, the following measurements (Table 6) are collected for both EKF and UKF.

Table 6 shows how the EKF is twice faster than UKF. By considering that the lidar/radar measurements are collected at approximately 30 fps rate. Then the measurement cycle is 33.3 msec which is large enough to be utilized for tracking 25 objects using EKF or 13 objects using UKF according to the data in Table 7.

8. Conclusion

In this paper, a real-time road-object detection and tracking method (LR_ODT) for autonomous cars is proposed, implemented and described in detail. The method uses a tailored unscented Kalman filter to perform data fusion for the mounted lidar and radar devices on the ego car. The raw data of the lidar/radar are getting clustered using both GB-DBSCAN and RANSAC algorithms to produce the raw object’s pose and to determine its geometrical shape. The LR_ODT method is fully implemented using GCC C $++$ in addition to advanced math libraries to optimize its real-time performance. The design steps, initialization and tuning of both the EKF and the UKF are described in detail. The consistency evaluation of both filters has been explained as well.

The validation results show that the proposed method is reliably able to detect and track three types of street objects: bicycle, car, and pedestrians on three different tracks and speed profiles. The employed generic object motion model is comprehensive and is described using five state variables. The UKF has outperformed the EKF on all test cases and all the state variable levels ( $-$ 24% average RMSE), despite its considerably more complex design and higher execution time.

Comparing the validation results of the UKF applied to a single sensor to the one employing the fusion of multiple sensors, show how outstanding is the improvement in tracking performance using the later ( $-$ 29% RMES with lidar and $-$ 38% RMSE with radar).

The measured throughput (execution time) using an affordable CPU proved that the LR_ODT method is very suitable for real-time multi-object detection and tracking.

In the future, it is intended to add a front-camera to the presented fusion technique and further investigate the benefits it will add to the overall tracking performance. Furthermore, the LR_ODT will be augmented with other road objects like guardrails, trucks, animals, etc.

References

Farag

. Traffic signs classification by deep learning for advanced driving assistance systems. Intelligent Decision Technologies, IOS Press. 2019; 13(3): 215-231.

Farag

Saleh

. Road lane-lines detection in real-time for advanced driving assistance systems. Intern Conf on Innovation and Intelligence for Informatics, Computing, and Technologies (3ICT’18), Bahrain. 2018; 18-20.

Farag

. A comprehensive real-time road-lanes tracking technique for autonomous driving. International Journal of Computing and Digital Systems (IJCDS). 2020; 9(3): 349-362.

Farag

Saleh

. Behavior cloning for autonomous driving using convolutional neural networks. Intern Conf on Innovation and Intelligence for Informatics, Computing, and Technologies (3ICT’18), Bahrain. 2018; 18-20.

Farag

. Recognition of traffic signs by convolutional neural nets for self-driving vehicles. International Journal of Knowledge-based and Intelligent Engineering Systems, IOS Press. 2018; 22(3): 205-214.

Farag

Saleh

. Tuning of PID track followers for autonomous driving. Intern Conf on Innovation and Intelligence for Informatics, Computing, and Technologies (3ICT’18), Bahrain. 2018; 18-20.

Farag

. Complex trajectory tracking using pid control for autonomous driving. International Journal of Intelligent Transportation Systems Research, Springer. 2020; 18(5): 356-366.

Farag

. Complex-track following in real-time using model-based predictive control. International Journal of Intelligent Transportation Systems Research, Springer. 2021; 19(4): 112-127.

Farag

Saleh

. An advanced road-lanes finding scheme for self-driving cars. 2nd Smart Cities Symposium (SCS’19), IET Digital Library, Bahrain. 2019; 24-26.

10.

Farag

. Safe-driving cloning by deep learning for autonomous cars. International Journal of Advanced Mechatronic Systems, Inderscience Publishers. 2019; 7(6): 390-397.

11.

Farag

. Cloning safe driving behavior for self-driving cars using convolutional neural networks. Recent Patents on Computer Science, Bentham Science Publishers, The Netherlands. 2019; 12(2): 120-127.

12.

Farag

Saleh

. An advanced vehicle detection and tracking scheme for self-driving cars. 2nd Smart Cities Symposium (SCS’19), IET Digital Library, Bahrain. 2019; 24-26.

13.

Yurtsever

Lambert

Carballo

Takeda

. A survey of autonomous driving: Common practices and emerging technologies. arXiv: 1906.05113v2; 2020.

14.

Che

Jung

Olsen

. Object recognition, segmentation, and classification of mobile laser scanning point Clouds: A state of the art review. 2019; 19: 810; doi: 10.3390/s19040810.

15.

Xie

Tian

Zhu

. A review of point cloud semantic segmentation. arXiv: 1908.08854v2; 2019.

16.

Lidman

Luu

. Clustering, shape extraction and velocity estimation applied to radar detections. M.Sc. thesis, Dept of Elect Eng, Chalmers University of Technology, Gothenburg, Sweden; 2018.

17.

Farag

. Real-time detection of road lane-lines for autonomous driving. Recent Advances in Computer Science and Communications, Betham Science. 2020; 13(2): 265-274.

18.

Dietmayer

Kellner

Klappstein

. Grid-based dbscan for clustering extended objects in radar data. IEEE Intelligent Vehicles Symposium, Alcala de Henares, Spain; 2012.

19.

Fischler

Bolles

. Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography. Comm ACM. 1981; 24(6): 381-395.

20.

Gohring

Wang

Schnurmacher

Ganjineh

. Radar/ lidar sensor fusion for car-following on highways. IEEE Int Conf Autom Robot Appl. 2011; 407-412.

21.

Kaempchen

Fuerstenberg

Skibicki

Dietmayer

KCJ

. Sensor fusion for multiple automotive active safety and comfort applications. in: Advanced Microsystems for Automotive Applications. Valldorf

Gessner

, eds, 2004; 2: 137-163.

22.

Zarchan

Musoff

. Fundamentals of kalman filtering: A practical approach. American Institute of Aeronautics and Astronautics, Incorporated, 4

{}^{\text{th}}

Ed., 2013. ISBN 978-1-62410-276-9.

23.

Chavez-Garcia

Aycard

. Multiple sensor fusion and classification for moving object detection and tracking. IEEE Transactions on Intelligent Transportation Systems. 2015; 17(2): 252-534.

24.

Rangesh

Trivedi

. No blind spots: Full-surround multi-object tracking for autonomous vehicles using cameras and LiDARs. IEEE Transactions on Intelligent Vehicles. 2019; 4(4).

25.

Hajri

Rahal

. Real-time lidar and radar high-level fusion for obstacle detection and tracking with evaluation on a ground truth. arXiv: 1807.11264v2; 2019.

26.

Jahromi

Tulabandhula

Cetin

. Real-time hybrid multi-sensor fusion framework for perception in autonomous vehicles. Sensors. 2019; 19(20). 4357, doi: 10.3390/s19204357.

27.

Wan

Van Der Merwe

. The unscented Kalman filter for nonlinear estimation. IEEE Adaptive Sys for Signal Processing, Comm, and Control Symposium, Alberta, Canada; 2000.

28.

Julier

Uhlmann

. Unscented filtering and nonlinear estimation. Proceedings of the IEEE. 2004; 92(3): 401-422.

29.

Einicke

White

. Robust extended kalman filtering. IEEE Trans Signal Process. 1999; 47(9): 2596-2599.

30.

Best

Bogdanski

. Extending the Kalman filter for structured identification of linear and nonlinear systems. Int J Modelling, Identification, and Control. 2017; 27(2).

31.

Schubert

Richter

Wanielik

. Comparison and evaluation of advanced motion models for vehicle tracking. 11th Inter Conf on Information Fusion, Cologne, Germany; 2008.

32.

Sander

Ester

Kriegel

. A density-based algorithm for discovering clusters in large spatial databases with noise. Proc of the 2nd Inter Conf on Knowledge Discovery and Data Mining. 1996; 226-231.

33.

GCC C

++

[homepage on the Internet]. https://gcc.gnu.org/, accessed on 11

{}^{\text{th}}

; 2020.

34.

Ubuntu Linux [homepage on the Internet]. https://www.ubuntu.com/, accessed on 11

{}^{\text{th}}

; 2020.

35.

Nagiub

Farag

. Automatic selection of compiler options using genetic techniques for embedded software design. IEEE 14th Inter. Symposium on Comp. Intelligence and Informatics (CINTI), Budapest, Hungary; 2013.

36.

Eigen [homepage on the Internet]. http://eigen.tuxfamily.org/index.php?title=Main_Page, accessed on 11

{}^{\text{th}}

; 2020.

37.

Piché

. Online tests of Kalman filter consistency. International Journal of Adaptive Control and Signal Processing. 2016; 30(1): 115-124.

38.

Zhao

Huang

. On Initialization of the kalman filter. 6th Inter Symposium on Adv Control of Ind Processes (AdCONIP), Taipei, Taiwan. 2017; 28-31.

39.

Saho

. Kalman filter for moving object tracking: Performance analysis and filter design. Intech Open, Rijeka; 2018. doi: 10.5772/intechopen.71731.

40.

Farag

. Synthesis of intelligent hybrid systems for modeling and control. Ph.D. Thesis. University of Waterloo. Canada. 1998.

Lidar and radar fusion for real-time road-objects detection and tracking

Abstract

Keywords

1. Introduction

6. Implementation of Kalman filters

6.1 Noise parameters setting

Table 1 The Kalman filters noise parameters

Table 4 Consistency evaluation of Kalman filters

References

Table 1
The Kalman filters noise parameters

Table 4
Consistency evaluation of Kalman filters