New dynamic error spectrum for performance evaluation

Abstract

In performance evaluation, the widely used root-mean-square error is easily affected by large error terms and is also an incomprehensive measure. Therefore, the error spectrum as a comprehensive measure was proposed for parameter estimation. However, error spectrum (ES) is a three-dimension plot (among ES, r axis and time t axis) in the whole time horizon in dynamic evaluation system, which is not intuitive and easy to be analyzed. To smooth this, a new dynamic error spectrum (NDES) is proposed in dynamic evaluation system in this paper. Firstly, the NDES is defined for EPE in dynamic systems. Secondly, the computation method is proposed to calculate the NDES. Thirdly, several nice properties of NDES are presented for dynamic system performance evaluation. Finally, the effectiveness of the proposed new dynamic error spectrum is verified by a numerical example.

Keywords

Performance evaluation decision support systems parameter estimation new dynamic error spectrum

1 Introduction

Estimation performance evaluation (EPE) has been widely applied in estimation or filtering [1, 2], track fusion [3], and target tracking [4], etc. Usually, EPE is divided into two components: performance evaluation and peformance ranking, where performance evaluation should rely mainly on numerical information, while performance ranking depends on preference information [5]. An extensive literatures are by now available on both performance ranking [6 –9], and evaluation [10 –12]. Since the existing measures can only reflects one aspect of EPE, the error spectrum was presented in [13]. In EPE, ES can reveal more information because it aggregates several commonly incomprehensive measures. However, ES has some limitations and drawbacks, which attracts lots of research to improve the ES [14 –16]. In particular, it is difficult to evaluate the dynamic system performance based on ES, because ES is a three dimensional (3D) plot over the total time span. Accordingly, a dynamic error spectrum (DES) based on the average value of ES was introduced in [16]. Furthermore, we find that DES combines several incomprehensive metrics into a single metric, which reflects the estimation accuracy of an estimator the same as the incomprehensive metric. Therefore, we proposed range error spectrum induced area (RESA) and DES induced area (DESA) to overcome the problem of DES [17, 18]. However, the above proposed dynamic measures (i.e., DES, RESA and DESA) need to combine all the three-dimensional plot (ES curve in dynamic systems) into a single two-dimensional plot. Therefore, it is still not easy to find the better estimator when two ES curves intersect with each other among whole time horizon. Clearly, a worthwhile problem is how to apply ES to dynamic evaluation systems.

So, a new dynamic error spectrum (NDES) is proposed to the dynamic system evaluation, which is different from the DES introduced in [16]. First, the NDES is defined according to the definition of ES. Second, the computation method is proposed to calculate the NDES. Finally, several nice properties of NDES are presented to evaluate dynamic systems.

This paper is organized as follows. In Section 1, the definition of the NDES is proposed according to the definition of ES. Then, the computational method is proposed to calculate the NDES in Section 1. Furthermore, some nice properties of the NDES are presented in Section 1. Finally, a numerical example is provided in Section 1 to illustrate the superiority of the proposed metrics. Section 1 concludes this paper.

2 Definition of NDES

As discussed in the introduction Section 1, Li proposed the ES to the EPE, which is a three-dimension (3D) plot when applied to dynamic evaluation systems [13, 17]. Although ES can be drawn as 3D plot without difficulties by many software, it is still difficult for it to be analyzed in dynamic evaluation systems. To alleviate this, a new dynamic error spectrum for dynamic evaluation systems is proposed in the following section. First, a new dynamic error spectrum (NDES) is defined for EPE in dynamic systems, which is still an aggregation of many incomprehensive measures. Second, the properties of the NDES are presented according to the original ES. Finally, the volume error spectrum is proposed to calculate the NDES.

Denote $\hat{a}$ as a (point) estimator, then the estimation error $\tilde{a}$ holds for $\tilde{a} = a - \hat{a}$ , where a is the quantity to be estimated. Furthermore, denote $\tilde{a} (t)$ as the estimation error at any time instant t, and let $e (t) = ∥ \tilde{a} (t) ∥$ , then, if ∥· ∥ is the 1-norm ∥ · ∥ ₁ and $\tilde{a} (t)$ is a 1-dimension vector, i.e, real number, we have $e (t) = | | \tilde{a} (t) | |_{1} = | \tilde{a} (t) |$ (1) and for a multi-dimension vector, i.e., $\tilde{a} (t) = [{\tilde{a}}_{1} (t), {\tilde{a}}_{2} (t), \dots, {\tilde{a}}_{D} (t)]$ we have $e (t) = ∥ \tilde{a} (t) ∥_{1} = \sum_{d = 1}^{D} lvert {\tilde{a}}_{d} (t) rvert$ (2)

Furthermore, at a time instant t, for an estimation error set ${\tilde{a} (t)_{k}}_{k = 1}^{M}$ (where M is the number of Monte Carlo runs), we also have $\begin{matrix} e (t)_{k} = | | \tilde{a {(t)}_{k}} | |_{1} \\ = {\begin{matrix} | \tilde{a} {(t)}_{k} | \\ \sum_{d = 1}^{D} | {\tilde{a}}_{d} {(t)}_{k} | \end{matrix} \begin{matrix} \tilde{a} {(t)}_{k} is a real number \\ \tilde{a} {(t)}_{k} is a vector \end{matrix} \end{matrix}$ (3) Likewise, if ∥· ∥ is the 2-norm ∥ · ∥ ₂, then Equation (3) is changed to $\begin{matrix} e (t)_{k} = | | \tilde{a {(t)}_{k}} | |_{2} \\ = {\begin{matrix} \sqrt{| \tilde{a} {(t)}_{k} |^{2}} \\ (\sum_{d = 1}^{D} | {\tilde{a}}_{d} {(t)}_{k} |^{2})^{\frac{1}{2}} \end{matrix} \begin{matrix} \tilde{a} {(t)}_{k} is a real number \\ \tilde{a} {(t)}_{k} is a vector \end{matrix} \end{matrix}$ (4)

Thus, for r ∈ [- ∞ , + ∞] and t ∈ [T₀, T], the NDES is defined as $\begin{matrix} S_{e (t)} (r, t) = \int_{T_{0}}^{T} {E [e {(t)}^{r}]}^{1 / r} dt \\ = \int_{T_{0}}^{T} {[\int e {(t)}^{r} dF (e (t))]}^{1 / r} dt \end{matrix}$ (5) where F( e (t)) is the cumulative distribution function (CDF) of the norm of the estimation error at time instant t.

For a continuous e (t), the NDES is rewritten as $S_{e (t)} (r, t) = \int_{T_{0}}^{T} {\int e {(t)}^{r} [f (e (t))] d e (t)}^{1 / r} dt$ (6) where f( e (t)) is the probability density function (PDF) of the norm of the estimation error at time instant t.

Similarly, for a discrete e (t), the NDES is represented by $S_{e (t)} (r, t) = \int_{T_{0}}^{T} [\sum [p_{k} (e (t))] \cdot e {(t)}_{k}^{r}]^{1 / r} dt$ (7) where p_k( e (t)) stands for the probability mass function (PMF)of the norm of the estimation error at time instant t.

Therefore, we can summarize the definition of the NDES as follows. $\begin{matrix} S_{e} (r, t) \\ = {\begin{matrix} \int_{T_{0}}^{T} {\int e {(t)}^{r} [f (e (t))] d e (t)}^{\frac{1}{r}} dt \\ \int_{T_{0}}^{T} {\sum [p_{i} (e (t))] \cdot e {(t)}_{i}^{r}}^{\frac{1}{r}} dt \end{matrix} \begin{matrix} e (t) continuous \\ e (t) discrete \end{matrix} \end{matrix}$ (8)

Clearly, for any time interval t ∈ [T₀, T], the smaller S_{e
(t)}(r, t) is, the better the estimator is.

Unfortunately, NDES still suffers from the computation problem, because the error distribution is rarely available in dynamic system. Therefore, we will propose a volume error spectrum (VES) to approximate the above NDES in the following part.

3 Computation of NDES

Previously, we proposed an area error spectrum (AES), in which the area under ES curve was used [17,18, 17,18]. That is, $AES (r, t) = \int S (r, t) dr$ (9)

For r ∈ [-1, 2], we have $AES (r, t) = \int_{- 1}^{2} S (r, t) dr$ . Inspired by AES, suppose r ∈ [- ∞ , + ∞] and $t \in [T_{0}, T] = {T_{i}}_{i = 1}^{m}$ , if ${S (r, t_{i})}_{i = 1}^{m}$ holds for identically distributed random variables, then the NDES can be approximately calculated by $VES (r, t) \approx {\begin{matrix} \frac{1}{m} \sum_{i = 1}^{m} {[\frac{1}{M} \sum_{k = 1}^{M} {(e {(t_{i})}_{k})}^{r}]}^{1 / r} r \neq 0 \\ \frac{1}{m} \sum_{i = 1}^{m} {[\prod_{k = 1}^{M} e {(t_{i})}_{k}]}^{1 / n} r = 0 \end{matrix}$ (10)

In this paper, we call the right expression of Equation (10) volume error spectrum (VES), which represents the estimation accuracy within the time interval $[T_{0}, T] = {T_{i}}_{i = 1}^{m}$ of t.

Furthermore, according to Equation (10), we can obtain the following results.

(a)When t is fixed, VES is transformed to the AES, which represents the concentration of the estimation errors.

(b)when S(r, t) >0, then VES(r, t) >0.

As stated before, we can see that VES is a real number in EPE. So, it can be applied to dynamic evaluation systems.

4 Properties of NDES

From Equation (8), we can see that NDES has a series of nice important properties for dynamic performance evaluation.

(a) For any fixed time instant t₀, we have $\begin{matrix} S_{e (t_{0})} (r, t_{0}) \\ = {\begin{matrix} {[\int e {(t_{0})}^{r} [f (e (t_{0}))] d e (t_{0})]}^{1 / r} \\ {[\sum p_{i} e {(t_{0})}_{i}^{r}]}^{1 / r} \end{matrix} \begin{matrix} e (t_{0}) continuous \\ e (t_{0}) discrete \end{matrix} \end{matrix}$ (11)

Clearly, the NDES at t = t₀ becomes the ES, which has all the features of the ES.

(b) For any fixed time instant t₀ and r = 2, S_{e
(t₀)}(r = 2, t₀) =(E [ e (t₀) ²]) ^1/2. If e (t₀) is discrete, $S_{e (t_{0})} (r = 2, t_{0}) = [\sum_{k = 1}^{M} {(∥ \tilde{a} {(t_{0})}_{k} ∥)}^{2} / M]^{1 / 2}$ .

(c) For any fixed time instant t₀ and r = 1, S_{e
(t₀)}(r = 1, t₀) = E [ e (t₀)]. If e (t₀) is discrete, $S_{e (t_{0})} (r = 1, t_{0}) = \sum_{k = 1}^{M} ∥ \tilde{a} {(t_{0})}_{k} ∥ / M$ .

(d) For any fixed time instant t₀ and r = 0, $S_{e (t_{0})} (r = 0, t_{0}) \overset{Δ}{=} lim_{r \to 0} S_{e (t_{0})} (r = 0, t_{0}) = exp (E [ln e (t_{0})])$ . If e (t₀) is discrete, $S_{e (t_{0})} (r = 0, t_{0}) \overset{Δ}{=} exp {\sum_{k = 1}^{M} \frac{\ln [\tilde{a} {(t_{0})}_{k}]}{M}}$ .

(e) For any fixed time instant t₀ and r = -1, S_{e
(t₀)}(r = -1, t₀) = {E [ e (t₀) ^-1]} ^-1. If e (t₀) is discrete, $S_{e (t_{0})} (r = - 1, t_{0}) = {\sum_{k = 1}^{M} {[\tilde{a} {(t_{0})}_{k}]}^{- 1} / M}^{- 1}$ .

(f) For any fixed time instant t₀, if r =+ ∞ or r = -∞, S(+ ∞ , t₀) = max { e (t)} or S(- ∞ , t₀) = min { e (t)}.

So, we can see from (b)-(f) that the most useful parameter intervals of the NDES are the interval [-1, 2] of r and the interval [T₀, T] of t, respectively, which are applied in this paper.

(g) For any fixed r ∈ [- ∞ , + ∞] and any time instant $t \in [T_{0}, T] = {T_{i}}_{i = 1}^{m}$ , S_{e
(t)}(r, t) is strictly bounded: $min {e (t)} \leq S_{e (t)} (r, t) \leq max {e (t)}$ (12) where $\begin{matrix} min {e (t)} = min {[\begin{matrix} e {(T_{0})}_{1} & e {(T_{0})}_{2} & \dots & e {(T_{0})}_{M} \\ e {(T_{1})}_{1} & e {(T_{1})}_{2} & \dots & e {(T_{1})}_{M} \\ ⋮ & ⋮ & ⋮ & ⋮ \\ e {(T_{m})}_{1} & e {(T_{m})}_{2} & \dots & e {(T_{m})}_{M} \end{matrix}]} \\ = min_{i} {min_{k} {e (T_{i})_{k}}_{k = 1}^{M}}_{i = 0}^{m} \end{matrix}$

and $\begin{matrix} max {e (t)} = max {[\begin{matrix} e {(T_{0})}_{1} & e {(T_{0})}_{2} & \dots & e {(T_{0})}_{M} \\ e {(T_{1})}_{1} & e {(T_{1})}_{2} & \dots & e {(T_{1})}_{M} \\ ⋮ & ⋮ & ⋮ & ⋮ \\ e {(T_{m})}_{1} & e {(T_{m})}_{2} & \dots & e {(T_{m})}_{M} \end{matrix}]} \\ = max_{i} {max_{k} {e (T_{i})_{k}}_{k = 1}^{M}}_{i = 0}^{m} \end{matrix}$

Proof: Let $min_{k} {e {(T_{i})}_{k}}_{k = 1}^{M} = min_{k} {e {(T_{i})}_{1}, \dots, e {(T_{i})}_{M}} = e {(T_{i})}_{min}$ and $max_{k} {e {(T_{i})}_{k}}_{k = 1}^{M} = max_{k} {e {(T_{i})}_{1}, \dots, e {(T_{i})}_{M}} = e {(T_{i})}_{max}$ be the maximum and minimum error norm at time T_i, respectively.

Furthermore, let $\begin{matrix} min {e (t)} = min_{i} {min_{k} {e (T_{i})_{k}}_{k = 1}^{M}}_{i = 0}^{m} \\ = min_{i} {e (T_{i})_{min}}_{i = 0}^{m} = e (T_{min})_{min} \\ max {e (t)} = max_{i} {max_{k} {e (T_{i})_{k}}_{k = 1}^{M}}_{i = 0}^{m} \\ = max_{i} {e (T_{i})_{max}}_{i = 0}^{m} = e (T_{max})_{max} \end{matrix}$ be the maximum and minimum error norm in the time interval [T₀, T], respectively. And since ${e ({T_{i}}_{i = 1}^{m})_{k}}_{k = 1}^{M}$ are all positive and according to Equation (10), we have $\begin{matrix} S_{e (t)} (r, t) = \frac{1}{m} \sum_{i = 1}^{m} {\frac{1}{M} \sum_{k = 1}^{M} {[e {(T_{i})}_{k}]}^{r}}^{1 / r} \\ = \frac{1}{m} \sum_{i = 1}^{m} {\frac{{[e {(T_{i})}_{1}]}^{r} + {[e {(T_{i})}_{2}]}^{r} + \dots + {[e {(T_{i})}_{M}]}^{r}}{M}}^{1 / r} \\ \leq \frac{1}{m} \sum_{i = 1}^{m} {\frac{{[e {(T_{i})}_{max}]}^{r} + {[e {(T_{i})}_{max}]}^{r} + \dots + {[e {(T_{i})}_{max}]}^{r}}{M}}^{1 / r} \\ = \frac{1}{m} \sum_{i = 1}^{m} {{[e {(T_{i})}_{max}]}^{r}}^{1 / r} \\ = \frac{{{[e {(T_{1})}_{max}]}^{r} + {[e {(T_{2})}_{max}]}^{r} + \dots + {[e {(T_{m})}_{max}]}^{r}}^{1 / r}}{m} \\ \leq \frac{{{[e {(T_{max})}_{max}]}^{r} + {[e {(T_{max})}_{max}]}^{r} + \dots + {[e {(T_{max})}_{max}]}^{r}}^{1 / r}}{m} \\ = e (T_{max})_{max} \\ = max {e (t)} \end{matrix}$ (13)

Similarly, $\begin{matrix} S_{e (t)} (r, t) = \frac{1}{m} \sum_{i = 1}^{m} {\frac{{[e {(T_{i})}_{1}]}^{r} + {[e {(T_{i})}_{2}]}^{r} + \dots + {[e {(T_{i})}_{M}]}^{r}}{M}}^{1 / r} \\ \geq \frac{1}{m} \sum_{i = 1}^{m} {\frac{{[e {(T_{i})}_{min}]}^{r} + {[e {(T_{i})}_{min}]}^{r} + \dots + {[e {(T_{i})}_{min}]}^{r}}{M}}^{1 / r} \\ = \frac{1}{m} \sum_{i = 1}^{m} {{[e {(T_{i})}_{min}]}^{r}}^{1 / r} \\ \geq \frac{{{[e {(T_{1})}_{min}]}^{r} + {[e {(T_{2})}_{min}]}^{r} + \dots + {[e {(T_{m})}_{min}]}^{r}}^{1 / r}}{m} \\ \geq \frac{{{[e {(T_{min})}_{min}]}^{r} + {[e {(T_{min})}_{min}]}^{r} + \dots + {[e {(T_{min})}_{min}]}^{r}}^{1 / r}}{m} \\ = e (T_{min})_{min} \\ = min {e (t)} \end{matrix}$ (14)

Clearly, by Equation (13) and Equation (14), we can conclude that $min {e (t)} \leq S_{e (t)} (r, t) \leq max {e (t)}$

So, S_{e
(t)}(r, t) is strictly bounded.

(h) For any fixed r ∈ [- ∞ , + ∞] and e (t) remains unchanged for any time instant t ∈ [T₀, T], then: S_{e
(t)}(r, t) is strictly monotone increasing with respect to r, i.e., S_{e
(t)}(r, t) < S_{e
(t)}(s, t) if r < s. In other words, the curve of S_{e
(t)}(s, t) curve is higher than the S_{e
(t)}(r, t).

5 Numerical example

Hereafter, a numerical example is designed to illustrate the superiority of the NDES in performance evaluation.

5.1 The state estimation models

Suppose that a nonlinear non-Gaussion state estimation model is given by ${\begin{matrix} x_{k + 1} = f (x_{k}) + w_{k} \\ y_{k + 1} = h (x_{k + 1}) + v_{k + 1} \end{matrix}$ (15) where f(·) and h(·) are nonlinear functions as follows $\begin{matrix} f (x_{k + 1}) = 0.5 x_{k} + sin (0.04 \times π \times t) + 1 \\ h (x_{k + 1}) = {\begin{matrix} {x_{k + 1}}^{2} / 5 + sin (x_{k + 1}) \\ x_{k + 1} / 2 - 2 \end{matrix} \begin{matrix} k < 30 \\ Otherwise \end{matrix} \end{matrix}$ (16)

and the state noise w _k and the measurement noise v _k are ${\begin{matrix} w_{k} \sim Gamma (0, 0.01) \\ v_{k + 1} \sim N (0, 0.01) \end{matrix}$ (17) where Gamma(α, β) is the Gamma distribution with the shape α = 0 and date parameters β = 0.01, and N(μ, Σ) is the Gaussian distribution with the mean μ = 0 and variance Σ = 0.01.

Furthermore, we used two types of nonlinear estimation algorithms (NEA) [20 –23], such as extended Kalman filter (EKF) [24, 25] and unscented Kalman filter (UKF) [26 –28] to estimate the above nonlinear Gaussian models, where the noise distributions predictive and filtering were assumed Gaussians, and the linearization of the functions in the process and observation equations was employed in EKF. However, the UKF shows that it is easier to approximate a probability distribution than to approximate an arbitrary nonlinear function or transformation, that is, the UKF is a more suitable method for calculating the statistics of a random variable.

5.2 Parameter initialization

The initialization parameters of the EKF and UKF were summarized in Table 1.

Table 1
Initialization parameters

Algorithm Initialization parameters Time

EKF $\begin{matrix} {\hat{x}}_{0} = 1, P_{0} = 3 / 4 \\ Q_{0} = 3 / 4, R_{0} = 0.01 \end{matrix}$ 50s

UKF $\begin{matrix} {\hat{x}}_{0} = 1, P_{0} = 3 / 4, Q_{0} = 3 / 4 \\ R_{0} = 0.01, α = 1, β = 0, λ = 0 \end{matrix}$ 50s

Algorithm	Initialization parameters	Time
EKF	$\begin{matrix} {\hat{x}}_{0} = 1, P_{0} = 3 / 4 \\ Q_{0} = 3 / 4, R_{0} = 0.01 \end{matrix}$	50s
UKF	$\begin{matrix} {\hat{x}}_{0} = 1, P_{0} = 3 / 4, Q_{0} = 3 / 4 \\ R_{0} = 0.01, α = 1, β = 0, λ = 0 \end{matrix}$	50s

5.3 Analysis of the simulation results

In the following, the NDES was applied to evaluate the performance of the EKF and UKF. Let T = 50s(m = 50), after one Monte-Carlo run, we obtained a tracking result of the above two types of NEA, respectively. Furthermore, over 500 (M = 500) Monte-Carlo runs and at every time instant t, the average tracking result was utilized to show the performance of the EKF and UKF shown in Fig. 1, which shows that the measurement y matches the state x within the time interval [0, 8s] and [22s, 30s], and on the contrary, the measurement y mismatches the state x within the time interval [9s, 21s] and [31s, 50s].

Fig. 1

The average tracking results of the EKF and UKF.

In this paper, the average tracking result was calculated as ${\begin{matrix} \bar{x} {(t)}^{EKF} = \sum_{k = 1}^{M} \hat{x} (t)_{k}^{EKF} / M \\ \bar{x} {(t)}^{UKF} = \sum_{k = 1}^{M} \hat{x} (t)_{k}^{UKF} / M \end{matrix}$ (18)

Thus, the corresponding estimation errors at time instant t were given by ${\begin{matrix} e {(t)}^{EKF} = {\hat{x} (t)_{k}^{EKF} - x (t)}_{k = 1}^{M} \\ e {(t)}^{UKF} = {\hat{x} (t)_{k}^{UKF} - x (t)}_{k = 1}^{M} \end{matrix}$ (19)

Substituting Equation (19) into the NDES, the evaluated results were shown in Fig. 2, which shows the evaluated results obtained by the NDES. Figure 3 are the vertical view, the front view and the right view of the NDES, respectively.

Fig. 2

The NDES of the EKF and UKF.

Fig. 3

t - r plane to the NDES.

As depicted in Fig. 2, it is hard to see which NEA performs better. However, we can see from Fig. 3 that the NDES aggregates several commonly incomprehensive measures such as the root mean square error (RMSE), average Euclidean error (AEE), geometric average error (GAE), and harmonic average error (HAE).

To illustrate the advantages of NDEs, Fig. 4 shows the evaluation results based on the above commonly incomprehensive measures, which are consistent with the results obtained by the NDES evaluation in Fig. 3.

From Figs. 3 and 4, we can see further that the UKF outperforms the EKF within the time intervals [0s, 31s] according to the AEE or the RMSE. On the contrary, the latter is better than the former according to the HAE. So, it is difficult to see which NEA performs better according to the above incomprehensive measures. Fortunately, the NDES can provide a good solution to this problem based on Equation (10). In other words, for [T₀, T] = [0, 5] and r = [-1, 2], applying the VES to this case, we obtain $NDE S_{EKF} = 0.4869 > NDE S_{UKF} = 0.2453$ (20)

Fig. 4

Incomprehensive measures of the EKF and UKF.

That is $EKF ≺ UKF$ (21) where A ≻ B means that A outperforms B.

Obviously, the calculation accuracy of the UKF is superior to the EKF.

In addition, we can also obtain the computational time of the EKF and the UKF shown in Fig. 5.

Fig. 5

The estimation time of the EKF and UKF.

Furthermore, the average computational time is calculated as ${\begin{matrix} AC T_{EKF} = \sum_{k = 1}^{M} (\sum_{i = 1}^{m} t_{k, i}^{EKF} / m) / M = 0.0152 \\ AC T_{UKF} = \sum_{k = 1}^{M} (\sum_{i = 1}^{m} t_{k, i}^{UKF} / m) / M = 0.0431 \end{matrix}$ (22)

Although the computational load of the UKF is larger than the EKF, the UKF still meets the computational requirements.

As stated before, we can conclude that the UKF has obvious advantages in calculation accuracy except the computational load usually meeting the computational requirements [29].

6 Conclusion

The main contribution of this paper is that a new dynamic error spectrum (NDES) was proposed for dynamic system performance evaluation. Firstly, the NDES was defined according to the definition of ES. Secondly, the computation method was proposed to calculate the NDES. Thirdly, several nice properties of NDES were presented for dynamic system performance evaluation. Finally, simulations show that the NDES is intuitive and reflects the estimation performance directly in dynamic evaluation systems. Furthermore, the NDES can applied the metric to dynamic systems and can be more powerful, and a quick evaluation result were given by using VES in dynamic systems. In future work, we will apply NDES to the evaluation of nonlinear estimation algorithm and dynamic system performance evaluation and ranking.

Footnotes

Acknowledgments

We would like to thank anonymous reviewers for their helpful comments and suggestions. This work was supported in part by the national natural science foundation of China through grant No. 72271243, No. 71801222 and No. 61973253, the national science foundation of shaanxi province of China through grant No. 2018JQ6019, the national postdoctoral program for innovative talents through grant No. BX201700104.

References

X.R.

and Zhao

Z.L.

, Evaluation of estimation algorithms part I:incomprehensive measures of performance, IEEE Transactions onAerospace and Electronic Systems 42(4) (2006), 1340–1358.

Tyagi

and Sumathi

, Comprehensive Performance Evaluation ofComputationally Efficient Discrete Fourier Transforms for FrequencyEstimation, IEEE Transactions on Instrumentation andMeasurement 69(5) (2020), 2155–2163.

Zhang

, Lan

and Li

X.R.

, Performance evaluation of jointtracking and classification, IEEE Transactions on Systems, Man,and Cybernetics: Systems 51(2) (2021), 1149–1163.

Yang

, Zhang

W.A.

and Yu

, Performance evaluation of distributedlinear regression kalman filtering fusion, IEEE Transactions onAutomatic Control 66(6) (2021), 2889–2896.

Peng

W.S.

, Li

Y.H.

, Fang

Y.W.

et al., Radar Chart for EstimationPerformance Evaluation, IEEE Access 7 (2019), 113880–113888.

Peng

W.S.

, Fang

Y.W.

, Yong

X.J.

et al., Measures formultiple-attribute estimation ranking, Optik 127(12) (2016), 9479–9487.

Yin

, Li

X.R.

and Lan

, Pairwise Comparison Based Ranking VectorApproach to Estimation Performance Ranking, IEEE Transactionson Systems, Man, and Cybernetics: Systems 48(6) (2018), 942–953.

X.Q.

, Duan

Z.S.

, Hanebeck

U.D.

Performance ranking of multiple nonlinear filters using ranking vector and voting fusion, In Proc. 20th Int. Conf. Information Fusion, xi’an, China July 2017, pp. 574–581.

Shao

, Duan

and Tian

, Performance Ranking of Kalman FilterWith Pre-Determined Initial State Prior, IEEE Signal ProcessingLetters 28 (2021), 902–906.

10.

, Gao

, Zhang

et al., Deep learning for channelestimation: interpretation, performance, and comparison, IEEETransactions on Wireless Communications 20(4) (2021), 2398–2412.

11.

, He

, Yang

et al., Diffusion normalized least meanM-estimate algorithms: design and performance Analysis, IEEETransactions on Signal Processing 68 (2020), 2199–2214.

12.

Meng

, Zhang

, Stĺźbercň

G.L.

et al., Blindfast CFO estimation and performance analysis for OFDM, IEEETransactions on Vehicular Technology 69(10) (2020), 11501–11514.

13.

X.R.

, Zhao

Z.L.

, Duan

Z.S.

Error spectrum and desirability level for estimation performance evaluation, In Proc. Workshop on Estimation, Tracking and Fusion: A Tribute to Fred Daum, Monterey, CA, USA, May 2007, pp. 1–7.

14.

Liu

, Li

X.R.

Computation of error spectrum for estimation performance evaluation, In Proc 2013 Int. Conf. Information Fusion, Istanbul, Turkey, July 2013, pp. 477–483.

15.

Peng

W.S.

, Fang

Y.W.

, Zhan

R.J.

et al., Two ApproximationAlgorithms of Error Spectrum for Estimation Performance Evaluation, Optick 127(5) (2015), 2811–2821.

16.

Mao

Y.H.

, Han

C.Z.

and Duan

Z.S.

, Dynamic error spectrum forestimation performance evaluation: a case study on interactingmultiple model algorithm, IET Signal Processing 8(2) (2014), 202–210.

17.

Peng

W.S.

, Fang

Y.W.

and Chai

, Enhanced dynamic error spectrum for estimation performance evaluation in target tracking, Optick 127(8) (2016), 3943–3949.

18.

Peng

W.S.

, Fang

Y.W.

, Duan

Z.S.

and Wang

H.B.

, Enhanced Error Spectrum for Estimation Performance Evaluation, Optick 127(8) (2016), 5084–5091.

19.

Yin

H.L.

, Li

X.R.

and Lan

, Iterative Mid-Range with Applicationto Estimation Performance Evaluation, IEEE Signal ProcessingLetters 22(11) (2015), 2044–2048.

20.

Wang

, Lyu

and Ren

, Unscented-Transformation-BasedDistributed Nonlinear State Estimation: Algorithm, Analysis, andExperiments, IEEE Transactions on Control Systems Technology 27(5) (2019), 2016–2029.

21.

Emilio Mariana-Daniela

A.S.

, Zamar

C.I.M.

, Juan

et al., Sustainable Management of Digital Transformation in HigherEducation: Global Research Trends, Sustainability 12(5) (2020), 2107.

22.

Chen

and Liu

, Robust Distributed Parameter Estimation ofNonlinear Systems With Missing Data Over Networks, IEEETransactions on Aerospace and Electronic Systems 56(3) (2020), 2228–2244.

23.

Meng

and Li

, Distributed Nonlinear Estimation Over UnbalancedDirected Networks, IEEE Transactions on Signal Processing 68 (2020), 6212–6223.

24.

Wang

, Huang

, Gao

et al., Adaptive Extended Kalman FilterBased Dynamic Equivalent Method of PMSG Wind Farm Cluster, IEEETransactions on Industry Applications 57(3) (2021), 2908–2917.

25.

Dai

and Jing

, Lightweight Extended Kalman Filter for MARGSensors Attitude Estimation, IEEE Sensors Journal 21(13) (2021), 14749–14758.

26.

Zhao

and Mili

, A Decentralized H-Infinity Unscented KalmanFilter for Dynamic State Estimation Against Uncertainties, IEEETransactions on Smart Grid 10(5) (2019), 4870–4880.

27.

Vaishnav

and Santra

, Continuous Human Activity ClassificationWith Unscented Kalman Filter Tracking Using FMCW Radar, IEEESensors Letters 4(5) (2020), 1–4.

28.

Dang

, Chen

, Wang

et al., Robust Power System StateEstimation With Minimum Error Entropy Unscented Kalman Filter, IEEE Transactions on Instrumentation and Measurement 69(11) (2020), 8797–8808.

29.

Liu

, Wang

, Chen

et al., Protocol-Based Unscented KalmanFiltering in the Presence of Stochastic Uncertainties, IEEETransactions on Automatic Control 65(3) (2020), 1303–1309.

New dynamic error spectrum for performance evaluation

Abstract

Keywords

1 Introduction

2 Definition of NDES

5.1 The state estimation models

Table 1 Initialization parameters Algorithm Initialization parameters Time EKF x ˆ 0 = 1 , P 0 = 3 / 4 Q 0 = 3 / 4 , R 0 = 0.01 50s UKF x ˆ 0 = 1 , P 0 = 3 / 4 , Q 0 = 3 / 4 R 0 = 0.01 , α = 1 , β = 0 , λ = 0 50s

Footnotes

Acknowledgments

References