Kalman tracking algorithm of ping-pong robot based on fuzzy real-time image

2.1 Related work

Gupta studied the development and implementation of a new robust vision controller for humanoid robots. The vision controller is composed of two parts: 1) robust algorithm, which can track people’s line of sight in its camera view; 2) servo controller, which generates necessary motion commands so that the robot can follow the target person. Tracking algorithms use point-based features (such as robust features for acceleration) to detect people under challenging conditions, such as changes in light, posture, complete or partial occlusion, and sudden camera movement. His innovative contributions to the tracking algorithm include: 1) dynamic object model, which develops over time to cope with short-term changes while maintaining stability in the long term; 2) the online k-d tree-based classifier and kalman filter are used to distinguish the situation of attitude change and the situation of partial or complete occlusion; 3) he proposed a method to detect postural changes caused by rotation outside the plane, which was a difficult problem and led to frequent tracking failures of human follow-up robots. He proposes an improved version of a visual servo controller that uses feedback linearization to overcome the vibration in previously used slide-mode based controllers. Through various simulation and actual experiments on the actual mobile robot platform, he proved the effectiveness of the method he proposed [14, 15]. Yang proposed a new nonlinear predictive control strategy based on extreme learning machine to solve the path tracking control problem of wheeled mobile robot with external disturbance. Hybrid chaotic optimization algorithm (HCOA) can be used to solve the real-time nonlinear minimization problem of the cost function of mobile robot. The algorithm can avoid falling into local minima and improve the convergence and control accuracy when dealing with large space and high-dimensional optimization problems. His proposed improved bidirectional extreme learning machine is used to model the mobile robot factories and estimate future factory output. The experimental results of the tracking automatic mobile robot studied by him show that compared with the traditional neural network prediction controller, this controller can provide higher accuracy and faster tracking performance [16, 17]. Based on the fact that the motion of a rotating ball can be divided into translational motion and rotation relative to the center of the ball, Zhang proposed a novel visual system that can provide a high-precision mode of real-time position and rotation information of the flying ball. He used a frame-based approach to identify the natural brand of the ball under normal lighting conditions. You can then restore the ball’s 3D position in spherical coordinates. Through observation and analysis, he found that the axis and angular velocity of spin did not change during flight, and he could use the plane fitting method based on the consensus of weighted random samples to estimate the spin state. By combining the position information and rotation information in the dynamic model based on force, the extended kalman filter can achieve accurate trajectory prediction. His experimental results have proved the validity and accuracy of this method [18, 19]. Elsheikh proposed a practical model-free path planning and path tracking algorithm, which uses multiple cameras and decentralized fuzzy logic controller for a non-integrated indoor wheeled mobile robot. His algorithm is divided into three stages. The first stage uses the multi-template fast moving (MSFM) path planning method. In general, the path generated by the direct execution of the fast moving method is not guaranteed to be safe or smooth. The robot can then touch corners, walls and other obstacles. Based on the size of the robot, the algorithm adds a preprocessing stage before MSFM programming method to solve such problems. The second phase uses visual information extracted from images captured by multiple cameras to estimate the position and direction of the mobile robot in each frame. In the third stage, a decentralized control strategy is used to connect three identical proportional differential fuzzy logic controllers (PD FLC) in parallel as a path follower to keep the robot in the desired path. His research results show that the design he developed can effectively estimate the shortest path, avoid obstacles and guide the robot to track the path in real time [20, 21].

2.2. Kalman filtering theory

(1) Kalman filter

Kalman filtering algorithm is the most widely used in modern target filtering system, and most other derived algorithms are developed on the basis of kalman filtering. Discrete kalman filter was first proposed, which overcame the application defects of wiener filter. After that, continuous time kalman filter was popularized, which completely solved the engineering implementation problem of the filter algorithm. Kalman algorithm is an optimal condition estimation algorithm, its basic principle is linear, unbiased, minimum error variance. The essence of this method is to use the recursive data processing method to obtain new state estimates from the measured values obtained from the system through the determined recursive equation. In kalman filter algorithm, white noise is applied to the whole system signal process and produces a linearized output result. The kalman algorithm contains several statistical properties, which are obtained by the combination of the state space equation that represents the target model, the observation equation that represents the filter estimation characteristic, and the observation noise and the system noise. Therefore, kalman filter algorithm can estimate one – dimensional and multi – dimensional processes as well as stationary and non – stationary processes. Based on kalman theory, after years of research, many derived filtering algorithms have been developed and achieved good filtering effects, and their application scope has been extended to navigation, communication, seismic detection, mineral development, biomedicine and other aspects. Compared with other types of filtering algorithms, kalman algorithm has the following three significant characteristics: one is that the algorithm has two forms of continuous and discrete, and can be analyzed in the time domain, and the computer can directly implement the discrete algorithm; the second is algorithm recursion, which continuously obtains new state estimation through recursive equation. The measurement information at different moments does not need to be stored by the computer, and the estimation information at the current moment can be obtained in real time. The statistical characteristics of the estimated parameters can be determined by the statistical characteristics of the input noise and the system state equation. Based on the above three characteristics, compared with other kalman filtering algorithm is more widely used in target tracking, and has the following advantages: (1) the system can gain access automatically by the algorithm, so the universal applicability of the kalman algorithm is stronger, through partial parameters change, the same filter design can be applied to other models; (2) the covariance matrix can be used to conveniently measure the estimation accuracy. Meanwhile, in non-single target tracking, the formation of tracking gate and the determination of threshold can be measured and obtained by this method. (3) the residual (novelty) in kalman filter can be used as a means of target maneuver detection and identification. Through the consistency analysis of the change of residual (novelty), we can determine whether the approximate target motion model is reliable and consistent with the real motion characteristics of the target. (4) during the tracking and detection process, kalman gain sequence can be adaptively adjusted according to the change of state, so as to achieve the optimal filtering effect.

Kalman filter is a method to estimate the expected signal according to the model and observation of the filter signal. This method introduces the concept of state space in stochastic estimation theory and treats signal processing as the output of linear system under the action of white noise. The equation describing the relationship between input and output is called the equation of state. A recursive algorithm occurs when a state estimate of the current time requires measurements of the previous time. A very simple and convenient feature of recursive algorithm is that it allows serialization of the received data instead of batch processing, so we don’t need to store all the data, and we don’t need to re-process the existing data when the data is updated. Kalman filter is a real – time recursive algorithm for stochastic signal estimation. The filter estimates the random signal by observing the input system and takes the estimated signal as the output of the filter. The observation update and time update algorithms determine the input and output of the filter. According to the statistical characteristics of system noise and observation noise, the observation equation and system state equation are used to estimate the filter signal [22].

Kalman filter describes the system through dynamic equation and observation equation. Even without knowing the exact nature of the model, it can use the past of the signal to correct the current state and even estimate the future state. The initialization of kalman filter is relatively simple. It can start at any point. The finite order filtering operation can accurately estimate the real position of the target, and the estimation of the future motion state can be continuously modified by using the actual motion parameters. The recursive filtering method is used for calculation, which has the characteristics of small computation amount, strong real-time performance and optimization of state space model through computer program. The principle is shown in Fig. 1.

OPEN IN VIEWER

Fig. 1

The filter loop of the Kalman filter.

The mathematical model of kalman filter is first order differential equation, not high order differential equation. Both continuous and discrete systems are first order differential equations, especially in computer processing. For each filter calculation, only the estimation data of the previous moment and the current observation data can be used to estimate the estimation value of the next moment [23]. The recursive nature of kalman filter is very suitable for computer operation, because it does not need to store historical data, reducing the need for computer storage. In target tracking, kalman filter is used to predict the target position as the starting point of Camshift iteration process, which greatly reduces the number of iterations and thus improves the computing speed.

(2) Kalman filter basic equation of random linear discrete system.

The process of kalman filter is described by difference equation and discrete observation equation with random initial state, observation noise and process noise. Without considering the control effect, the equation of random linear discrete system is X k = AX k - 1 + BU k - 1 + W k - 1(1) Z k = HX k + V k(2)

State variable X ∈ Rⁿ, observation variable Z ∈ R^m, random signal V_k and W_k represent observation noise and system process noise respectively. Assuming that they are mutually independent and normally distributed white noise, the statistical characteristics of observed noise and system process noise are shown in formula (3).

E [ W k ] = 0 , E [ W k W j T ] = Q k δ kj E ( V k ) = 0 , E [ V k V j T ] = R k δ kj E [ W k V j T ] = 0(3)

Where, Q _k is the p×p dimensional symmetric non-negative definite variance matrix of system process noise W_k; R_k is systematic observation.

The m×m dimension symmetric positive definite variance matrix of noise R_k; δ _kj is the Kronecker function.

Kalman filter consists of measuring updating equation and time updating equation. The modified posteriori estimate is obtained by combining the calculated observed variables with the prior estimate by using the measurement update equation, and the result of the posterior estimate is taken as the prior estimate of the state at the next moment. The value of the error covariance and the current state variable are deduced by using the time updating equation. To sum up, kalman filter estimates the state of the system through real-time correction: first, the state of the system at a certain moment is estimated through the time update equation, and then the observed variables are measured through the update equation to correct the estimated state of the system.

The observed value at moment k is Z_k, and the optimal state estimation X_k-1 of X ˆ k - 1 at moment k-1 has been obtained. According to equations (1) and (2), the basic Kalman filter equation of random linear discrete system can be obtained, and its state prediction equation is: X ˆ k - = A X ˆ K - 1 + BU K - 1(4)

Covariance equation for prediction state P K - = AP K - 1 A K T + Q(5)

Filter gain equation K K = P K - H T ( HP K - H T + R ) - 1(6)

State optimization estimation equation X ˆ K = X ˆ K - + K K ( Z K - H X ˆ K - )(7)

Covariance equation of state optimization estimation P K = ( I - K K H ) P K -(8)

Where, Equation (5) can be further written as K K = P K H T R - 1(9)

Equation (7) can be further written P K = P K - + H T R - 1 H(10)

As shown in Fig. 2, it is the block diagram of Kalman filter of random linear discrete system obtained from Equations (4) and (7). The observed variable is the input signal of the filter, and the obtained optimization estimate is the output signal of the filter.

OPEN IN VIEWER

Fig. 2

Block diagram of Kalman filter for random linear discrete system.

Using the posterior estimate calculated by the system observation equation and time update equation as the next prior estimate, the whole recursive process is repeated. The recursive repetition of Kalman filter determines that it is more time efficient than other filters. It only needs recursion based on the current state estimation of the observed variable and the posterior estimation of the previous point, which is simple to realize [24].

Another representation of kalman filter algorithm is shown in Fig. 3. The kalman filter algorithm is divided into two parts: measurement update and time update. Where the error covariance can be calculated independently, X ˆ k - 1 and P_K-1 are the initial estimates.

OPEN IN VIEWER

Fig. 3

Flow chart of Kalman filter algorithm.

(3) Characteristics of Kalman filtering algorithm

By analyzing and studying the basic equation of Kalman filter for stochastic linear discrete system, the algorithm equation is summarized to have the following characteristics:

1) Kalman filter is the signal process of desired signal continuous prediction and correction, filtering equation is a recursive algorithm based on time axis, don’t need to store large amounts of data in the calculation process, as long as there is a new measurement data, a new prediction can be calculated, so the kalman filter algorithm has less amount of calculation, and high real-time performance, the advantages of easy to computer implementation.

2) Kalman filter can be regarded as a linear system under the action of random white noise and random signal and the signal is the system input and output [25], but the relationship between the input and output based on the measuring equation and state equation, thus the filter method is not only applicable to stationary random signal filtering, also applies to gauss – markov sequences such as filtering, application scope is very broad.

(1) Feature-based tracking method

In the two consecutive frames of images in video, there are a lot of feature information of moving targets. Since the time interval between frames is very short, it can be considered that the feature information of these targets is constant for a period of time, and it can effectively track moving targets according to the feature information. In this way, even if the target is partially occluded, the problem of the target being partially occluded can still be solved according to the feature information contained in the whole target. In this paper, CamShift tracking algorithm based on features is studied. The feature-based tracking method has its obvious advantages: (1) Iin the tracking process, the target features are easy to be detected and can be well matched with the target features; (2) Because the establishment is a simple feature model, and feature information has a short time stability, so the tracking algorithm based on features is relatively simple; (3) Because the established feature models are independent of each other, it is not necessary to consider the shape and size of the target in the process of target tracking. But it also has its significant disadvantages: (1) If the target is simple but the background is complex, it will affect the accuracy of target feature information extraction; (2) The nonlinearity of the target characteristic parameters leads to the fact that the recovered characteristic parameters in the target tracking are easily affected by noise; (3) If the target is blocked at the initial moment, it is difficult to initialize the target features. Therefore, the effectiveness of the target tracking method based on features is greatly affected by the accuracy of the initial feature information, and the target tracking algorithm will become more complicated due to external interference.

(2) Tracking method based on active contour

The edge contour feature of moving target can provide the target feature information independent of the shape and motion of the target. Snake algorithm is an effective detection and tracking algorithm. It uses external force, binding force and internal force to make the curve move towards the target contour. A curve is considered a curve when it stops moving, is the outline of the target being tracked. Since this method is based on the global information of the target contour, no prior knowledge of the moving target is needed to obtain the contour curve, so it is widely used in target tracking, image segmentation and edge detection. However, the snake model also has some disadvantages: 1. The initial contour should not be too far from the real edge of the target, otherwise the tracking results will be inaccurate. Traditional model cannot get the curve of the deep concave part of the target. Active contour algorithm includes two algorithms: geodesic active contour and parametric active contour. The mathematical model of geodesic active contour uses the level set method to represent the curve in the high-dimensional space, which only contains the parameters related to the curve geometry, while the parametric active contour uses the contour parameters to represent the motion curve. Compared with the geodesic active contour model, the parametric active contour model must be controlled by the outside or given the initial value, while the geodesic active contour model can adjust the geometric changes of curves in the process of motion without any assistance from the outside conditions. Active geodesic contour model is more suitable for multi-motion target tracking.

(3) 3D based tracking method

The motion target tracking method based on 3D is to use the camera to shoot different angles of the target in the scene, so that the 3D model with a complete geometric shape becomes a plane image, and the tracking is realized according to the change of the model position in the image. In the XTRACK system, the position of the vehicle is obtained by detecting the initial frame in the video sequence image by optical flow method, and the detected target position is compared with the reference point initially set in the system, so as to calculate the speed and direction of the target. Its initial angular velocity is obtained through a multidimensional vehicle model. The moving object tracking method based on 3D can recover the 3D model of moving object based on plane image, which has been widely applied in the moving object tracking system of video. The 3d-based tracking method can make good use of the results of model knowledge to track moving objects in complex scenes, but it has the disadvantages of poor real-time performance and heavy workload.

(4) Tracking method based on motion estimation

Moving target tracking based on motion estimation is one of the most widely used methods in target tracking. The motion parameters in the moving target tracking method based on motion estimation are obtained by optical flow detection, which mainly includes two methods: global optical flow field and feature point optical flow field. Compared with the global optical flow field method, the optical flow field of feature points can obtain the target position only by matching the target features, which has the characteristics of high real-time performance and small computation. However, because the optical flow field is based on feature points, it is difficult to obtain the global information of moving targets. In general, the estimation feature information of moving target may not be accurate and reliable due to occlusion, shadow, and change of light source, monochromatic interference and noise. In addition, the existing optical flow algorithm has poor real-time performance and high complexity. Unless there is special hardware support, it is difficult to realize the real-time detection of moving objects [26].

3.1 Experimental design

The real device used in this experiment is shown in Fig. 4. The robot adopts monocular vision and motion servo joints with 7 degrees of freedom developed by the robot. R_k is according to the result of camera calibration, set to measure the maximum error. In this experiment, R_k initial value is set to 10 mm, with the decrease of the d_k, R_k value increased gradually to about 40 mm. This equipment uses the camera and the BALS – ER company A602FC series of color digital camera, win – dows XP operating system, CPU for P4 2 · 6 GHz), OK_C30A image acquisition card.

OPEN IN VIEWER

Fig. 4

Table tennis robot.

The experimental data in this paper are realized by camera. During the ping-pong test of the ping-pong robot, the scene was photographed and the images were processed by the computer at a high speed to complete the field experiment. In order to speed up the time of image positioning algorithm, camera keyhole perspective model (a simple linear geometric perspective model) was adopted for the experimental data acquisition in this paper.

4.1 Analysis of Kalman tracking algorithm of ping-pong robot based on fuzzy real-time image

(1) Analysis of tracking effect

As shown in Fig. 5, describes the ping-pong X axis direction of the speed v tracking results. As can be seen from the Fig. 4, adaptive measurement covariance of the Kalman filter initial tracking phase (solid line) and conventional Kalman filter (dashed line) are set up relatively small measurement covariance, therefore the filter output almost no obvious differences. But as time goes on, the measurement of adaptive covariance of the Kalman filter covariance increases gradually, and the tracking results tend to be stable, and the output of the general Kal – man filter noise larger all the time, not convergence.

OPEN IN VIEWER

Fig. 5

X-axis velocity tracking results.

(2) Analysis of anti-noise interference ability

As shown in Fig. 6, the anti-jamming performance of filter, is a important index to measure real-time visual tracking. As the experimental results verify the algorithm in this paper, the original measurement data with another in the field of digital filtering algorithms for filtering – sliding window filtering algorithm were compared. Figure 6 depicts introduce noise in the process of Z axis measurement of speed tracking curve. As shown in Fig. 6, in the time of 200 Ms nearby, introduced the measurement noise signal, the results of measurement are usually deviated from the actual trajectory, differential speed performance for larger disturbance (as shown in the upper point of a circle). From the curve analysis, adaptive measurement covariance of the Kalman filter (solid line) can more effectively suppress the noise impact on output, less volatile, in contrast, with a dotted line sliding window filter output curve near the disturbance of the emergence of a large pulse jitter [27]. Therefore, in this paper, the method has a better anti-jamming ability.

OPEN IN VIEWER

Fig. 6

Tracking results after adding noise.

(3) Image and light interference analysis

Due to reasons such as image processing, light or shade, unable to get the accurate information of table tennis, means that data loss may occur. Figure 7 are analyzed in this case the tracking results. As shown in Fig. 7, in 150 ms to 210 ms, no any measurement signal input, then from the point of speed tracking curve, adaptive measurement covariance Kalman filtering algorithm for data loss, has better ability to adapt, the output is relatively smooth, and the output of the sliding window filtering algorithm has larger dithering.

OPEN IN VIEWER

Fig. 7

Data loss tracking results.

(4) Analysis results of calculated quantity

As shown in Fig. 8, it is the curve diagram of the target center point obtained by the original algorithm and the improved algorithm after the moving ping-pong ball encounters the same color interference. Table 1 still shows the absolute error between the center point position of the tracking target and the true value.

OPEN IN VIEWER

Fig. 8

Coordinates of target center point with or without the same color interference processing.