Prediction and analysis of sphere motion trajectory based on deep learning algorithm optimization

Abstract

Ball sports have great variability in the game and the intelligent control of the rules of ball movement can effectively improve the training effect of athletes. However, the current research on artificial intelligence of spherical motion trajectory prediction points is basically blank. Based on this, this study is based on deep learning technology, and obtains the main experimental data through network data collection in the research and builds the table tennis spatial position image data set under various environments with accurate annotation based on the traditional deep learning. At the same time, the convolutional neural network is used as the location recognition algorithm, and a prediction algorithm for predicting the trajectory of table tennis is proposed based on the recurrent neural network. In addition, this paper designs comparative experiments to analyze the effectiveness of the algorithm model, and evaluates the real-time recognition, location and trajectory prediction capabilities, and conducts quantitative analysis. The research shows that the algorithm has certain practical effects and can provide theoretical reference for subsequent related research.

Keywords

Deep learning neural network trajectory recognition algorithm prediction model

1 Introduction

The detection and tracking technology of moving targets is an important research content of stereo vision technology. The essence is to analyze the sequence of multiple video frames captured by two or more cameras, extract a series of imaging coordinate pairs of moving targets in the image sequence according to the characteristics of the moving targets, analyze the coordinate pairs, further obtain the three-dimensional position sequence of the moving target, and realize the motion tracking of the target [1]. Currently, motion detection and tracking technology incorporates advanced technologies in many other fields, such as image processing, artificialintelligence, pattern recognition and automatic control. It has a wide range of applications in robot navigation, public scene monitoring and military vision guidance [2]. In the past visual technology, monocular vision technology has been applied more and has been better developed. However, due to the limitations of monocular vision, it is very difficult for a moving monocular camera to track moving targets. The binocular stereo vision system simulates the binocular mechanism of human beings, and acquires the target image pairs synchronously and simultaneously, and can extract the parallax information of the target in the stereo image pair in real time and realize the tracking of the moving target. Therefore, the research on the detection and tracking of moving targets using binocular stereo vision technology has very important significance in the fields of target measurement, target tracking and three-dimensional information recovery [3].

In table tennis, table tennis is small and flying fast. This requires the table tennis robot to have a continuous rapid response capability, and it takes a short time to complete the trajectory prediction of the high-speed sports table tennis and make an accurate hit. This puts great demands on the real-time and accuracy of the control of table tennis robots. Therefore, the development and wide application of table tennis robots must first ensure effective table tennis trajectory prediction. Only in the premise of sufficient analysis of the characteristics of the ball, can the table tennis robot’s counterattack action be more timely and accurate [4].

Aiming at the problem of the lack of accuracy of the recognition, location and trajectory prediction of the rotating table tennis in the traditional method, based on the neural network method which has better adaptability and generalization to complex environment, this paper studies the real-time recognition, location and trajectory prediction of rotary flying table tennis based on the platform of table tennis humanoid robot system.

2 Related work

Professor John Billingsley of the United Kingdom first proposed the ping-pong robot competition in 1983. In 1985 and 1988, he held four ping-pong robot competitions. In the meantime, in order to encourage the accuracy of table tennis robots to hit the ball, they also developed the rules for table tennis robot competition [5]. The rule stipulates that the table tennis table is 2m×0.5m, which is smaller than the conventional table size, and the sports space for the table tennis robot to hit the ball is installed in three wire frames respectively on the two sections of the table and the ball net. These rules are difficult for the table tennis robot to hit the ball, and for the era when the visual system and the motion control are not developed at the time, the level of the table tennis robot can be fully utilized, and the high hitting rate isguaranteed.

In 1987, Russell L and Andersson of AT&T Bell Laboratories in the United States jointly developed a relatively perfect table tennis machine system [6]. The table tennis robot has three major components: a real-time vision system, a control system, and a mechanical system. Among them, the PUMA260 robot arm developed by Unimation of the United States was used to hang the arm on the mechanical support, and the link and the racket were fixed in the front of the arm, and the ball position information was obtained through four 756×242 pixel, 60 Hz sampling speed cameras.

In the mid-1980s, Toshiba Corporation of Japan developed a robotic system that hits the wall [7]. It uses a seven-degree-of-freedom humanoid manipulator and binocular vision system, and the image acquisition rate is 100 frames per second, which improves the accuracy of the hit. However, the degree of repetition of the system is relatively high, and the ping-pong track is basically the same.

In the late 1980s, the Swiss Federal Institute of Technology in Zurich designed a six-degree-of-freedom table tennis robot to participate in the table tennis robot competition [8]. The robot consists of a three-degree-of-freedom robotic arm and a three-degree-of-freedom mechanical wrist. It uses a binocular vision system with a pixel resolution of 579×422 and a sampling frequency of 50 Hz. Like the American Andersson, the robot uses a distributed computer network based on the MC6802 and MC68000 processors developed by Motorola and completes table tennis trajectory prediction and robot control. Moreover, the robot won the World Robot Table Tennis Championship twice.

In 1996, Jon Price, an engineer at Salt Lake City, USA, designed the first humanoid robot, SARCOS [9], which was installed on a wheeled moving mechanism to perform a batting action according to the operator’s instructions. The robot detects the opponent’s batting action with a sensor and controls the arm to return the ball through the computer. However, the robot is only a humanoid robot that can execute motion commands.

In 2002, Professor Miyazaki of Osaka University of Japan proposed a method of controlling table tennis robots. It can return the ball to the desired position in the specified time [10]. The robot is based on a binocular vision system and uses a high-speed digital camera with a resolution of 640×416 pixels. Moreover, the system has two translation joints and two rotation joints, a total of four degrees of freedom. In addition, it mainly adopts control strategies such as trajectory clustering analysis and local weighted regression analysis (LWR), and its learning control ability can accurately control the ping-pong drop point.

L. of Spain Acosta et al. applied a robotic arm with a double racket in 2003 and successfully developed a low-cost table tennis robot [11]. Each racquet of the robot has three degrees of freedom, using only a monocular camera, and uses the geometric relationship between the ball and the projection of the ball on the table to determine the spatial position information of the ping pong. The robot’s returning strength is small, and the hitting distance is limited, so the table is also much smaller than the average table. Moreover, in the low-speed man-machine-matching experiment, the success rate of returning the ball is only about 80%. In addition, the robot uses only one camera, avoiding visual time synchronization problems and saving costs, but due to experimental limitations, the requirements for scenes and lighting conditions are strict.

In December 2007, the new humanoid table tennis robot developed by TOSY of Vietnam [12] was unveiled at the International Robot Exhibition held in Tokyo, Japan. The robot has 39 “joints” for flexible rotation and it uses a binocular active vision system. It is loaded with two 200fps cameras that dynamically change the posture as the head moves and respond quickly by detecting the trajectory of the ball. Moreover, the robot’s expression is rich, if it fails to catch the ball, it will make a shaking head, and when it succeeds, it will nod. In addition, its “brain” is an artificial intelligence nervous system, and the best return route is selected by analyzing the ball trajectory.

In 1997, the research team of Professor Zhao Qing of Shanghai Jiaotong University cooperated with Professor Miyazaki Miyazaki of the Japanese University to propose a trajectory prediction method based on LWR (LocalWeightRegression) through the theoretical study of table tennis trajectory prediction and simulation algorithm [13]. In 2001, Yuan Jianchang and others from Northwest Textile Institute of Technology made in-depth theoretical research on the wrist structure of table tennis robots and analyzed the possibility of implementing robotic arms [14]. In addition, Zhang Peiyan and others from Shanghai Jiaotong University have developed volleyball robots based on industrial robots and real-time vision. The research team of Zhejiang University began researching table tennis robots in 2004. So far, they have developed three generations of table tennis robots [15]. The first generation of table tennis robots was introduced in 2006, using a monocular vision system. The system determines the three-dimensional coordinates of the table tennis based on the geometric relationship between the elements “monocular camera”, “light”, “table tennis” and “shadow”. The robot’s single-round return success rate is about 60%. It has laid a foundation for the research team’s further work because it has realized the basic motion control servo platform.

3 Theoretical analysis

In table tennis, in order to achieve a shot, the end must have a certain speed in the Cartesian space. Therefore, it is necessary to study a Jacobian matrix of six degrees of freedom to convert the hitting speed of the racket to the joint space.

3.1 Jacobian matrix solution

The relationship between space velocity and Cartesian space velocity can be expressed as [16]:

$\overset{\cdot}{x} = J (q) \overset{\cdot}{q}$ (1)

The element in column j of row i is

$J_{ij} (q) = \frac{\partial x_{i} (q)}{\partial q_{j}}, i = 1, 2, \dots 6; j = 1, 2, \dots, 6 .$ (2)

Among them, the first three lines indicate the transfer ratio of the end line speed v, and the last three lines indicate the transfer ratio of the end angular velocity w. Therefore, it can be expressed as [17]: $[\begin{matrix} v \\ w \end{matrix}] = J (q) \overset{\cdot}{q}$ (3)

The Jacobian matrix can be obtained by definition method, vector product method and integral transformation method. The definition method has a large amount of calculation, and a large number of derivation calculations are required, and the process of vector product and integral transformation is more complicated.

Joint variables can be defined as $q_{i} = ɛ_{i} d_{i} + \bar{ɛ_{i}} θ_{i}$ . Among them, d_i is the joint variable of the moving joint and θ_i is the joint variable of the rotating joint [18]. $ɛ_{i} = {\begin{matrix} 1, \\ 0, \end{matrix} \begin{matrix} Rotating joint of joint i \\ Joint joint of joint i \end{matrix}$ (4)

$\bar{ɛ_{i}} = 1 - ɛ_{i}$ (5)

As shown in Fig. 1, according to the rigid body kinematics, the motion of i is converted to the end effector n, then the end line velocity and angular velocity are

Fig.1

Schematic diagram of the solution of the Jacobian matrix.

${\begin{matrix} v = \sum_{i = 1}^{n} [ɛ_{i} V_{i} + \bar{ɛ_{i}} (Ω_{i} \times P_{in})] \\ w = \sum_{i = 1}^{n} \bar{ɛ_{i}} Ω \end{matrix}$ (6)

According to the definition of the D-H coordinate system and joint variables, the rotating joint has [19]: $Ω_{i} = Z_{i} q_{i}^{\cdot}$ (7)

The moving joints has: $V_{i} = Z_{i} q_{i}^{\cdot}$ (8)

After formula (7), (8) is substituted into formula (6), the following formula is obtained.

${\begin{matrix} v = \sum_{i = 1}^{n} [ɛ_{i} Z_{i} + \bar{ɛ_{i}} (Z_{i} \times P_{in})] \overset{\cdot}{q_{i}} \\ w = \sum_{i = 1}^{n} (\bar{ɛ_{i}} Z_{i}) q_{i}^{\cdot} \end{matrix}$ (8)

From $\overset{\cdot}{x} = (\begin{matrix} v \\ w \end{matrix}) = J \overset{\cdot}{q} = (\begin{matrix} J_{v} \\ J_{w} \end{matrix}) \overset{\cdot}{q}$ , it can be known that:

$J = (\begin{matrix} J_{v} \\ J_{w} \end{matrix}) = {\begin{matrix} \sum_{i = 1}^{n} [ɛ_{i} V_{i} + \bar{ɛ_{i}} (Z_{i} \times P_{in})] \\ \sum_{i = 1}^{n} (\bar{ɛ_{i}} Z_{i}) \end{matrix}$ (9)

Equation (9) is the formula for finding the Jacobian matrix by the vector product method. For the table tennis hitting process, the six joints are all rotating joints, that is, $ɛ_{i} = 0, i = 1, \dots 6, \bar{ɛ_{i}} = 1, i = 1, \dots, 6$ , so it can be written

$J = (\begin{matrix} J_{v} \\ J_{w} \end{matrix}) = {\begin{matrix} \sum_{i = 1}^{n} (Z_{i} \times P_{in}) \\ \sum_{i = 1}^{n} Z_{i} \end{matrix}$ (10)

This is the table tennis arm vector product expression.

However, in the solution process, the solution of Z_i × P_in is more difficult. Therefore, this paper combines the vector product and the definition method to obtain the Jacobian matrix.

$J = (\begin{matrix} J_{v} \\ J_{w} \end{matrix}) = {\begin{matrix} \frac{\partial x_{p}}{\partial q_{1}} \frac{\partial x_{p}}{\partial q_{2}} \dots \frac{\partial x_{p}}{\partial q_{n}} \\ \sum_{i = 1}^{n} Z_{i} \end{matrix}$ (11)

3.2 Singularity analysis

The Jacobian matrix depends on the shape q. When a particular shape makes det(J (q)) = 0, the Jacobian matrix loses its ability to convert the Cartesian velocity to the joint velocity. This shape is called the singular point of the Jacobian matrix and should be removed during use.

According to the above analysis, the determinant of the Jacobian matrix is found, and whether the determinant is zero can be used as the criterion for the singularity of the Jacobian matrix [20].

It can be seen from the calculation that the determinant of the Jacobian matrix is as follows

$\begin{matrix} det (J (q)) = - a_{2} d_{4} cos (q_{3}) sin (q_{5}) \\ [\begin{matrix} d_{4} cos (q_{2}) sin (q_{3}) - a_{2} cos (q_{2}) \\ + d_{4} cos (q_{3}) sin (q_{2}) \end{matrix}] \end{matrix}$ (12)

Thanks to the table tennis hitting point, there are many options. When calculating a certain shape, it is judged whether the Jacobian matrix is singular. If it is singular, the batting point is abandoned, thereby avoiding singularity [21].

4 Improved recognition algorithm

A recurrent neural network with an LSTM unit is a special neural network structure. It can process time series data by recording and processing the input of the first N frames (N21) and the network intermediate results, thereby integrating the multi-frame information to complete tasks such as classification, regression, and prediction of the input. The LSTMfN network has the same basic form as other forms of networks, and is composed of an input layer, a hidden layer, and an output layer. The difference is that the hidden layer not only accepts the current frame input, but also accepts the previous frame output and the network intermediate result as input. The basic structure of the network is shown in Fig. 2. The figure shows a schematic diagram of a LSTM network with a hidden layer for five lengths of time [22].

Fig.2

Schematic diagram of recurrent neural network.

The LSTM unit is in fact a special design form of the recursive neural network hidden layer node, and a hidden layer can contain multiple LSTM units. The internal detailed structure of the unit is shown in Fig. 3. Each unit consists of a core node, an input gate, a forgetting gate and an outputgate [23].

Fig.3

Schematic diagram of LSTM unit structure.

Assuming that the current input value is x^t, the output of the Cell node in the same LSTM unit at the previous time is $s_{c}^{t - 1}$ , and the output of other nodes (such as input gates, etc.) in the same LSTM unit at the previous time is $b_{h}^{t - 1}$ . Then, the output $b_{l}^{t}$ of the input gate in the unit at the current time is obtained by the weighted summation of current input, the Cell output at the previous moment and the of the other nodes at the previous moment and nonlinearactivation [24]: $\begin{matrix} a_{l}^{t} = \sum_{i = 1}^{I} w_{il} x_{i}^{t} + \sum_{c = 1}^{C} w_{cl} s_{c}^{t - 1} + \sum_{c = 1}^{C} w_{hl} b_{h}^{t - 1} \\ b_{l}^{t} = f (a_{l}^{t}) \end{matrix}$ (13)

Among them, w_il is a weight parameter of the input term to the input gate; w_cl is the weight parameter of the Cell output item to the input gate at the previous moment; w_hl is the weight parameter of the output of other nodes at the previous moment; f is a simple nonlinear activation function. If there is no specific explanation, the nonlinear activation functions used in this paper are all Sigmoid functions.

Similar to the input gate, the output $b_{φ}^{t}$ of the forgetting gate in the current time unit is also composed of three parts: x^t, $s_{c}^{t - 1}$ , and $b_{h}^{t - 1}$ . It is obtained by weighted summation and nonlinear activation of the Sigmoid function [25]: $\begin{matrix} a_{φ}^{t} = \sum_{l = 1}^{I} w_{l φ} x_{i}^{t} + \sum_{c = 1}^{C} w_{c φ} s_{c}^{t - 1} + \sum_{c = 1}^{C} w_{h φ} b_{h}^{t - 1} \\ b_{φ}^{t} = f (a_{φ}^{t}) \end{matrix}$ (14)

The output s of the Cell node in the current time unit is determined by the input of the current time, the output of the current time input gate, the output of the current moment forgetting gate, the output of the Cell at the previous moment, and the output of other nodes at the previous moment: $\begin{matrix} a_{C}^{t} = \sum_{l = 1}^{I} w_{lC} x_{i}^{t} + \sum_{c = 1}^{C} w_{hc} b_{h}^{t - 1} \\ s_{c}^{t} = b_{φ}^{t} s_{c}^{t - 1} + b_{l}^{t} g (a_{c}^{t}) \end{matrix}$ (15)

Among them, g is also a Sigmoid function. The output $b_{w}^{t}$ of the output gate is obtained by the weighted summation of the input of the current moment and the output of the Cell at the current time and the nonlinear activation of the Sigmoid function: $\begin{matrix} a_{w}^{t} = \sum_{l = 1}^{I} w_{lw} x_{i}^{t} + \sum_{c = 1}^{C} w_{cw} s_{c}^{t - 1} + \sum_{c = 1}^{C} w_{hw} b_{h}^{t - 1} \\ b_{φ w}^{t} = f (a_{w}^{t}) \end{matrix}$ (16)

Finally, the total output $b_{c}^{t}$ of the LSTM unit is: $b_{c}^{t} = b_{w}^{t} h (s_{c}^{t})$ (17)

Among them, h is the Sigmoid function. Based on the above analysis, the output of the unit can be obtained. $b_{c}^{t} = \begin{matrix} f (\sum_{i = 1}^{I} w_{il} x_{i}^{t} + \sum_{c = 1}^{C} w_{cl} s_{c}^{t - 1} + \sum_{c = 1}^{C} w_{hl} b_{h}^{t - 1}) \\ h (\sum_{l = 1}^{I} w_{l φ} x_{i}^{t} + \sum_{c = 1}^{C} w_{c φ} s_{c}^{t - 1} + \sum_{c = 1}^{C} w_{h φ} b_{h}^{t - 1}) \\ + f (\sum_{i = 1}^{I} w_{il} x_{i}^{t} + \sum_{c = 1}^{C} w_{cl} s_{c}^{t - 1} + \sum_{c = 1}^{C} w_{hl} b_{h}^{t - 1}) \\ g \sum_{l = 1}^{I} w_{lC} x_{i}^{t} + \sum_{c = 1}^{C} w_{hc} b_{h}^{t - 1} \end{matrix}$ (18)

As can be seen from the above equation, the output of a single LSTM unit at time t is related to the network input at time t and the internal node output of the unit at time £— 1. Moreover, the output of these internal nodes is the selection result of all input-passed weight parameters and nonlinear activation functions at [0, 1,..., t– 1] moments. The operating mechanism enables the RNN network with the LSTM unit to predict the future trajectory of the table tennis based on past trajectory data.

In addition to the error propagation between the layers as well as other forms of neural networks, the back-propagation algorithm for recurrent neural network applications also requires error propagation over time, i.e., back propagation through time. There are a large number of weight parameters to be trained in the RNN network with LSTM units. Among them, a large part is the weight parameter of the network output at time t to the network output at time t-1. According to the gradient descent method and the chain rule, this paper deduces the propagation of the trajectory prediction network timing error.

Assuming that the output of the network at time t is N-dimensional vector O_N and the input label is y_n, the error between the predicted value and the label value is calculated by EuclideanLoss: $L (o, y) = \frac{1}{2 N} \sum_{n = 1}^{N} {∥ o_{n} - y_{n} ∥}_{2}^{2}$ (19)

It is also known that in the time t, O_N is obtained by the output $b_{c}^{t}$ of the LSTM unit through a layer of fully connected layers, and the relationship between the two is: $O_{N} = \sum_{i = 1}^{N} w_{on} b_{c}^{t}$ (20)

According to the chain law, the gradient $ξ_{c}^{t}$ of the error between the predicted value and the tag value to the output $b_{c}^{t}$ of the LSTM unit is: $ξ_{c}^{t} = \frac{\partial L (o, y)}{\partial b_{c}^{t}} = \frac{\partial L (o, y)}{\partial O_{N}} \frac{\partial O_{N}}{\partial b_{c}^{t}} = (O_{N} - y_{N}) w_{on}$ (21)

The chain rule continues to be used, and the gradient $δ_{w}^{t}$ of the error to internal output gate of the LSTM unit is. $\frac{\partial L (o, y)}{\partial a_{m}^{t}} = \frac{\partial L (o, y)}{\partial b_{c}^{t}} \frac{\partial b_{c}^{t}}{\partial b_{w}^{t}} \frac{\partial b_{w}^{t}}{\partial a_{w}^{t}} = ξ_{c}^{t} h (s_{c}^{t}) f^{'}$ (21)

It can be seen from Equation (4.6) that as the core node of the LSTM unit, the Cell node has an influence on the unit output at time t, the Cell output at time t + 1, the input gate at time t + 1, and the output gate at time t + 1 in the network forward process. Therefore, when performing inverse gradient calculation, the error of reverse transmission from these four directions should also be considered. $\begin{matrix} ξ_{s}^{t} = \frac{\partial L (o, y)}{\partial s_{c}^{t}} = ξ_{c}^{t} b_{w}^{t} h^{'} (s_{c}^{t}) + b_{φ}^{t + 1} ξ_{c}^{t + 1} \\ + w_{c φ} δ_{φ}^{t + 1} + δ_{w}^{t} w_{cw} \end{matrix}$ (22)

It can be seen from the equation that for the error conduction of the Cell node, the internal calculation result of the node at the next moment is needed. Therefore, when training a recurrent neural network with an LSTM unit, it is usually extended according to a certain length of time, and the network parameters in one sequence are simultaneously updated.

Using the chain rule, the gradient $δ_{φ}^{t}$ of the forgotten gate is: $δ_{φ}^{t} = \frac{\partial L (o, y)}{\partial a_{φ}^{t}} = \frac{\partial L (o, y)}{\partial b_{φ}^{t}} \frac{\partial b_{φ}^{t}}{\partial a_{c φ}^{t}} = ξ_{c}^{t} s_{c}^{t - 1} f^{'}$ (23)

At this point, the error between the predicted value and the tag value is calculated for the conduction gradient of each node inside the LSTM unit. Then, according to the gradient descent method, the weights of each node are updated by applying these gradient values.

5 Algorithm verification

In this paper, 1800 tracks are randomly selected to form the training data set, and another 200 tracks constitute the test data set. After that, the above network is trained and tested. In order to better compare the impact of network expansion accuracy according to different step sizes and select the step size that is most suitable for trajectory prediction tasks, this paper chooses three different step sizes to train the network. Figures 4 and 5 show the convergence process of training error and test error during the training of 50,000 times of three networks with expansion steps of 5, 10, and 15, respectively. The error (Loss) is measured in m, which represents the average spatial distance between the predicted position P (x_p, y_p, z_p) and the true position P_r (x_r, y_r, z_r) in the test of 200 times. The calculation is as follows:

Fig.4

Training error image.

Fig.5

Test error image.

$\begin{matrix} ξ = \sqrt{{(x_{p} - x_{r})}^{2} + {(y_{p} - y_{r})}^{2} + {(z_{p} - z_{r})}^{2}} \\ \bar{ξ} = \frac{1}{200} \sum_{i = 1}^{200} ξ_{i} \end{matrix}$ (24)

Table 4 and Fig. 5 show the training error and test error values after the network training in the above three step sizes. It can be seen from the table that after sufficient training, the network of three steps can converge to the training and test accuracy of 5mm and below, and the test error is slightly larger than the training error. Among them, the network training accuracy and test accuracy of the expansion step size of 5 are better than the other two networks, and the accuracy of the other two networks decreases with the increase of the step size. After the completion of the training, the prediction error of the test data set of the three-step trajectory prediction network is reduced to less than 5 mm. However, since the method predicts the long-term trajectory by means of iterative calculation, that is, the current time network output is used as the next time network input for prediction, the cumulative error of the network prediction causes the overall prediction error to increase as the number of predicted frames increases. As mentioned in the overview, table tennis robots generally need to complete the prediction of the future 400ms flight trajectory within 100ms. However, the hardware platform visual system used in this paper has a frame rate of 120 fps, which means that the network needs to predict the next 40 frames based on about 10 frames of data.

In order to measure the impact of cumulative error on the overall error, in this paper, the first 10 frames of data are input to the trained network to predict the spatial position of the 14th, 19th, 24th, 29th, 34th, and 39th table tennis balls in the future. Moreover, the prediction error is counted as shown in Table 2 and Fig. 6.

Fig.6

Schematic diagram of network prediction error with predicted frame number.

Fig.7

Time-consuming statistics diagram of network prediction.

Fig.8

Error statistics diagram of collision points.

Table 1

Training error and test error

	Training error (×10^–3m)	Test error (×10³m)
Step size 5	2.749	3.628
Step size 10	3.149	3.877
Step size 15	3.769	4.250

Table 2

Error table of long-term forecast

Predicted frame number	Step size 5	Step size 10	Step size 15
14	4.029	4.286	4.505
19	4.578	4.817	4.839
24	5.317	5.437	5.348
29	6.257	6.282	6.054
34	7.356	7.293	6.715
39	8.289	8.140	7.406

For the table tennis robot vision system, in addition to predicting accuracy, predicting real-time performance is also an important requirement. This paper also uses the test data set to calculate the average time taken by the above three networks to predict the future trajectory of table tennis, and the test hardware platform is the same as the training hardware platform. Table 3 shows the average time used by the three networks to predict future frames 1, 19, and 39. It should be noted that the statistics in the table do not count the time for pre-preparation work such as network initialization, parameter loading, and generation of calculation graphs. It can be seen from the table that the recursive-based neural network structure proposed in this paper is more complicated. Therefore, its calculation speed is slow, and the operation time increases as the number of predicted frames increases.

Table 3

Time-consuming statistics table of network prediction

Number of frames	1	19	39
Time-consuming of 5 step size (ms)	0.54	11.77	24.61
Time-consuming of 10 step size (ms)	0.89	18.4	38.67
Time-consuming of 15 step size (ms)	1.32	28.57	54.28

During the ping pong flight, when the ball table collides, the ball’s centroid height z is equal to the ball’s radius r. At this time, the error of the collision point only needs to consider the two dimensions of x and y. The prediction at the collision point can more typically reflect the accuracy and adaptability of the prediction method. For the same test data set, the statistical results of the collision point error of the proposed method and the comparison method are shown in Table 4. The MSE in the table represents the mean square error between the predicted value and the true value; RMSE represents the root mean square error between the predicted value and the true value; Bias represents the average offset between the predicted value and the true value; SD represents the variance between the predicted value and the predicted average value. Among the above indicators, MSE and RMSE reflect the accuracy of the prediction. The smaller the two values, the higher the accuracy of the prediction. Moreover, Bias reflects the extent to which the predicted value deviates from the true value as a whole. The smaller the value, the closer the predicted value is to the true value. SD reflects the degree of dispersion of the predicted value. The smaller the value, the larger the predicted value is, and the probability that the predicted value deviates significantly from the true value is smaller, and the adaptability of the prediction method is better.

Table 4

Error statistics table of collision points

	Algorithm of this paper		Comparison algorithm
	x	y	x’	y’
MSE(*10^- 4M²)	0.168	0.183	1.315	2.026
RMSE(*10^- 2M²)	0.416	0.435	1.164	1.444
Bias(*10^- 2M²)	0.292	0.307	0.840	0.861
SD(*10^- 2M²)	0.891	0.931	1.216	2.959

It can be seen from the above figures and tables that the prediction value obtained by the method based recurrent neural network proposed in this paper have smaller deviation values and the data aggregation degree is better. Moreover, its prediction accuracy and adaptability in the x and y dimensions are greatly improved compared to the prediction method based on the continuous motion model. Figure 9 is a more visual representation of the error distribution of predicted drop points on the test data set for the method of this paper and comparison method. In the figure, each point represents the error between the predicted value and the true value in one test. Among them, the green point is the prediction result of the research algorithm, and the red point is the comparison algorithm prediction result.

Fig.9

Distribution image of collision point error.

6 Results analysis and discussion

The traditional physical model and trajectory prediction method for motion modeling by force analysis based tennis ball are often not well adapted to table tennis in a variety of rotating states, and the accuracy is limited by the model. This paper does not rely on prior knowledge and proposes a method of table tennis trajectory prediction based on recurrent neural network. Moreover, by constructing a large number of nonlinear LSTM elements to approximate the high-order nonlinear table tennis motion process, the long-term trajectory prediction of flying table tennis in various rotating states is realized. The experimental results prove the prediction accuracy of the proposed method and meet the real-time requirements of the table tennis robot’s visual perception system.

Table tennis is regarded as a spring model, and the collision process between table tennis and the table is theoretically analyzed and mathematically derived. The study found that the collision between table tennis and the table is a continuous gradual process. The support for table tennis is a sinusoidal function, the maximum amplitude is linearly related to the incident velocity before the table tennis collision, and the correlation coefficient depends on the physical properties of the table tennis. At the same time, the collision period between table tennis and the table is a constant that depends only on the physical properties of the table tennis and is not related to the state of motion of the table tennis before the collision. Then, we use the ultra-high-speed camera design experiment to observe the collision process of table tennis and the table with different incident speeds, and further analyze and verify the relationship between the collision period with the maximum amplitude of the support force and the motion state according to the observation results.

In the research of table tennis trajectory prediction, this paper builds a single-input, single-output recurrent neural network, which allows the network to predict only the space position of the next frame. This structure streamlines the network size and improves real-time prediction accuracy. However, it makes the prediction of long-term trajectory need to adopt the method of iterative calculation, and makes the error accumulate in the iterative process, which affects the long-term observation accuracy to a certain extent, and also increases the computation time of the network long-term prediction. In the future research, we can try to build a multi-input, multi-output recurrent neural network, let the network directly learn to predict long-term trajectories, thus saving computation time and further improving predictionaccuracy.

The neural network based ping-pong positioning and trajectory prediction method proposed in this paper does not depend on any prior model. Moreover, the built-in identification positioning network and the internal parameters of the trajectory prediction network are all trained by the data set and belong to the non-model open framework method. In addition, these two networks are built using the open source platform toolkit, which is highly modular and easy to change the hierarchical structure. Therefore, it is only necessary to use the corresponding data set with rich object features for retraining, and then supplement with the necessary hierarchical structure adjustment. The method proposed in this paper can be applied to other similar fields, such as the identification and trajectory prediction of other balls such as tennis.

The overall perception of the environment is the main task of the table tennis robot visual servo system, and also the basis for the robot to play table tennis. The work of this paper solves the robot’s perception of rotating table tennis, including accurate recognition, precise positioning and long-term trajectory prediction. However, table tennis is an interactive movement, and the robot needs to have the ability to perceive the opponent, including opponent motion recognition, intention recognition, target prediction and so on. This work has not been carried out in this paper and further research is needed.

7 Conclusion

Based on the neural network method which has better adaptability and generalization to complex environment, this paper studies the real-time recognition, location and trajectory prediction of rotary flying table tennis based on the platform of table tennis humanoid robot system. Moreover, in table tennis, in order to achieve a shot, the end must have a certain speed in the Cartesian space. Therefore, it is necessary to study a Jacobian matrix of six degrees of freedom to convert the hitting speed of the racket to the joint space. In this paper, 1800 tracks are randomly selected to form the training data set, and another 200 tracks constitute the test data set to train and test the above network. At the same time, in order to better compare the influence of different step sizes on the accuracy of network convergence to select the step size that is most suitable for trajectory prediction tasks, this paper chooses three different step sizes to train the network. The research shows that the recursive neural network method proposed in this paper has a smaller deviation from the true value and better data aggregation. Moreover, the prediction accuracy and adaptability of the method in the x and y dimensions are greatly improved compared with the prediction method based on the continuous motion model.

Footnotes

Acknowledgment

This work was supported by Topics of Guangdong Sports Bureau: analysis of the technical and tactical characteristics of Guangdong table tennis player Liu Shiwen’s main opponents in the Rio Olympic Games. Number: GDSS 2016133.

References

Unterthiner

, Mayr

, Klambauer , Günter , et al., Toxicity Prediction using Deep Learning Computer Science 3(8), 2015.

Spencer

, Eickholt

, Cheng

A Deep Learning Network Approach to ab initio Protein Secondary Structure Prediction IEEE/ACM transactions on computational biology and bioinformatics / IEEE, ACM (2014), 1.

, Yu

, Wang

, et al., Large-scale transportation network congestion evolution prediction using deep learning theory, PLOS ONE (2015), 10.

Sheng

, Siqi

, Zhen

, et al., Accurate de novo prediction of protein contact map by ultra-deep learning model, PLOS Computational Biology 13(1) (2017), e1005324.

Angermueller

, Lee

H.J.

, Reik

, et al., DeepCpG: Accurate prediction of single-cell DNA methylation states using deep learning, Genome Biology 18(1) (2017), 67.

Polson

N.G.

, Sokolov

V.O.

Deep learning for short-term traffic flow prediction, Transportation Research Part C: Emerging Technologies 79 (2017), 1–17.

Cheng

, Guo

, Wang

, et al., MiRTDL: A deep learning approach for miRNA target prediction, IEEE/ACM Transactions on Computational Biology & Bioinformatics 13(6) (2016), 1161–1169.

Sun

, Zhou

, Lai

, et al., Sequence-based prediction of protein protein interaction using a deep-learning algorithm, BMC Bioinformatics 18(1) (2017), 277.

Huang

, Li

, Zhang

, et al., Deep learning driven visual path prediction from a single image, IEEE Transactions on Image Processing (2016), 1–1.

10.

Wen

, Zhang

, Niu

, et al., Deep-learning-based drug-target interaction prediction, Journal of Proteome Research 16(4) (2017), 1401.

11.

Fan

, Xiao

, Zhao

A short-term building cooling load prediction method using deep learning algorithms, Applied Energy 195 (2017), 222–233.

12.

, Wang

, Umarov

, et al., DEEPre: Sequence-based enzyme EC number prediction by deep learning, Bioinformatics, 2017.

13.

, Wang

, Yu

, et al., Deep Learning based Radiomics (DLR) and its usage in noninvasive IDH1 prediction for low grade glioma, Scientific Reports 7(1) (2017), 5467.

14.

Hosseini

M.P.

, Pompili

, Elisevich

, et al., Optimized deep learning for eeg big data and seizure prediction BCI via internet of things, IEEE Transactions on Big Data (2017), 1–1.

15.

Chong

, Han

, Park

F.C.

, Deep learning networks for stock market analysis and prediction: Methodology, data representations, and case studies, Expert Systems with Applications 83 (2017), 187–205.

16.

Yifeng

, Wenqiang

, Wasserman

W.W.

, Genome-wide prediction of cis-regulatory regions using supervised deep learning methods, BMC Bioinformatics 19(1) (2018), 202.

17.

Hamanaka

, Taneishi

, Iwata

, et al. CGBVS-DNN: Prediction of compound-protein interactions, Based on Deep Learning Molecular Informatics 36(1-2), 2017.

18.

, Wang

, Song

, et al., Deep air learning: Interpolation, prediction, and feature analysis of fine-grained air quality, IEEE Transactions on Knowledge and Data Engineering 30(12) (2018), 2285–2297.

19.

Kim

H.K.

, Min

, Song

, et al., Deep learning improves prediction of CRISPR– Cpf1 guide RNA activity, Nature Biotechnology (2018).

20.

Nie

, Jiang

, Guo

, et al., Traffic matrix prediction and estimation based on deep learning in large-scale IP backbone networks, Journal of Network & Computer Applications 76(C) (2016), 16–22.

21.

Kiral-Kornek

, Roy

, Nurse

, et al., Epileptic seizure prediction using big data and deep learning: Toward a mobile system, EBioMedicine 27(C) (2017), 103–111.

22.

Lao

, Chen

, Li

Z.C.

, et al., A Deep Learning-Based Radiomics Model for Prediction of Survival in Glioblastoma Multiforme Scientific Reports, 7(1) (2017), 10353.

23.

Stahl

, Schneider

, Brock

, EPSILON-CP: Using deep learning to combine information from multiple sources for protein contact prediction, BMC Bioinformatics 18(1), 2017.

24.

Das

, Ghosh

S.K.

, Deep-STEP: A deep learning approach for spatiotemporal prediction of remote sensing data, IEEE Geoscience & Remote Sensing Letters 13(12) (2016), 1984–1988.

25.

Junxing

, Deep learning and its application in deep gas reservoir prediction, Computing Techniques for Geophysical and Geochemical Exploration 39(6) (2017), 775–782.