A robot demonstration method based on LWR and Q-learning algorithm

Abstract

A robot demonstration method is proposed based on the combination of locally weighted regression(LWR) and Q-learning algorithm. It is applied on a 6-DOF hitting-ball-system. This method can adapt to the work task by learning from demonstration and generating new actions. With the LWR algorithm, the mapping between target values and actions is established. According to deviation of landing position, a Q-learning algorithm is proposed to adjust the parameters of manipulator and compensate the errors caused by model and the controller. The model of LWR fits a local small space to approximate the global state and decision space. It turns out to reduce the dimension and simplify the training of Q-learning. The convergence rate is enhanced and the precision of performing task is improved. The simulation and experiment demonstrate the applicability of the proposed method.

Keywords

Reinforcement learning Q-learning locally weighted regression program by demonstration

1 Introduction

In human life and production, the robot application plays a more and more important role. People put a higher request for robot to adapt to the complex and changing environment. Robots need to be able to deal with problems from the environment and human interaction, and improve their adaptability of executing tasks. Imitation learning is a promising mechanism to learn new knowledge, which is also known as the program by demonstration(PbD). This method can reduce the complexity of the learning search space by observing the quality of the samples, which turns out to have great learning efficiency [1]. It is also an important research direction in the field of robotics [2 –6].

The general learning form of imitating learning is action replication. Action replication is based on the action of the demonstrator, by solving the regression problem from the demonstration to the executing actions to determine the control strategy, and imitators implement control strategies for action reproduction to achieve imitation learning. The traditional demonstration process divides the perceptual information into discrete sub-targets and point-to-point motor operations so that the demonstration task is decomposed into a series of state-action-state transformation [7].

The demonstration actions belong to the discrete state space. In order to improve the adaptability of the robot to the environment, PbD uses more and more machine learning methods such as Gaussian process method to solve the generalization problem of action learning [8, 9]. Gaussian process method(GPM) is a nuclear learning machine with probabilistic meaning, which can give a probabilistic interpretation of the predicted output. It is based on the assumption that the observations and predictions are subject to a joint normal distribution, then the posterior distribution of the predictions would be obtained by solving the covariance matrix of the observations and the input of the training set. GPM has been applied to the regression and classification problems successfully [10, 11]. However, there are obvious drawbacks in the GPM. Due to the complexity of computation is large, the computational efficiency and real-time performance is poor, especially when processing large amounts of data. Besides, GPM is restricted to the strict assumption of Gaussian noise distribution, which means that when the actual model does not follow the Gaussian distribution, GPM performs poorly.

The locally weighted linear regression(LWR) learning method is a kind of data learning algorithm which is widely used in the robot adaptive control system and performs well on dealing with complex control problems. LWR is a memory-based learning method. Its basic idea is to construct a local model based on the stored empirical data points, which are weighted by a distance function. The weighted distance function ensures that the points in the empirical data points that are near to the input point have a greater effect on the local model. LWR fits well in the training region. In addition, it hardly depends on the choice of features and is able to train a good fitting model only with a simple linear model [12 –14].

However, considering the errors and uncertainties of the system and the model, it is not enough to predict the parameters of the robot by the simple prediction model. We need to add other learning algorithms to increase the robustness of the model. Some study focus on some self-learning method, such as strategy gradient search and inverse reinforcement learning(IRL) [15 –17]. Inverse reinforcement learning method is mainly to obtain the optimal cost function when the imitator imitates to learn the demonstration behavior and search for control strategy, which minimums the cost function. But, both policy gradient search and IRL need to collect plenty of training data which consumes a great deal of time and labor. In the process of robot demonstration programming, Guenter used reinforcement learning find the optimal parameters of a Gaussian mixture model(GMM) which indicated the advantage of the reinforcement learning that it can handle a lot of different and unexpected situations [18]. Reinforcement learning is to solve problems that an autonomous agent which is capable of observing the state of its environment and taking a series of actions to change those states, and the agent can achieve their goals by learning to choose the optimal action. With the deep research on the reinforcement learning method, it is more and more mature in dealing with the interaction between the robot and the environment and the motion prediction problems [19 –23].

In this paper, in order to demonstrate the manipulator to perform the hitting ball task, the system are required to generalize new actions according to the observed new target position(environment). A simple non-linear predict model may not satisfy the accuracy since the errors and uncertainty of model and controller. Reinforcement learning has advantage of dealing with uncertain and unexpected situations. However, due to the huge sample space, the training time of reinforcement learning is too long and the convergence rate is slow. It seems difficult to apply the reinforcement learning method directly to the real robot system. We propose a control method combining the LWR and reinforcement learning, which has a good performance on precision and require less time for the training process. In the next section, related work in the action learning and regenerating is summarized.

2 Related work

Traditionally, machine learning(ML) technique is used to solve the problem of generating new actions. Muench proposed the ML techniques to generalize demonstrate actions and design an adaptive controller to regenerate the task according to the new environment [8]. However, the learning system was simple and could not perform the execution autonomously. Based on the mixture of motor primitives, in the table tennis robot experiment, the Gaussian process regression was used to establish the mapping between the parameters of robot and the original characteristic of the motor. A probabilistic model was given to predict the state parameters of the robot [9]. Since the drawbacks of GPM, the performance of the model is poor in some cases.

In recent year, LWR learning method is widely used in the robot adaptive control system and performs well on dealing with complex control problems. In [12 –14], LWR learning method was used to establish the mapping between environment and robot parameters. In batting game experiments, Matsushima established the mapping between the robot parameters and motion of the ball, and proposed a LWR-based method to generalize new actions and then to compensate the error of the servo system by using the Iterative learning control method. In [13], Schaal improved the traditional LWR, which allows it to perform well on high-dimension space, and the feasibility of the method is verified by a 7-DOF manipulator. On the platform of table tennis robot, Huang firstly used LWR as a “lazy learning", and then added the FCMAC to the LWR algorithm as an “active learning”, which can update the experience database online, and fine-tuning the parameters of the manipulator to improve the placement accuracy of the ball [14].

However, considering the errors and uncertainties of the system and the model, more complex learning methods are studied in some research. In [15], Peters discussed various methods of strategy gradient search. These methods have the ability of self-learning, which is proved to have good performance after being applied to demonstration of baseball batting experiments. In [16, 17], an IRL method was proposed to search control strategies for robot demonstration learning. Aiming at physical human-robot collaboration problem, Ghadirzadeh proposed a reinforcement learning framework based on data-efficient, robot learns tasks from its own motion sensor in unsupervised way [21]. In the grasp task based on the tactile information, Chebotar divided the problem into two steps. In the first step, a stable grasping predictor was trained according to the tactile information and grasping effect. In the second step, a reinforcement learning model was trained according to the tactile feedback information and the forecasting result of the predictive machine. The action of the second grasping was adjusted so that the success rate of grasping object is greatly improved [22]. In [23], Wang used the Q-Learning reinforcement learning method to deal with the humanoid robot fixed-point-multi-direction soccer ball kicking problem. The motion curves of the robot’s foot played as the action set while the discretized football falling regions were considered as the state set. The kicking task was regarded as a simple reinforcement learning that “Action acquisition status and directly access to the final reward value”. However, these methods require plenty of training data which consumes a great deal of time and labor. As a result, it is difficult to use them on the real robot directly.

Inspired by the approach in [22], we divide the problem into two steps. Firstly, the mapping between the two sets is established by using LWR learning method. Then, a Q-learning reinforcement learning algorithm is proposed which is added to the LWR, aiming to adjust the parameters of manipulator and compensate the error caused by model and the controller. In addition, this method also reduces the dimension of Q-learning sample space and decision space, which results to greatly reduce the Q-learning training time. The two algorithms complement each other which perform well on the demonstration of robot hitting ball experiment. Finally, the feasibility of the control method is verified by both simulation and experiment.

3 System description and modeling

As shown in Fig. 1, the hitting ball robot consists of experimental platform, a 6DOF UR manipulator with a bat and an industrial camera which is fixed above the platform. The optical axis of the camera is perpendicular to the experimental platform, and the ball is located at the fixed position on a hitting platform. The coordinate systems are also illustrated. {B} is the manipulator base coordinate system and {T} is the tool center position(TCP) coordinate system.

Fig.1

Hitting ball system.

During the experiment, in order to simplify the model and reduce the variables, the parameters of robot are selected to use the pose parameters and speed of the end-effector H =(P, R, v). P =(x, y, z) is the position of the hitting ball relative to {B} the manipulator base coordinate system. R =(R_x, R_y, R_z) is the equivalent rotation vector, which is predefined by the manipulator system; v =(v_x, v_y, v_z) is the speed of TCP when the racket hits the ball.

The equivalent rotation vector R is derived as follows:

Construct the equivalent rotation matrix: $R_{k} (θ) = [\begin{matrix} r_{11} & r_{12} & r_{13} \\ r_{21} & r_{22} & r_{23} \\ r_{31} & r_{32} & r_{33} \end{matrix}]$ (1)

Equivalent rotation angle: $θ = Acos (\frac{r_{11} + r_{22} + r_{33} - 1}{2})$ (2)

Equivalent rotation axis vector: $K = \frac{1}{2 sin θ} [\begin{matrix} r_{32} - r_{23} \\ r_{13} - r_{31} \\ r_{21} - r_{12} \end{matrix}]$ (3)

Predefined pose parameters: $R = θ K$ (4)

{V} is the pixel plane coordinate system. The information collected by the industrial camera is shown in Fig. 2. In this experiment, the camera is not calibrated, and the pixel coordinate information is collected as the target observation set directly.

Fig.2

Information collected from camera.

4 Robot demonstration method based on the LWR and Q-learning reinforcement learning algorithm

4.1 LWR learning algorithm

LWR is a memory-based learning method. Its basic idea is to construct a local model based on the stored empirical data points, which are weighted by a distance function. Then a polynomial model is selected and the sample points are fitted by least square method(LSM). The LWR method combines the locally polynomial fitting and locally weighted to approximate the local function model near the prediction points according to the position of them, and has strong robustness.

4.2 Learning model parameters

From the second part, it is known that the action parameters of the manipulatorare selected as the pose parameters, position parameters and the velocity of the TCP H =(P, R, v). In this experiment, in order to simplify the model and reduce the parameter variables further, the position of hitting ball is fixed so that P turns out to be a constant. The pose of the bat is shown in Fig. 3. Z_T axis and Z_B axis are parallel and reverse to each other. From the formulas above, R_Z would always be zero and(R_x, R_y) would be the variable. In addition, the hitting way is chosen as pushing along a straight line, which means the bat would move in a straight line and the hitting speed is always perpendicular to a plane Y_TO_TZ_T, so that the velocity vector v turns out to be a scalar.

Fig.3

Coordinate system of TCP.

Furthermore, the position of the ball is expressed by the coordinate of the center of it which is collected by an industrial camera, and several more notations are introduced: $D_{d} = (x_{vd}, y_{vd})$ (5) $D_{a} = (x_{va}, y_{va})$ (6) $D_{Δ} = (d_{xv}, d_{yv}) = (x_{vd} - x_{va}, y_{vd} - y_{va})$ (7)

Where D_d and D_a denote the desired landing position and actual landing position, respectively, and D_v denotes the deviation of the desired and actual landing position.

4.2.1 Input-output maps

The landing position set D_a and the corresponding manipulator parameter set H =(R_x, R_y, v) are built by demonstrating the manipulator to perform the hitting ball task. Then, the map that represents the relationship between D_a and H is constructed by using LWR method andthe parameters of the model are obtained by the multivariate least squares method(MLSM). The process is as follows:

Training sets obtained by demonstration:

$\begin{matrix} {(D_{a}^{(i)}, H^{(i)}), i = 1, \dots, m}, \\ D_{a}^{(i)}, \in R^{2}, H^{(i)} \in R^{3} \end{matrix}$ (8)

The relationship between input and output: $H = D_{a} Θ$ (9)

Where Θ is the matrix of parameters, and Θ ∈ R^2×3.

Use MLSM to compute the parameters in LWR model. The cost function is:

$J (Θ) = \frac{1}{2} \sum_{i = 1}^{m} \sum_{j = 1}^{3} W_{ii} {({(Θ^{T} D_{a}^{(i)})}_{j} - H_{j}^{(i)})}^{2}$ (10)

Where W is the matrix of distance weights and denoted as following: $W = \exp (- \frac{1}{2 τ^{2}} {(D_{a} - D_{aq})}^{T} D (D_{a} - D_{aq}))$ (11)

Where D_aq ∈ R² is the input, and D ∈ R^m×n is an identity matrix. $W_{ii} = \exp (- \frac{1}{2 τ^{2}} {(D_{a}^{(i)} - D_{aq})}^{T} D (D_{a}^{(i)} - D_{aq}))$ (12)

Where τ is the bandwidth parameter which is used to control the rate of weights declining with distance. When τ is large, most of the datas is used for the training of the regression model, while τ is small, only a few local points are used to train the regression model.

Finally, the obtained parameters are expressed as: $Θ = {(X^{T} WX)}^{- 1} X^{T} WY$ (13)

The LWR method is described as:

LWR: Fit Θ to min J( Θ )

Return D_aq Θ .

4.2.2 Local condition for LWR

The essence of the LWR method is to approximate a local model to the input data point by assigning distance weights to all data points in the database. In the above section, the parameters of the model are determined by using LSM. However, matrix operation in this method is time-consuming, especially when the matrix dimension is large. In this paper, a local condition for LWR is defined to ensure the local fitting and reduce the matrix dimension, which is denoted as following:

Set the distance threshold. Only when the distance from the input point is less than the distance threshold is it considered as a local model point.

Set the quantity limit. Find a few nearest data points away from the input data point in the database.

Thus, when the target pixel position is observed, the LWR model can be used to compute the initial action parameters of the robot hitting. When the empirical data are not sufficient, the LWR method only provides a rough solution, and there may be a large error in the hitting task results by using the action parameter obtained by this method. In the next section, a Q-learning reinforcement learning model would be trained to correct the action parameters further where the landing position error would be the input, so that precision of the robot hitting results would be improved.

4.3 Q-learning reinforcement learning

Reinforcement learning is to solve problems that an autonomous agent which is capable of observing the state of its environment and taking a series of actions to change those states. It achieves goals by learning to choose the optimal action. When the agent takes each action in its environment, the demonstrator provides reward or penalty information to indicate whether the status of the result is correct or not. The task of the agent is to learn from these indirect, delayed rewards, and the task of learning is to obtain a control strategy that can select the appropriate action from any initial state so that the agent can maximize rewards over time, see Fig. 4.

Fig.4

Reinforcement learning.

Q-learning algorithm is a reinforcement learning method similar to the dynamic programming algorithm proposed by Watkins in 1989, and learns the optimal action strategy by interacting with the environment. It considers the interaction between the agent and the environment as a Markov decision process(MDP) that the current state of the agent and the chosen action determine a fixed state transition probability distribution, the next state, and get an immediate reward. Q-learning defines an evaluation function Q(s, a) whose value is the maximum discount cumulative reward starting from the state s and the action a. Then the evaluation function is learned and the optimal strategy is selected according to it. The evaluation function is defined as follows: $Q (s, a) = r (s, a) + γ \max a' Q (δ (s, a), a')$ (14)

Where the r(s, a) is the immediate reward of action a for state s; γ is the discount factor; δ(s, a) is the state transmitted from s after taking action a. Each Q(s, a) corresponds to a Q value, and in the learning process, the action is selected according to Q value. The expression of the optimal strategy is as follows: $π^{*} (s) = {argmax}_{a} Q (s, a)$ (15)

The symbol $\hat{Q}$ is used to represent the estimation of the actual Q function by learning machine that a large table represents its hypothesis $\hat{Q}$ where each state-action pair <s, a> corresponds to an entry. $\hat{Q} (s, a)$ is approximated recursively according to following rule: $\hat{Q} (s, a) \leftarrow r + γ {max}_{a'} \hat{Q} (s', a')$ (16)

As long as the system is modeled as a deterministic Markov process, the reward function r is bounded and the action-select mechanism ensures each state-action pair could be infinitely accessed, the $\hat{Q}$ can converge to the real Q function at the limit. Then the optimal strategy would be obtained.

As a model-free learning method, Q-learning can construct learning objects without system model which turns out to be convenient. However, in order to achieve good performance, Q-learning requires a lot of empirical data(sample data), and the learning efficiency needs to be improved. When the state space and the decision space are large, the training convergence is slow.

4.4 Learning method combining LWR with Q-learning algorithm

4.4.1 The structure of learning method

Based on the LWR learning method, the mapping between the set of landing position and the set of the manipulator action parameters is established. The local model is built to fit the input sample points, which has the ability of generalization. LWR method only relies on the stored empirical data points, and the number of empirical data has a great influence on the performance of LWR method. However, the multi-DOFmanipulator has a high dimension, which means that a large amount of empirical data is needed to train the LWR model and the learning efficiency will decrease. Therefore, the LWR method is only used as a rough estimate when the empirical data are not sufficient.

Q-learning reinforcement learning can increase the ability of local generalization that it can further correct the manipulator parameters according to the actual error after hitting. Reinforcement learning, that divides the state space and action space into small parts, is expected to enhance the learning system control accuracy and compensate the uncertainty and error of model and system. When reinforcement learning is used alone, there would be a problem of dimension disaster in action and decision space. Thus, we propose a combination learning method of the LWR and Q-learning, where LWR is firstly used to fit a local small model and then the Q-learning is used in this local small model which greatly reduce the dimension of the state and decision space ensuring the same division accuracy. As a result, the training times decreased, and the learning efficiency and convergence rate of Q-learning are improved as well. Figure 5 shows the architecture of the learning method

Fig.5

Architecture of the learning method.

When the target position is given, the desired landing position of ball is obtained according to the coordinate information. Taking these parameters as the input of the controller, the optimal initial manipulator parameters are predicted in the LWR model.

Then the first hit is performed and the error between the actual and desired landing position observed by the camera would betaken as input to the Q-learning reinforcement learning model, where the adjustment parameters of manipulator would be obtained as output. Finally, the action parameters of second hit are sum of the initial parameters and the adjustment parameters and thus the manipulator performs a more accurate hit.

4.4.2 Adding Q-learning to LWR

In this paper, firstly, the method of LWR is used to make a rough estimation. Then suppose that the corresponding state transition function δ(s, a) of each point near the landing point obeys the same probability distribution, where the s represents the error of landing position and a represents the manipulator action correction parameters. The state space and the decision space are reduced from the whole original set of ball landing points and the manipulator actions set to the local small set of points near the landing point and the corresponding manipulator action set, which greatly reduces the dimension of the state and the decision space. The convergence time of training decrease as well.

For convenience, the state set S< d_xv, d_yv > is simply expressed as S : {x(x¹, x²)}, and the action set A< ΔR_x, ΔR_y, Δv > is simply expressed as A : {a(a¹, a², a³)}. The reward function R is defined as following: $R (x, a) = V_{r} e^{- λ Δ d}, 0 \leq Δ d \leq r$ (17)

Where V_r is the reward value when the deviation of landing position equals 0; r is the boundary of state region; λ is a sufficiently large parameter which can guarantee that e^-λΔd converges to 0 when Δd is equal to r; Δd is the deviation of landing position. The exponential reward function has such a characteristic that a significantly higher reward value is obtained in the area near the target point and a very small reward value is obtained in the area away from the target point. Compared to the linear reward function, it emphasizes the role of good training samples near the target location and weakens the disturbance of relatively poor training sample away from the target location more intensely. In this way, training result can be quickly converged to the actual Q function, and the convergence rate is improved. Algorithm 1 is adjusting parameters of hitting ball algorithm based on Q-learning.

For this algorithm, this chapter has the following explanations:

In Step 1, the action would be selected with uniform probability.

In Step 4 and 5, this paper used ɛ–greedy heuristics, where ɛ = 0.01.

Since the hitting ball task is a simple single-step reinforcement learning model that “actions obtain status and directly get the final reward,” which does not involve a multi-step Markov decision process, in Step 6, the value does not required foriterationof other states.

In Step 9, when the execution of state $\vec{a}$ is not the expected $\vec{x}$ but another state $\vec{x^{'}}$ , update the Q(x^′, a) as a means to use all the training samples effectively and reduce the training costs.

In Step 12, parameter ξ ∈ R⁺ represents minimum Q value of the end of training some states.

5 Simulation and experiment

5.1 The description of the hitting task experiment

The task of experiment is to hit the ball to the target position on the platform with abat that is on the end of the manipulator, where the target position is arbitrary within the working range of the camera field of view. At the beginning of the experiment, we demonstrate the manipulator to hit the position-fixed ball by using different poses and hitting speeds, and the landing position is observed by an industrial camera. The action parameter set and the target observation set are built as the training set. LWR method is used to establish the mapping between the two sets. During the experiment, the UR manipulator performs the hitting task using the action parameters predicted by the LWR method and the landing positionis obtained by the industrial camera. Through the error of actual observation and expected result, the Q-learning reinforcement learning model is trained, and the action parameters of the manipulator are further adjusted to improve the accuracy of the second hit.

In this paper, the simulation control system is set up in the ADAMS which is used to verify the method feasible and have good effect. Then, the experiment on the experimental platform built around the UR robot is performed to test the practical effect of the method.

5.2 Simulation of the control method

5.2.1 The training of LWR model

The LWR model is firstly trained. The ball is placed on a fixed position. Demonstrate manipulator to hit the ball from 19 different directions ranging from -45° to +45° every 5°, see Fig. 6. In each hitting direction, by adjusting the velocity value of the bat, which is ranging from 400mm/s to 1500mm/s, the landing position set takes 18 points in each direction. Record the data of each hit those are effective, including the landing position D_d =(x_vd, y_vd) and the parameters of the manipulator H =(R_x, R_y, v). 342 sets of data are acquired to establish the database. The mapping relationship between the two sets is built by the LWR method, and in this paper τ equals 0.1.

Fig.6

Hit the ball from 19 different directions.

5.2.2 Training of Q-learning model

In order to further improve the accuracy of the hitting task, an reinforcement learning model was trained to output adjusted action parameters with the first-hit deviation of landing position as the input. Select(950, 0) as the reference point. Taking this point as the center, a local rectangular region(length of 30 mm, width of 24 mm) is expanded and divided into 15 small rectangles(length of 6 mm, width of 8 mm).In this region, training is performed according to Algorithm 1.The training process is shown in the Fig. 7, the higher the gray value, the higher the Q value of the corresponding region. And the black area indicates the Q value is 0, while the white area indicates that the Q value reaches the convergence condition.

Fig.7

Machine learning training process.

It can be seen from the Fig. 7 that, using the Q-learning method, the robot will update the Q value in the Q matrix by learning, trying and rewarding, which turns out to get an optimal strategy set. The set can determine the current state value according to the deviation of the first ball landing position and select the optimal action, that is, the adjustedaction parameters, to perform the second hit. Finally, after nearly 650 training sessions, the Q value in each state of the Q matrix convergence to the maximum value, and the training completes. Compared with the method in [23], which used Q-learning algorithm into the whole action and decision space, the number of training reinforcement learning model is significantly reduced in our method.

5.2.3 Simulation results

After the model training, in the working range of the system, 58 target position points are randomly selected to perform two hits. The first round hit is based on the LWR model, and the second round hit is based on the LWR model adding the Q-learning. As a contrast, GPM is used to predict the parameters of the robot based on the same dataset. The error analysis is shown in the Fig. 8.

As shown in Fig. 8, the horizontal coordinate is the hit sequence, and the vertical coordinate is the error of landing position. The Green solid line represents the hitting error based on the GPM. In 58 hits, the maximum error is 47.94 mm, the minimumerror is8.33 mm, and the average error is 23.59 mm. The variance of the erroris 8.49, which indicates that the fluctuation of error is very large, and in some situations, there would be poor predictions. The reason for these phenomena may bethat the observation data do not follow the multivariable Gaussian distribution strictly or the initial parameters are not reasonable. The blue solid line represents the hitting error based on the LWR model. In 58 hits, the maximum error is 16.12 mm, the minimum error is 3.61 mm, and the average error is 8.00 mm. The variance of the erroris 2.24, which is much smaller that variance of GPM. The figure also illustrates that, the error using LWR is generally smaller than the GPM. Using the same dataset, LWR shows better performance than GPM in our hitting ball simulation.

Fig.8

Error analysis.

The red solid line is the hit error of the LWR model adding the Q-learning reinforcement learning method. In 58 hits, the maximum error is 6.13 mm, the minimum error is 0.24 mm, and the average error is 2.74 mm. It can be seen from the figure that, in the same sequence of hitting simulation, the LWR model adding Q-learning has the smallest deviation, and the hitting precision is improved. Besides the overall average deviation of landing position is also smaller than the simple LWR method. Our control method has good performance.

5.3 Experiments

Perform the hitting task experiment on the platform built around the UR. Firstly, 12 target points are randomly selected in the training range of LWR model, and the action parameters of the robot are calculated by LWR and LWR adding Q-learning respectively. The result of landing position and the calculated parameters of manipulator are shown in Tables 1 and 2.

Algorithm1

Input: E, A, x₂, γ, α; E: Environment; A: Space of action;

x₂: Initial state; γ: Discount of reward; α: Update step size;

Q (x, a) = 0, π (x, a) = \frac{1}{| A (x) |}

;

x = x₂;

for t = 1, 2, … do

r, x′ =

The reward and transfer states generated by

performing π^ɛ(X)

;

a′ = π(x′);

Q(x, a) = R(x, a) + Q(x, a);

π (x) = \underset{a^{''}}{argmax} Q (x, a^{″})

;

if x ≠ x′ then

Q(x′, a) = R(x′, a) + Q(x′, a);

π (x^{'}) = \underset{a^{″}}{argmax} Q (x, a^{″})

;

end if

if Q(x′, a) ≥ ξ then

separate out X′ from S;

if S = φ then

break;

end if

X = x′, a = a′

end for

Output: Strategy π

Table 1
Target landing position and Reality landing position

Num Target position Reality landing position Reality landing position

with LWR with LWR adding Q-learing

x(mm) y(mm) x(mm) y(mm) Error(mm) x(mm) y(mm) Error(mm)

1 314 77 332 82 18.7 313 75 5.7

2 162 –28 183 –23 21.6 163 –26 5.7

3 222 –83 238 –75 17.9 223 –84 6.4

4 339 –32 356 –28 17.5 336 –31 10.3

5 343 –93 360 –98 17.7 342 –91 7.8

6 229 –14 248 –18 19.4 228 –13 7.2

7 175 –55 195 –50 20.6 173 –54 7.6

8 286 77 302 71 17.1 287 75 7.8

9 136 –51 158 –56 22.6 135 –50 7.7

10 187 97 205 91 19.0 182 94 11.2

11 191 87 209 80 19.3 188 85 7.8

12 272 –81 289 –75 18.0 272 –78 5.6

Num	Target position	Reality landing position	Reality landing position
1	314	77	332	82	18.7	313	75	5.7
2	162	–28	183	–23	21.6	163	–26	5.7
3	222	–83	238	–75	17.9	223	–84	6.4
4	339	–32	356	–28	17.5	336	–31	10.3
5	343	–93	360	–98	17.7	342	–91	7.8
6	229	–14	248	–18	19.4	228	–13	7.2
7	175	–55	195	–50	20.6	173	–54	7.6
8	286	77	302	71	17.1	287	75	7.8
9	136	–51	158	–56	22.6	135	–50	7.7
10	187	97	205	91	19.0	182	94	11.2
11	191	87	209	80	19.3	188	85	7.8
12	272	–81	289	–75	18.0	272	–78	5.6

Table 2

Parameters of manipulator

Num	Initial parameters			Adjusted parameters
	V	P _x	P _y	V	P _x	P _y
1	1015	2.46	–1.96	985	2.46	–1.96
2	533	–2.04	2.39	503	–2.04	2.39
3	746	–1.82	2.56	716	–1.80	2.58
4	1069	–2.12	2.32	1039	–2.12	2.32
5	1114	–1.93	2.48	1084	–1.93	2.48
6	727	–2.16	2.28	697	–2.16	2.28
7	585	–1.89	2.51	555	–1.87	2.53
8	931	2.48	–1.93	901	2.48	–1.93
9	469	–1.82	2.56	439	–1.80	2.57
10	660	2.66	–1.67	630	2.68	–1.64
11	660	2.62	–1.73	630	2.63	–1.71
12	893	–1.89	2.51	863	–1.89	2.51

According to the tables, in the experiment of using LWR, the maximum error is 22.6 mm, while the minimum error is 17.1 mm, and the average error is 19.1 mm. On the other hand, in the experiment of using LWR adding Q-learning, the maximum error is 11.2 mm, while the minimum error is 5.6 mm, and the average error is 7.6 mm. It is obvious that LWR adding Q-learning has smaller errors and performs better. The distribution of the result is shown in Fig. 9.

Fig.9

The distribution of result.

In Fig. 9, the green point indicates the target landing position. The red point indicates the first hitlanding position by using LWR model, and the blue point indicates the second hit landing position by using LWR adding Q-learning method. The model of LWR has good training effect, but there is still possibility to continue to enhance the precision. And it is obvious that in each hit, the blue point is closer than the red one to the target point. The accuracy of the second hit has been significantly improved and the method of LWR adding Q-learning has better effect. The proposed method turns out to be feasible and effective.

6 Conclusion

A robot demonstration method is proposed based on the LWR and Q-learning reinforcement learning algorithm. This method can adapt to the work task by learning the demonstrate actions and generating new actions. The learning method consists of two parts. The first part is the learning model of multivariable locally weighted regression, which establishes the mapping relationship between the target value and the action. The second part is the Q-learning reinforcement learning model which outputs the adjustment parameters of action by using the error of the first step as input. The accuracy of the task is improved with the LWR and Q-learning algorithm. Besides, the regression learning fits a local small space to approximate the global state and decision space, which turns out to reduce the dimension and the number of training reinforcement learning. The convergence rate is improved as well. The feasibility and effectiveness of the proposed method are verified by establishing simulation on ADAMS. Areal hitting ball experiment on a 6-DOF UR robot is given and the validity of the algorithm is verified.

7 Future work

In this paper, our work is limited to a static initial state where the ball is located in a fixed position and the hitting pattern is predefined. In future work, we will further verify and enhance the adaptability of the method for using the manipulator to hit the ball in motion and learning the new hitting patterns. Besides, the model proposed only considered the kinematic model of the robot, and dynamic-related parameters were not taken into account. The method may be more robust by installing a force sensor on the end-effector, which would be helpful to collect the parameter information relative to the dynamic model.

Footnotes

Acknowledgments

This work was supported by the Science Fund Project No.61462089 of China’s NSF, the Scientific and Technological Research Program of Chongqing Municipal Education Commission No. KJ1501330, Chinese MIIT Intelligent Manufacturing and New Mode Application project “Application of new mode of intelligent manufacturing of Chinese medicine products”.

References

Schaal

, Learning from demonstration, Robotics & Autonomous Systems47(2–3)(2004), 65–67.

Nehaniv

C.L.

, Dautenhahn

, Imitation and social learning in robots, humans and animals, Cambridge University Press2007.

Englert

, Paraschos

, Peters

and Deisenroth

M.P.

, Model-based imitation learning by probabilistic trajectory matching, IEEE International Conference on Robotics and Automation IEEE2013, pp. 1922–1927.

Calinon

, Guenter

, Billard

, On Learning, Representing, and Generalizing a Task in a Humanoid Robot, IEEE Press, 2007.

Piot

, Geist

and Pietquin

, Bridging the gap between imitation learning and inverse reinforcement learning, IEEE Transactions on Neural Networks & Learning Systems28(8)(2017), 1814–1826.

Hwang

K.S.

, Jiang

W.C.

and Tseng

Y.C.

, An unified approach to inverse reinforcement learning by oppositive demonstrations, IEEE International Conference on Industrial Technology IEEE2016, pp. 1664–1668.

Levas

, Selfridge

, A user-friendly high-level robot teaching system, IEEE, IEEE International Conference on Robotics and Automation Proceedings,(Vol.1),(1984) pp. 413–416.

Kreuziger

, Kaiser

and Dillmann

, Robot Programming by Demonstration(RPD) - Using Machine Learning and User Interaction Methods for the Development of Easy and Comfortable Robot Programming Systems, International Symposium on Industrial Robots(1994), pp. 685–693.

Muelling

, Kober

, Peters

, Learning table tennis with a Mixture of Motor Primitives, IEEE-Ras International Conference on Humanoid Robots IEEE2010, pp. 411–416.

10.

Rasmussen

C.E.

and Williams

C.K.I.

, Gaussian Processes for Machine Learning(Adaptive Computation and Machine Learning), The, MIT Press2005.

11.

Garcia-Sillas

, Gorrostieta-Hurtado

, Soto-Vargas

, Diaz-Delgado

and Rodriguez-Rivero

, Learning from demonstration with Gaussian processes, Mechatronics, Adaptive and Intelligent Systems IEEE(2016), pp. 1–6.

12.

Matsushima

, Hashimoto

, Takeuchi

and Miyazaki

, A learning approach to robotic table tennis, IEEE Transactions on Robotics21(4)(2005), 767–771.

13.

Schaal

, Atkeson

C.G.

, Vijayakumar

, Real-time robot learning with locally weighted statistical learning. In Robotics and Automation, Proceedings ICRA’00 IEEE International Conference on(Vol. 1), IEEE2000 pp. 288–293.

14.

Huang

, Xu

, Tan

and Su

, Adding active learning to LWR for ping-pong playing robot, IEEE Transactions on Control Systems Technology21(4)(2013), 1489–1494.

15.

Peters

and Schaal

, Policy gradient methods for robotics, In Intelligent Robots and Systems, 2006 IEEE/RSJ International Conference on IEEE, 2006, pp. 2219–2225

16.

Englert

, Paraschos

, Peters

, Deisenroth

M.P.

, Model-based imitation learning by probabilistic trajectory matching, In Robotics and Automation(ICRA), 2013 IEEE International Conference on IEEE, 2013, pp. 1922–1927.

17.

Englert

, Paraschos

, Deisenroth

M.P.

and Peters

, Probabilistic model-based imitation learning, Adaptive Behavior21(5)(2013), 388–403.

18.

Guenter

, Hersch

, Calinon

and Billard

, Reinforcement learning for imitating constrained reaching movements, Advanced Robotics21(13)(2007), 1521–1544.

19.

Bayoumi

, Bennewitz

, Learning optimal navigation actions for foresighted robot behavior during assistance tasks, In Robotics and Automation(ICRA), 2016 IEEE International Conference on IEEE, 2016, pp. 207–212.

20.

Ansari

, Falotico

, Mollard

, Busch

, Cianchetti

, Laschi

, A Multiagent Reinforcement Learning approach for inverse kinematics of high dimensional manipulators with precision positioning, In Biomedical Robotics and Biomechatronics(BioRob), 2016 6th IEEE International Conference on IEEE, 2016, pp. 457–463.

21.

Ghadirzadeh

, Bütepage

, Maki

, Kragic

, Björkman

, A sensorimotor reinforcement learning framework for physical human-robot interaction, In Intelligent Robots and Systems(IROS), 2016 IEEE/RSJ International Conference on IEEE, 2016, pp. 2682–2688.

22.

Chebotar

, Hausman

, Su

, Sukhatme

G.S.

, Schaal

, Self-supervised regrasping using spatio-temporal tactile features and reinforcement learning, In Intelligent Robots and Systems(IROS), 2016 IEEE/RSJ International Conference on IEEE, 2016, pp. 1960–1966.

23.

Wang

, Liang

, Zhou

, Zhang

, Kicking motion design of humanoid robots using gradual accumulation learning method based on Q-learning, In Control and Decision Conference(CCDC), 2016 Chinese IEEE, 2016, pp. 5274–5279.