Research on adaptive grasping with object pose uncertainty by multi-fingered robot hand

Abstract

In the process of grasping objects by robot, the uncertainty of the object pose has greater impact on the stability of grasping. In this article, an adaptive grasping strategy is proposed to solve the problem of pose uncertainty in the grasping process. The strategy is based on the finger state function to switch control method, so as to realize the contact with the object by the fingers, and the application of grasp force to ensure the pre-grasp stability of the object; then, the object pose error is estimated through the contact between finger and object, and the pose error of the object is compensated according to the different error ranges to reduce the influence of the pose error on the grasp stability. Finally, the grasping experiments by humanoid robot are carried out to verify the performance of the adaptive grasping method proposed in this article.

Keywords

Adaptive grasp robot hand pose uncertainty estimation error compensation

Introduction

The grasp of unknown objects is the basic ability for robots to perform a variety of complex tasks instead of hand. When the object model and pose are accurately known, a set of points that meet the requirements of grasp can be obtained by grasp planning, and the fingers reach the grasp point and apply the grasp force according to the trajectory planning so as to complete the stable grasp of the object. However, the actual grasp is a dynamic process with a variety of uncertainties. In the uncertainty of the grasp, the pose error of the object is ubiquitous and has a great influence on the grasp, which may be resulted from the following aspects: (1) the location error of the sensor, including location sensor, tactile sensor, and visual sensor; (2) a slight collision between the robot and the object in the course of the implementation, resulting in a slight change in the location of the object; and (3) model reconstruction error of the object. These errors will make the actual grasp inconsistent with the grasp plan, resulting in the low quality of actual grasp or even the fail of grasp.

Due to the inherent error of the robot, the relative position of the object and the finger can only be approximately known, which makes it difficult for the fingertip to be in precise contact with the object at the position obtained by the analysis. In response to these questions, Nguyen¹ defined the concept of independent contact regions (ICRs). The method for calculating ICR was proposed by Roa and Suárez² and Krug et al.³. Zheng and Qian⁴ analyzed the effects of friction uncertainty and the uncertainty of the contact position on the sealing force in the grasping object and modeled the two uncertainties through rigorous mathematical description. Weisz and Allen⁵ put forward better evaluation indicator P(fc) to evaluate grasp uncertainty for grasp stability. Kim et al.^6,7 studied how to evaluate the stability of the object grasp with pose uncertainty by the grasp quality and proposed a quality evaluation indicator for the consideration of the contact dynamics and pose uncertainty.

Some methods are used to estimate the uncertainty directly for errors in the position of object. Chalon et al.⁸ proposed an online method for estimating the pose of object based on the particle filter to accurately estimate the pose of the object to be grasped, for the change of pose after grasp. Zhang and Trinkle⁹ also used the particle filter to simultaneously estimate the physical parameters of the object and to track the object when it is pushed. There are other ways to reduce the impact of uncertainty from the perspective of control strategies. Dragiev et al.¹⁰ proposed a control law for the grasp and exploration of unknown objects to select the grasp area of the model. Su et al.¹¹ proposed a grasp strategy based on the softness of under-actuated hand for the object position error, by which the stable grasp can be completed through the optimization of the grasp trajectory planning and sensor-based adjustment. Chen et al.¹² proposed an adaptive compliant grasp strategy to improve the impact of position uncertainty on self-grasp. There are also some studies based on learning and heuristic methods. Hsiao et al.¹³ proposed a heuristic grasp selection method based on the local features of unknown objects. In addition, a decision-theoretical method is also proposed to reduce the impact of the object attitude error on the grasp for the known shape of the object and the high premise mobility of the robot.^14,15 Stulp et al.¹⁶ proposed a grasp optimization method for uncertainty state estimation. Kehoe et al.¹⁷ studied the object position error and proposed a grasp algorithm that applies cloud computing structure based on the parallel gripper grasp. Guo et al. proposed an end-to-end deep vision network model to predict possible good grasps from real-world images in real time.^18,19

In order to enable the robot to perform tasks in the human environment, the grasp of uncertainty is one of the urgent problems to be solved. The current research on grasp uncertainty has been taken from the aspects of grasp control method, grasp quality assessment, pose error estimation, and grasp task space to increase the stability. The approach used includes both the improvement to traditional grasp methods and the decision-making and learning-based approaches. Each of the abovementioned research methods has limitation. The impact of uncertainty on the grasp has not yet been eliminated, so it is still worthy of study in dexterous grasp.

In this article, a multi-controller–based method to stably grasp a known object is proposed. The main challenge of the grasp comes from the pose uncertainty of object. To this end, the whole grasping procedure is divided into three stages, namely approaching, pre-grasp, and grasping. The contacts between multi-fingertips and object are established in pre-grasp, and the pose error is estimated using the tactile feedback and forward kinematics of robot hand. Finally, Q-distance is used as the criterion of grasp stability to improve the grasp performance considering the compensation of pose error.

The main contribution of this article is as follows. A new adaptive grasp strategy is proposed for the failure of the grasp by robot caused by the pose error. Based on the force coordination strategy, the stable grasp is realized in the presence of the pose uncertainty through the estimation and compensation of the pose error by tactile information.

Background

Hand controller

Two types of controllers are used in the grasping: impedence controller and force controller.

Impedance controller

Based on the detected force information, the adjustment values of position, velocity, and acceleration are obtained by position-based impedance control and according to the calculation using the expected impedance relation, and then, the adjusted target position is tracked and controlled by the position loop to realize the flexibility of the contact space and, meanwhile, ensure the accuracy of the free space position. The control law is shown as follows

τ = \hat{M} {\ddot{q}}_{d} + K_{D} \dot{\tilde{q}} + K_{p} \tilde{q} + \hat{C} + \hat{g} + {\hat{τ}}_{f} + τ_{ext}

where $\hat{M}$ is the estimation value of the inertia matrix, $\hat{C}$ is the estimation value of Coriolis force and centrifugal force, $\hat{g}$ is the estimation value of gravity term, ${\hat{τ}}_{f}$ is the estimation value of joint friction torque, τ_ext is joint disturbance torque, and K_D and K_p are expected damping matrix and stiffness matrix, respectively.

Force controller

The size of the external force of the finger F_ext and the target impedance relationship directly affect the position compensation amount X_f and ultimately affect the final position of the finger. The controller is suitable for finger free-space movement and the application of the grasp phase but not suitable for the control on contact stage between the finger and the object. In order to realize the contact between fingers and objects with a small force, the force control is switched after the contact, and the contact with the object by a small force is realized by setting smaller expected force.

F_d is the expected fingertip force, and the fingertip force error is

F_{e} = F_{d} - F_{ext}

$K_{p}^{f}$ and $K_{I}^{f}$ are the control parameters of force error and its integral term, respectively, and then, the control law of force controller in Cartesian space of the finger is

τ = K_{p}^{f} F_{e} + K_{I}^{f} \int F_{e} d t

Grasp planner based on finite-state machine

We use state-action space defined in the semantic level to describe the switch strategy of grasp planner. In action space:

A1: Finger controller switches to impedance control, finger continues to move as preplanning;

A2: Finger controller switches to impedance control, finger continues to move as replanning;

A3: Finger controller switches to force control, stopping the movement and maintaining the current contact force; and

A4: Finger controller switches to force control, finger applies the grasp force.

In state space:

S1: $F_{ext}^{i} < F_{r}^{i}, X^{i} < X_{r}^{i}$

S2: $F_{ext}^{i} < F_{r}^{i}, X^{i} \geq X_{r}^{i}$

S3: $F_{ext}^{i} > F_{r}^{i}, X^{i} < X_{r}^{i}$

S4: $F_{ext}^{i} > F_{r}^{i}, X^{i} > X_{r}^{i}$

S5: $F_{ext}^{i} > F_{r}^{i}, X^{i} = X_{r}^{i}$

where

X_{r}^{i}

and X_i are the target position and real position, respectively, and

F_{r}^{i}

is the target force of the contact between the finger and the object. According to the fingertip position X and the external force F_ext in the movement of finger, the state of the finger is determined and the grasping action is determined according to the finger state function.

For the ith finger, the state function of finger is defined as $S_{g}^{i}$ . The whole object adaptive grasp strategy process can be formalized as an finite state machine (FSM). The action of finger switched from A1 to A4 when the value of $S_{g}^{i}$ is switched from S1 to S5. The detail of state transition is described in the next section.

Grasp quality metric

Grasp stability should be firstly considered in the grasping process. In order to measure the disturbance resistance ability of a grasp, some grasp quality measures are proposed. Q-distance metric²⁰ is chosen to measure the stability of grasp, and it is defined as follows

d_{Q} = {\begin{array}{l} d_{Q^{+}}, 0 \notin CH (W) \\ d_{Q^{-}}, 0 \in CH (W) \end{array}

The geometric description of this metric is consistent with the previous defined grasp quality measures,²¹ but it can measure the grasp with or without force closure (FC). A grasp is FC if and only if $d_{Q^{-}} < 0$ , and smaller value of Q⁻ distance indicates the better capability of the grasp in resisting unknown external disturbances. When the grasp is not FC, the value of Q⁺ distance indicates that how far that the grasp achieves the FC condition. The distance of both Q⁺ and Q⁻ can be solved by finding the minimum of constrained linear multivariable function.

Adaptive grasp strategy for object pose uncertainty

The process of grasping objects is dynamic, which can be divided into three stages in this article: approach, contact, and grasp. When the dexterous hand reaches the designated position through the robot arm, the fingers of dexterous hand firstly move to the target point, and then, the finger comes into contact with the target point and exerts contact force to complete the grasp. When the object has a pose error, it may cause the following problems: (1) the fingers may contact with the object one after another in the process of grasp, which cannot be resolved by speed planning. As the grasp is not closed at this time, the finger contacting the object is equivalent to applying an interference torque to the object, resulting in an unstable state of the object during contact; (2) the finger does not come into contact with the object at the target point of position, but it is in contact with the neighborhood of the desired point or even fails in reaching the object, which may cause the low quality of grasp or even failure. The adaptive grasp strategy in this section will focus on these two aspects.

A multi-finger cooperative control for adaptive grasp

In this section, a multi-finger cooperative control strategy is designed for grasp objects. The strategy is based on finger impedance control and force control, by which the fingers contact with the object one after another through the coordination of each finger movement in the grasp process; meanwhile, the grasp force is exerted to avoid the instability in the process of grasp. In order to distinguish from the grasp process of object, the stage of maintaining a small contact force in the contact between the finger and the object without grasp force applied is called pre-grasp. During the grasp process, the state of the finger relative to the object is switched from approaching, contacting, and applying the grasping force, respectively. Different control methods are switched by judging the different states of the fingers during the grasp process to ensure the stability of the pre-grasp process.

When $S_{g}^{i} = S 1$ , the finger is approaching to the object but not contacting the object, then action 1 (A1) is taken; when $S_{g}^{i} = S 2$ , it means the finger reaching the target position but does not contact with object, which indicates the fail of contact due to the pose error, then action 2 (A2) is taken; and at this moment, $X^{i} > X_{r}^{i}$ ; then, the finger goes on to the new movement trajectory, and when the contact happens, $S_{g}^{i} = S 4$ , and then, action 3 (A3) is taken; when $S_{g}^{i} = S 3$ , it means the finger contacts with the object in advance before reaching the target position due to the pose error, and then action 3 (A3) is taken; when $S_{g}^{i} = S 5$ , the finger reaches the target position and contacts with the object, and then action 3 (A3) is taken.

The position-based impedance control in the Cartesian space is used when the finger begins to move. When $S_{g}^{i}$ of the finger is equal to S1 or S2, it indicates that the finger is not in contact with the object at this time, and the finger still uses the impedance control movement; when $S_{g}^{i}$ is equal to S3, S4, or S5, the finger has been in contact with the object, and the finger switches to force control, stopping the movement and maintaining the current contact force. In order to avoid the pose change of the object and to maintain contact with the object after the contact, $F_{r}^{i}$ should be set small enough. When all the finger states in plan that need contact have become S3, S4, or S5, it indicates that all the fingers of the dexterous hands have been in contact with the object, and the current position of each finger is recorded. The controllers of all fingers are then switched back to the impedance control and a further distance is taken in the direction of the force as a target position to ensure sufficient grasp force and eventually complete the grasp, which means action 4 (A4) is taken. Therefore, when all the fingers and objects have a slight contact, the fingers begin to apply the grasp force at the same time to complete the grasp, to avoid the unstable phenomenon in pre-grasp process.

Design of finger controller

The controller of the finger is the basis for implementing the multi-finger coordination strategy proposed in the previous section. In addition to ensuring the accuracy of the position, it is necessary to ensure a certain degree of flexibility, in order to avoid damage to objects and fingers; it shall keep as little contact force as possible to avoid pushing the object in the contact stage. So in the stage of grasping object, the position-based impedance control is applied to complete the grasp; and in contact stage between the finger and the object, the finger force control is applied to achieve precise control of the contact force.

Figure 1 shows the structure diagram of the controller used in the finger. The controller includes impedance control and force control, and the switching condition of the controller is determined by the state function vector of the finger S_g. The controller combines the advantages of impedance control and force control, which is suitable for the implementation of multi-force coordination grasp strategy.

Figure 1.

Diagram of a finger controller.

Estimation of pose error of objects based on pre-grasp tactile information

Even if the multi-force coordination strategy is used to ensure the stability of the pre-grasp process, the final grasp point will change and the quality of grasp may be affected due to the presence of the object position error. Therefore, in this section, how to estimate the pose error of the object by the information obtained in the pre-grasp process is mainly discussed.

In the pre-grasp process, the actual contact point between the finger and the object can be obtained by the arm–hand kinematics based on the tactile information. Using the object model, the expected contact point, and the actual contact point information, the pose error of the object can be calculated by geometric calculation. The pose error discussed in this article is limited to the extent that the pose error does not cause the hand to collide with the object or move away from the hand when approaching the object.

Several common objects, such as spheres, cuboids, cylinders, and vertebral, are taken for analysis. For the objects placed on a plane, the pose error in grasp includes the position error and attitude error on the plane. The definition of the object coordinate system is used in this chapter. Therefore, for the rotating body (such as sphere, cylinder, and cone), the error includes only the position error of the object in the plane (Δx, Δy); for the nonrotating body (cuboids and triple prisms), the angular error (Δθ) around the z-axis is also included in addition to the position error. The pose error estimations of the rotor and the nonrotating body are discussed by taking the sphere and the rectangular object as examples.

Figure 2 shows a grasp scenario, and the related coordinate frames and pose error are defined. The object with the height h is placed at the origin of the coordinate system, the center of mass is $O = (0, 0, z_{mc})$ , and the desired fingertip contact point is $E_{i} = (x_{r i}, y_{r i}, z_{r i})$ . When the object has pose error, its center of mass changes to $O^{'} = (Δ x, Δ y, z_{mc})$ , the actual coordinates of contact point $F_{i} = (x_{i}, y_{i}, z_{i})$ . The distance between the coordinates of the center of mass of the object and the center of mass of the actual position is defined as pose error. Figure 3 shows the projection of the object and the contact point in the plane x − y, where the circle represents the target contact point, and the oval represents the actual contact point.

Figure 2.

Grasp scenario.

Figure 3.

Model projection when grasping the ball.

For a sphere, let the radius of the sphere be R, then, the distance from the axis of symmetry of the object to the contact point can be expressed as follows

‖ O^{'} F_{i} ‖ = \sqrt{2 R z_{i} - {z_{i}}^{2}}

\sqrt{{(Δ x - x_{i})}^{2} + {(Δ y - y_{i})}^{2}} = ‖ O^{'} F_{i} ‖

When the number of contact points i ≥ 3, the object position errors Δx and Δy can be obtained from equations (5) and (6) as follows

Δ x = \frac{(c \cdot e - b \cdot f)}{(a \cdot e - b \cdot d)}

Δ y = \frac{(c \cdot d - a \cdot f)}{(b \cdot d - a \cdot e)}

where $a = 2 (x_{2} - x_{1})$ , $b = 2 (y_{1} - y_{2})$ , $c = x_{2}^{2} - x_{1}^{2} + y_{1}^{2} - y_{2}^{2} + ‖ O^{'} F_{1} ‖ - ‖ O^{'} F_{2} ‖$ , $d = 2 (x_{3} - x_{2})$ , $e = 2 (y_{2} - y_{3})$ , and $f = x_{3}^{2} - x_{2}^{2} + y_{2}^{2} - y_{3}^{2} + ‖ O^{'} F_{2} ‖ - ‖ O^{'} F_{3} ‖$ .

Similarly, the positional deviation of the rotors such as cylinders and frustums can be solved in this kind of manner.

For the box, it may also include posture deviation in addition to the position deviation. As the object is placed on the plane, only the attitude deviation (Δθ) around the z-axis is considered. Attitude deviation is usually more difficult to compensate with fingers. So two adjustments are adopted, as shown in Figure 4. First, the attitude deviation angle (Δθ) is estimated, and the attitude deviation is compensated by the motion of the manipulator; after adjustment, the new position is obtained and the position deviation is estimated by the following

Δ θ = arccos (\frac{c}{a b})

where $c = (x_{r 3} - x_{r 2}) (x_{3} - x_{2}) + (y_{r 3} - y_{r 2}) (y_{3} - y_{2})$ , $a = \sqrt{{(x_{r 3} - x_{r 2})}^{2} + {(y_{r 3} - y_{r 2})}^{2}}$ , and $b = \sqrt{{(x_{3} - x_{2})}^{2} + {(y_{3} - y_{2})}^{2}}$ .

Figure 4.

Model projection when grasping the cuboid includes pose error.

The position deviation is then estimated again by pre-grasp, and the final grasp is finished using the adjustment method for the second position compensation.

Δ x = \frac{(d_{1} - d_{2}) (x_{r 3} - x_{r 5})}{e}

Δ y = \frac{(d_{1} - d_{2}) (y_{r 3} - y_{r 5})}{e}

where $d_{1} = (x_{r 2} - x_{r 3}) y_{r 3} - (y_{r 2} - y_{r 3}) x_{r 3}$ , $d_{2} = (x_{5} - x_{3}) y_{3} - (y_{5} - y_{3}) x_{3}$ , and $e = {(x_{r 3} - x_{r 5})}^{2} + {(y_{r 3} - y_{r 5})}^{2}$ .

In this way, the pose deviation of the object by the dexterous hand can be estimated by one or two pre-grasps. As can be seen from Figures 2 and 3, when pose error exists, the fingers may not be in contact with the object or with the unplanned location. In the previous grasp planning, we have obtained the optimal grasp points, so the pose error will lead to a decline in the quality of the grasp. Therefore, the next section will continue to explore how to use the robot arm or finger to compensate for the deviation to ensure the stability of the grasp.

Adaptive grasp strategy based on pose error compensation

After the error of the object position is obtained, the error can be compensated by finger or hand. In addition, how to compensate for the error needs the analysis on the impact of different error ranges and different compensation methods on the stability of the grasp.

Analysis on the effect of pose error compensation on grasp stability

In order to evaluate the performance of multi-fingered grasp of objects, the researchers proposed a variety of indicators for different task requirements to assess the grasp, and these indicators are usually called grasp quality. In many indicators for the evaluation of the stability, Q-distance is selected as the evaluation indictor of grasp stability in this article.²⁰

When pose error exists, the failure of contact between the finger and the object may happen from the start to the desired location for some of the fingers. In order to ensure that the fingers are in full contact with the object, the finger is planned to move in a straight line along the direction of the desired contact force until it comes into contact with the object.

As shown in Figure 5, the spheroid with Φ = 72 mm is taken, for example, in which the dexterous hand is driven by the robot arm along the z-axis of the object coordinate system to approach, and there are four fingers in contact with the object. The initial positions of the fingers are $i n i_p = {I_{1}, I_{2}, I_{3}, I_{4}}$ , which are optimized by the previous researching method²² to obtain the expected positions $e x p_p = {E_{1}, E_{2}, E_{3}, E_{4}}$ , and the calculated grasp quality is d_Q = −0.38. The smaller dots in the figure represent the feasible contact areas where the fingers are subjected to kinematic constraints; while in the larger points, the circles, squares, and triangles represent the initial position, the expected position, and the actual contact position, respectively; blue, red, green, and black are on behalf of thumb, index finger, middle finger, and ring finger, respectively; and arrow is the direction of contact force. The initial position of the object is O(0, 0, 0), the dashed line indicates the object projection of expected position, and the actual position of the object is $O^{'} (12.5, - 12.5, 0)$ .

Figure 5.

Grasping the ball after fingers compensate for the position error when Δx = 12.5 mm and Δy = −12.5 mm.

The error is converted to the finger coordinate system and compensated by the finger, in which gray dotted line means the trajectory of the finger after adjusting the target point. The object is moved from the initial position to the expected position along the straight trajectory, where the ring finger (I₄) is in contact (F₄) with the object before reaching the desired position (E₄) and stops the movement, $S_{g}^{4} = S 3$ , and the other three fingers do not contact with the object after reaching the expected position, $S_{g}^{1} = S_{g}^{2} = S_{g}^{3} = S 2$ , so they continue to move along the original direction. For the thumb, if it continues to move along the original direction, it will be out of the workspace of the thumb due to the limitations of the workspace of the fingers, so we select the nearest point F₁ from the line $\bar{I_{1} E_{1}}$ in the workspace as the final contact point for the contact with the object. In this way, the actual contact point for the finger is $f i n a l_p = {F_{1}, F_{2}, F_{3}, F_{4}}$ . It can be seen that the thumb is limited by the workspace and far away from the expected point, which also leads to the final calculation value $d_{Q} = - 0.163$ , so the grasp quality declines a lot compared with the expected one.

In order to evaluate the impact of the two plans on the grasp quality more generally, the origin is taken as the starting point, $δ_{x} = 5 mm$ and $δ_{y} = 5 mm$ are the step lengths to change the position error of the sphere, and two different plans are used to plan the trajectory after expected points in each actual position. In addition, the grasp quality at each position of the object is calculated from the actual contact point obtained.

Figure 6 shows the grasp quality distribution in which the error is not compensated when there is a position error in the sphere. The position of the object in the plane x − y is represented by a number of identical squares, and the area of each box is 5 × 5 mm², where the brown–black box area is defined as reg_grasp, indicating the dexterous hand can grasp the object within the scope, the color from deep to shallow means the value d_Q from small to large in this position; and white square area is recorded as reg_out, which indicates the flexible workspace of the object out of the finger or the interfering area of the initial grasp position of the finger with the object, in which the object cannot be grasped. In reg_grasp, all the regions that d_Q < 0 are recorded as reg_fc, and the blue dashed line indicates the convex hull of the area $r e g_f c \subset r e g_g r a s p$ . In order to better compare the grasp quality distributions obtained by the two planning methods, the average grasp quality indictor avg_d_Q is defined to represent the average grasp quality when the object position error changes

{\bar{d}}_{Q} = \frac{Σ (\forall d_{Q}^{i} < 0)}{c o u n t}

where count indicates the number of the square units that reg_fc included.

Figure 6.

Grasp quality distribution map when grasping the object directly.

Figure 7 shows the grasp quality distribution when the finger compensation error is used. When the grasp is completed by the finger compensation method, the finger may be moved beyond the feasible contact area by compensating the motion plan, and the nearest point in the area from the new planning point is selected as the expected contact point. In Figure 7, there are 67 square regions that meet the requirement of close force grasp, and the average grasp quality indicator is ${\bar{d}}_{Q} = - 0.236$ ; compared with Figure 6, the grasp tolerance range is significantly improved (increased by 39.6%), but the average grasp quality has a certain decline (decreased by 4.8%).

Figure 7.

Grasp quality distribution when grasping the sphere with finger compensation.

According to the grasp quality distribution shown in Figure 7, when the error is in the smaller range, the quality of grasp is significantly improved compared with that in Figure 6; but when the error is large, the finger cannot completely compensate the position error due to kinematics and operability, which is the reason for the slight decrease in the average grasp quality, which is also the main problem of error compensation by finger. Therefore, when the error is beyond the adjustable range of finger greatly, the compensation error by finger does not mean the improvement of grasping stability. At this time, the complete compensation through the robot arm will ensure the contact of the fingers consistent with planning point.

For nonrotating bodies (such as cuboids and triangular prisms), the situation is more complex, of which the position uncertainty includes the position error and plane rotation angle error Δθ. Taking the rectangular box with the size of a = 40 mm, b = 60 mm, and h = 200 mm, for example, the initial position of the object is coordinate system origin, with the initial angle θ = 10°. As shown in Figure 8, there are five fingers in contact with the object, and the resulting grasp quality is d_Q = −0.37.

Figure 8.

Grasping the cuboid after fingers compensate for the position error when Δx = 7.5 mm, Δy = 12.5 mm, and Δθ = −20°.

The actual pose error of the object is set as Δx = 7.5 mm, Δy = 12.5 mm, and Δθ = −20°. Figure 8 shows the replanning grasp expectation points after the compensation for the position error only by the finger. The fingers are not beyond the workspace, and the actual point of contact $f i n a l_p = {F_{1}, F_{2}, F_{3}, F_{4}, F_{5}}$ , after calculating the grasp quality, d_Q = −0.2086. Compared with the expected contact point, the grasp quality declines to some extent, but the grasp still meets the force close. At this moment, the decrease in grasp quality is mainly caused by the angle error, and it can be seen that the angle error has a great influence on the grasp quality under the current position error.

In order to evaluate the effect of the cuboid pose error on grasp more comprehensively, the angle of the object is changed with the step δ_θ = 5°, and after each time, the position of the object is changed at the same time with δ_x = 5 mm and δ_y = 5 mm as step. Considering the actual grasp situation, the angle error is limited in $[- 30 °, 30 °]$ . As shown in Figure 9, the quality of the cuboid is influenced by the three factors δ_x, δ_y, and δ_θ, so the quality distribution can be expressed by cube. The deeper color means the better grasp quality, and the transparent cuboid is expressed as d_Q = 0, namely it does not leave the workspace of the finger, without meeting the force closure grasp. As the angle error increases, the number of cubes decreases, the number of transparent cubes increases, and the cube color becomes lighter.

Figure 9.

Grasp quality distribution when grasping cuboid with pose error.

In order to more intuitively compare the influence of the angle error on the grasp quality, the number of the squares count and the corresponding values of ${\bar{d}}_{Q}$ shown in Figure 10 are determined when the object position changes in the plane under different δ_θ values. It can be seen that the tangle error of the object has a great influence on the area and the average grasp quality of the force closure grasp. As the angle error increases, the value of count under the angle error gradually becomes smaller, ${\bar{d}}_{Q}$ gradually increases, and, namely, the average grasp quality declines.

Figure 10.

The number of the count squares and the corresponding values of ${\bar{d}}_{Q}$ with different δ_θ values.

Therefore, for nonrotating bodies, the posture error has a great influence on the grasp stability, and it is necessary to compensate for the posture error. As it is difficult to compensate the angular error through the three-degree-of-freedom (3-DOF) fingers, the application of the robot arm with more freedom is more reasonable. The following conclusions can be drawn from the abovementioned simulation analysis for different shapes and sizes of objects: (1) In the grasp, the position error has a great influence on the rotating body, and the angle error has a great influence on the nonrotating body. The effect of the position error on the nonrotating body is smaller than that of the rotating body. (2) The direction of principal axis in reg_grasp is vertical to the pinching direction of the dexterous hand thumb and the rest of the fingers. (3) In a small error range, the grasp quality has no significant change; when the pose error of the object is small, the use of force collaboration grasp strategy without compensation for the error does not affect the grasp stability.

Compensation strategy for pose error

This article adopts the hierarchical error compensation strategy. After estimating the current pose error of the object by the pre-grasp, the expected position of the finger is adjusted according to the pose error, and the grasp is repeated with the dexterity compensation error of the finger. As the adjustment of the finger is limited, when the error of the object is small, the adjustment of the finger can still compensate for the error and complete the grasp; when the object model or pose error is large, with the kinematics and operability constraints, finger adjustment will not be able to fully compensate for the error. In addition, the dexterous hand fingers have less freedom, so the angle error cannot be adjusted by finger. At this time, the use of the robot arm with more DOFs in the adjustment of the initial grasp location and posture becomes a better solution.

First, the position error function of the object is defined as follows

Δ p = \sqrt{Δ x^{2} + Δ y^{2}}

According to the analysis on the grasp quality distribution, the error can be divided into the following three intervals: $[0, c_{t}^{2}]$ , $(c_{t}^{1}, c_{t}^{2})$ , and $[c_{t}^{2}, c_{t}^{3}]$ , in which $c_{t}^{1} < c_{t}^{2} < c_{t}^{3}$ ; $Δ p > c_{t}^{3}$ means the object is out of the workspace of at least one finger, which is not included in this article. Based on the previous analysis of the grasp quality, the exact range of the error interval can be determined.

The adaptive error compensation strategy for the grasp of object by dexterous hand is shown in Figure 11. The model and pose of the object are known, and the expected contact point is obtained by the grasp planning. When the robotic arm sends the dexterous hand to the specified position, the finger starts to grasp. The entire grasp process is based on the force collaboration grasp strategy. The finger moves to the target position until it comes into contact with the object and maintains a small contact force. When all the expected fingers come into contact with the object, the finger touch and position sensor values are recorded, getting the actual contact point. According to the object model and the expected contact point, the object pose error is calculated.

Figure 11.

Flowchart of adaptive grasp strategy.

For rotating bodies and nonrotating bodies without posture deviation, the position error types are classified. When $0 \leq Δ p \leq c_{t}^{1}$ , the finger is directly switched to the impedance control for grasp; when $c_{t}^{1} < Δ p < c_{t}^{2}$ , only the expected position of the finger is replanned and the finger is adjusted to complete the grasp; when $c_{t}^{2} \leq Δ p \leq c_{t}^{3}$ , the error needs to be firstly compensated by the movement of the robot arm and then the grasp is conducted according to the previous expected position. For nonrotating bodies, we should determine whether there is posture error first. If the posture error is not zero, the robot arm posture should be adjusted so as to make it rotate with the corresponding axis of the object coordinate system with certain angle degree to compensate for the posture error and then regrasp. The error includes only position error, so it can be compensated according to the error compensation method of the rotating body.

The difference between compensating the error by the robot arm and the finger is that the finger compensation error only needs to be replanned; while the robot arm movement needs to be programmed first and then replan the grasp after the movement, which increases the moving time. The adoption of the robot arm for error compensation needs the comprehensive consideration for task requirements for grasp time and stability.

According to the abovementioned adaptive grasp strategies, even if the object has pose error, it can be compensated and the final grasp can be completed as long as the object is still in the dexterous workspace.

Adaptive grasp experiment

The five-fingered robot hand DLR/HIT Hand II is multisensory and highly integrated, and each finger has 3 DOFs and four joints.²³ The tactile sensor is made of pressure conductive rubber and mounted on the curved 3-D convex surface of fingertip. In the grasping experiment, the DLR/HIT Hand II is mounted on the wrist of 7-DOF robot arm, as shown in Figure 12. A set of object prototypes and daily objects are used in the experiment. Figure 13 shows the database of the objects in grasp experiment. All the objects belong to five standard shapes, and there are six objects of different dimensions for each shape.

Figure 12.

DLR/HIT Hand II and 7-DOF robot arm. DOF: degree of freedom.

Figure 13.

Database of the objects in grasp experiment.

In order to verify the validity of the adaptive grasp method proposed in this chapter, a total of 15 objects of four types were selected for the existence of pose error. According to the analysis on the impact of posture error on the stability, $c_{t}^{1} = 10 mm$ , $c_{t}^{2} = 20 mm$ , and $c_{t}^{3} = 30 mm$ . When the angle error is greater than ±5°, it needs adjustment. The step of compensating the movement of the finger is set as 5 mm. For different pose errors, the force collaboration strategy and the adaptive grasp strategy are used to grasp the object and compare the grasp effect.

The coordinate paper is pasted on the experimental platform, and the object is placed at the origin of the coordinate system of the coordinate paper, by which the actual pose error of the object can be accurately recorded, in order to facilitate the comparison in the experiment with the estimated pose error by dexterous hand finger. In the experiment, the object is first placed at the origin of the object coordinate system, and the object is pushed according to the scale of the standard paper to create the expected pose error. When Δx = 15 mm and Δy = −10 mm, the force is only used in conjunction with the grasp strategy without error compensation, and as shown in Figure 14, the thumb contacts with the object first, the index finger and middle finger continue the movement to contact with the object after reaching the expected position (Figure 14(d) and (e)). Although the method can ensure that three fingers are in contact with the object (Figure 14(f)), the force-closure grasp is not formed due to large position error, so the final grasp fails (Figure 14(g)). Because of the simultaneous contact strategy, the whole process of grasping objects is stable.

Figure 14.

The process of grasping the ball by force collaborative grasp strategy when Δx = 15 mm and Δy = −10 mm.

When the adaptive grasp method is used, the error is compensated by finger movement as Δp = 18 and $c_{t}^{1} < Δ p < c_{t}^{2}$ . Figure 15 shows the grasp process, and Figure 16 shows the changes in the position, force, and state functions of each finger in grasp. The thumb and middle finger’s first contact with the object is at 1.2 s, but at this time, the two fingers do not reach the expected position ( $S_{g}^{1} = S_{g}^{3} = S 3$ ), and the finger controller switches to force control to maintain contact (Figure 15(e)); the index finger moves to the expected position without touching the object ( $S_{g}^{2} = S 2$ ), so it continues to do compensation movement, which contacts with the object after two times compensation movement at 2.8 s, when $S_{g}^{2} = S 4$ ( Figure 15(f)). The tactile reading is obtained at this time, and the estimated error of the available object can be calculated $Δ \hat{x} = 12.1 mm$ and $Δ \hat{y} = - 7.8 mm$ . The error is converted to the amount of compensation for each finger through the coordinate transformation; at this time, the finger switches to impedance control and returns to the initial position (Figure 15(g)), beginning to grasp again at 10 s. Due to the deviation between the estimated error and the actual error, the thumb and middle finger move to the expected position when contact with the object ( $S_{g}^{1} = S_{g}^{3} = S 5$ ); the index finger contacts with the object at 12.3 s through a compensation, $S_{g}^{2} = S 4$ , (Figure 15(h) and (i)); when the three fingers are in contact with the object, the finger controller switches to impedance control, successfully completing the grasp (Figure 15(j)).

Figure 15.

The process of grasping the ball by adaptive grasp strategy when Δx = 15 mm and Δy = −10 mm.

Figure 16.

(a) to (c) Fingertip positions, (d) and (e) forces, and (f) state function in finger coordinate when grasping by adaptive grasp strategy (Δx = 15 mm and Δy = −10 mm).

When the error range is further expanded, and limited by finger kinematics and finger maneuverability requirements, the adjustment of the finger cannot completely compensate for the position error of the object. For example, when the object error reaches Δx = 25 mm and Δy = −10 mm, Δp = 27 and $c_{t}^{2} < Δ p < c_{t}^{3}$ ; the grasp fails even if the error is compensated by the finger.

According to the error range of object, this error should be compensated by the movement of the robot arm. Figure 17 shows the grasp process by the use of error compensation method. As the object error is too large, the index finger contacts with the object through the four compensation movements after reaching the expected position (Figure 17(d) and (e)); when all the expected fingers contact with objects, object position deviation can be obtained according to the calculations of fingertips and joint angle readings Δp = 27 and $c_{t}^{2} < Δ p < c_{t}^{3}$ .

Figure 17.

The process of grasping the ball by adaptive grasp strategy when Δx = 25 mm and Δy = −10 mm.

Then, the fingers return to the initial position, according to the object position error, the robot arm moves $Δ \hat{x}$ and $Δ \hat{y}$ (Figure 17(f) and (g)) relative to the object coordinate system; the dexterous hand regrasp after the adjustment of the robot arm, with the expected position unchanged; however, the deviation between the calculated values and the actual ones causes a certain error of the object relative to the finger, so the middle finger reaches the expected position and then performs a compensation movement before it comes into contact with the object (Figure 17(h) and (i)); as this error is less than $c_{t}^{1}$ , the use of only collaboration force grasp method can guarantee the stability. Finally, the hand successfully completes the grasp (Figure 17(j)).

For the long cylinder with h = 200 mm and Φ = 40 mm, when the position error direction is the same as the one that the hand approaching vector, the impact on the grasp stability is large, so a small error compensation range is set for the object separately, and $c_{t}^{1} = 6 mm$ , $c_{t}^{2} = 12 mm$ , and $c_{t}^{3} = 30 mm$ are set according to the simulation results.

Figure 18 shows the progress of grasping cylinder by dexterous hand, the position errors are Δx = −12 mm, Δy = −10 mm, and $c_{t}^{2} < Δ p < c_{t}^{3}$ . Dexterous hand approaching vector is parallel with desktop. When the impedance control is used for grasp, the object is displaced in the direction of the arrow, in an unstable state (Figure 18(d)) during the grasping process, and the object is disengaged from grasp range of the finger, resulting in a failure (Figure 18(e)).

Figure 18.

The process of grasping the cylinder by impedance control (Δx = −12 mm and Δy = −10 mm).

With the adaptive grasp method, the grasp process is shown in Figure 19. The thumb and the index finger contact the object and maintain contact (Figure 19(c)), followed by the middle finger and ring finger (Figure 19(d)). The estimated position error value of the object can be calculated by the tactile sensor $Δ \hat{x} = - 13.5 mm$ and $Δ \hat{y} = - 11.2 mm$ . Then, the finger returns to the initial position, and the robot arm is compensated by the errors $Δ \hat{x}$ and $Δ \hat{y}$ (Figure 19(e) and (f)) and grasps. After the robotic adjustment is completed, the regrasp by dexterous hand consistent with the expected position and the final grasp is successful (Figure 19(g) and (h)).

Figure 19.

The process of grasping the cylinder by adaptive grasp strategy (Δx = −12 mm and Δy = −10 mm).

Finally, the building block in the experiment is taken as an example to explain the adjusting process for the posture error of the object. The building blocks are nonrotating, with the length, width, and height measuring 31, 62, and 100 mm, respectively, which are characterized by lightweight and small friction with the desktop. In the object coordinate system, the object is rotated by 30° relative to the object coordinate system and moves 5 mm and 5 mm along the x-axis and the y-axis, respectively. Figure 20 shows the grasp process of the finger directly using impedance control. When the finger starts to grasp, it can be seen that the middle finger slips out of the edge of the object after the contact, only two fingers contact with the object (Figure 20(c)), and the pose of the object is changed after the completion of grasp, which has been in unstable state actually (Figure 20(d)), so the object is dropped during the lift of arm, and the grasp fails (Figure 20(e)). It can be seen that the posture deviation has a great influence on the stability of the nonrotating body.

Figure 20.

The process of grasping the building block by impedance control (Δx = −5 mm, Δy = 5 mm, and Δθ = 30°.

Figure 21 shows the process of grasping the blocks when the adaptive grasp method is used. Figure 22 shows the fingertip position, fingertip force, and state function in each finger coordinate system during the grasping process. At 1.2 and 1.5 s, the middle finger and the index finger contact with the object and switch the force control to maintain contact force. As the contact force is small, the middle finger does not slip, $S_{g}^{2} =S5$ and $S_{g}^{3} = S 3$ ; the thumb contacts with the object at 3 s after reaching the expected position and two compensation movement, and $S_{g}^{1} =S4$ (Figure 21(c)); when the three fingers are in contact with the object, the current object posture is estimated according to the tactile sensor readings, and the object posture deviation is obtained as $Δ \hat{θ} = 26.5 °$ ; then, all the fingers move to the initial position, the robot arm ends rotate with the plane by $Δ \hat{θ}$ to compensate for the error (Figure 21(d) and (e)); then, the dexterous hand regrasps (Figure 21(f) and (g)) and compared with the expected grasp planning, there is a small error between the position of the finger and the object, but the grasp can be completed successfully without position error compensation. In the second grasp, the thumb contacts with the object still through a compensation movement due to the existence of position error and, finally, completes the grasp successfully (Figure 21(h)).

Figure 21.

The process of grasping the building block by adaptive grasp strategy (Δx = −5 mm, Δy = 5 mm, and Δθ = 30°.

Figure 22.

(a) to (c) Fingertip positions, (d) forces, and (e) state function in finger coordinate when grasping by adaptive grasp strategy (Δx = −5 mm, Δy = 5 mm, and Δθ = 30°.

In addition, in the pose error results estimated by the grasp experiment, there is about 3 mm deviation in position and about 4° deviation in posture between the estimated error obtained by the pre-grasp and the actual pose error of the object due to the sensor error and the control precision. This error will cause the pose error still present when regrasping, but, by error compensation, the actual error has been greatly reduced, so only the force collaboration grasp method can guarantee stable grasp.

The results of the experiment for 15 kinds of objects are shown in Figure 23. For each object, the grasp is conducted for six times in the three error ranges respectively. When the position error is in $(0, c_{t}^{1}]$ , the success rate of grasping by force collaboration grasp method is 95.6% (86/90), which is much higher than 77.8% (70/90) by impedance control method. When the position error is in $(c_{t}^{1}, c_{t}^{2}]$ , the success rate of impedance control is only 38.9% (35/90), and the success rate is 93.3% (84/90) by adjusting the finger by the adaptive grasp method. When the position error is in $(c_{t}^{2}, c_{t}^{3}]$ , the success rate of direct grasp is reduced to 21% (19/90), and the success rate of grasp is maintained at 90% (81/90) after the adaptive grasp method is applied.

Figure 23.

The results of grasp experiments.

Therefore, when the position error is in $(c_{t}^{1}, c_{t}^{3}]$ , the success rate of direct grasp will decline greatly (54/180), and the use of adaptive method makes the success rate maintained more than 90%. The validity of the adaptive grasp method with force collaboration grasp and error compensation is verified by the abovementioned experiment.

Conclusions

In this article, an adaptive grasp strategy based on collaboration control and error estimation compensation is proposed for the influence of object pose uncertainty on grasp, by which the impedance control and force control are switched among different states of the finger to realize the process of the fingers contacting with the object one after another and, meanwhile, with the application of grasp force to ensure the stability of the object during the pre-grasp process. In the error compensation method, the pose error of the object is estimated and analyzed by finger tactile information in pre-grasp. Different methods are used to compensate for the pose error of the object according to different error ranges to reduce the impact of pose error of the object. The results show that the success rate of the object can be improved using the adaptive grasp method in the case of the uncertainty of the object pose.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported in part by the Research Project of the State Key Laboratory of Mechanical System and Vibration (no. MSV201609), the Self-Planned Task of the State Key Laboratory of Robotics and System (no. SKLRS201612B), the National Natural Science Foundation of China (no. 61503095), the Foundation for Innovative Research Groups of the National Natural Science Foundation of China (no. 51521003), and the Fundamental Research Funds for the Central Universities (no. HIT.NSRIF.201640).

References

Nguyen

. Constructing force-closure grasps. Int J Robotics Res 1988; 7(3): 3–16.

Roa

Suárez

. Computation of independent contact regions for grasping 3-d objects. IEEE Trans Robotics 2009; 25(4): 839–850.

Krug

Dimitrov

Charusta

. On the efficient computation of independent contact regions for force closure grasps. In: IEEE/RSJ international conference on intelligent robots and systems (IROS), Taipei, 18–22 October 2010, pp. 586–591.

Zheng

Qian

. Coping with the grasping uncertainties in force-closure analysis. Int J Robotics Res 2005; 24(4): 311–327.

Weisz

Allen

. Pose error robust grasping from contact wrench space metrics. In: IEEE international conference on robotics and automation (ICRA), Saint Paul, USA, 14–18 May 2012, pp. 557–562.

Kim

Iwamoto

Kuffner

. Physically-based grasp quality evaluation under uncertainty. In: IEEE international conference on robotics and automation (ICRA), Saint Paul, USA, 14–18 May 2012, pp. 3258–3263.

Kim

Iwamoto

Kuffner

. Physically based grasp quality evaluation under pose uncertainty. IEEE Trans Robotics 2013; 29(6): 1424–1439.

Chalon

Reinecke

Pfanne

. Online in-hand object localization. In: IEEE/RSJ international conference on intelligent robots and systems, Tokyo, Japan, 3–7 November 2013, pp. 2977–2984.

Zhang

Trinkle

. The application of particle filtering to grasping acquisition with visual occlusion and tactile sensing. In: IEEE international conference on robotics and automation (ICRA), 2012, pp. 3805–3812.

10.

Dragiev

Toussaint

Gienger

. Uncertainty aware grasping and tactile exploration. In: IEEE international conference on robotics and automation (ICRA), Karlsruhe, Germany, 6–10 May 2013, pp. 113–119.

11.

Lee

. Robust grasping for an under-actuated anthropomorphic hand under object position uncertainty. In: 12th IEEE-RAS international conference on humanoid robots (humanoids 2012), Osaka, Japan, 29 November–1 December 2012, pp. 719–725.

12.

Chen

Wimböck

Roa

. An adaptive compliant multi-finger approach-to-grasp strategy for objects with position uncertainties. In: IEEE international conference on robotics and automation (ICRA), Seattle, USA, 26–30 May 2015, pp. 4911–4918.

13.

Hsiao

Chitta

Ciocarlie

. Contact-reactive grasping of objects with partial shape information. In: IEEE/RSJ international conference on intelligent robots and systems (IROS), Taipei, 18–22 October 2010, pp. 1228–1235.

14.

Hsiao

Kaelbling

Lozano-Pérez

. Task-driven tactile exploration. In: Robotics science and systems (RSS), Zaragoza, Spain, 27–30 June 2010, pp. 1–8.

15.

Hsiao

Kaelbling

Lozano-Pérez

. Robust grasping under object pose uncertainty. Auton Robots 2011; 31(2–3): 253–268.

16.

Stulp

Theodorou

Buchli

. Learning to grasp under uncertainty. In: IEEE international conference on robotics and automation (ICRA), Shanghai, China, 15 August 2011, pp. 5703–5708.

17.

Kehoe

Berenson

Goldberg

. Toward cloud-based grasping with uncertainty in shape: estimating lower bounds on achieving force closure with zero-slip push grasps. In: IEEE international conference on robotics and automation (ICRA), Saint Paul, USA, 28 June 2012, pp. 576–583.

18.

Object discovery and grasp detection with a shared convolutional neural network. In: 2016 IEEE international conference on, robotics and automation (ICRA), Stockholm, Sweden, 16–21 May 2016, pp. 2038–2043. IEEE.

19.

Guo

Sun

Kong

. Deep vision networks for real-time robotic grasp detection. Int J Adv Robotic Syst 2016: 14(1): 1729881416682706.

20.

Zhu

Wang

. Synthesis of force-closure grasps on 3-D objects based on the Q distance. IEEE Trans Robotics Autom 2003; 19(4): 669–679.

21.

Ferrari

Canny

. Planning optimal grasps. In: IEEE international conference on robotics and automation (ICRA), Nice, France, 12–14 May 1992, pp. 2290–2295.

22.

Liu

Meusel

. Multisensory five-finger dexterous hand: the DLR/HIT Hand II. In: IEEE/RSJ international conference on intelligent robots and systems, Nice, France, 22–26 September 2008, pp. 3692–3697. IEEE.

23.

Zhang

Fan

. Grasp configurations optimization of dexterous robotic hand based on haptic exploration information. Int J Humanoid Robotics 2017; 14(2): 1750013.