A three-step model for the detection of stable grasp points with machine learning

Abstract

Robotic grasping in dynamic environments is still one of the main challenges in automation tasks. Advances in deep learning methods and computational power suggest that the problem of robotic grasping can be solved by using a huge amount of training data and deep networks. Despite these huge accomplishments, the acceptance and usage in real-world scenarios is still limited. This is mainly due to the fact that the collection of the training data is expensive, and that the trained network is a black box. While the collection of the training data can sometimes be facilitated by carrying it out in simulation, the trained networks, however, remain a black box. In this study, a three-step model is presented that profits both from the advantages of using a simulation approach and deep neural networks to identify and evaluate grasp points. In addition, it even offers an explanation for failed grasp attempts. The first step is to find all grasp points where the gripper can be lowered onto the table without colliding with the object. The second step is to determine, for the grasp points and gripper parameters from the first step, how the object moves while the gripper is closed. Finally, in the third step, for all grasp points from the second step, it is predicted whether the object slips out of the gripper during lifting. By this simplification, it is possible to understand for each grasp point why it is stable and – just as important – why others are unstable or not feasible. All of the models employed in each of the three steps and the resulting Overall Model are evaluated. The predicted grasp points from the Overall Model are compared to the grasp points determined analytically by a force-closure algorithm, to validate the stability of the predicted grasps.

Keywords

Robotic grasping machine learning deep learning optimization simulation force-closure

1. Introduction

Robots have already been successfully established in many areas of industry. Mobile robots are used extensively in the logistics of big warehouses, and cooperating robots are used as adaptive assistance systems for human workers. However, for the picking of unsorted product parts, still no fully automated solution is available on a large scale, since the variability of the working environment and the (delicate) motor interaction therein is subject to higher fluctuations than in the aforementioned areas. Furthermore, robotic grasping involves several individual tasks like object classification, object segmentation, pose estimation and path planning, which are huge research tasks on their own. The intense use of machine learning methods and especially deep learning for image processing have boosted the progress in the domain of robotic grasping. In the following, the subtasks, the applied state-of-the-art machine learning methods and their shortcomings are introduced.

1.1 State of the art

Robotic grasping is the task of finding poses of a robotic end-effector, such that any given object can be picked up successfully [1]. These poses are called grasping poses/postures. Vision-based robotic grasping uses data from vision sensors for the identification of grasp postures. In vision-based robotic grasping, the first task is to process the sensory data like RGB images, depth images, stereo camera images or 3D-point clouds to determine the shape(s) and pose(s) of the perceived object(s). A review on deep learning based object detection can be found in Zhao et al. and Liu et al. [2, 3, 4]. The second step is to determine, from the information obtained from the sensory data, the desired pose of the robot arm with respect to the object such that the object is stable between the gripper tips. The stability of a grasp can be measured by different metrics as compared in Rubert et al. [5]. The third step is to determine a collision-free path from the actual to the desired pose of the robot arm [6, 7]. In each of these steps, an increased usage of machine learning algorithms and especially deep learning with convolutional layers can be observed. An extensive review of deep learning methods used for robotic grasp detection can be found in [1]. Mainly two approaches are distinguished in robotic grasping [1, 8]: The first approach determines from the sensory image(s) the optimal grasp pose of the robot arm either analytically with force and form-closure objectives [9] or data-driven. A grasp planner determines subsequently the motor commands to drive the robot arm into the optimal grasp pose. In the second approach, the visuomotor controller is directly learned from the sensory image(s) as one monolithic model.

In the first approach, machine learning algorithms are primary used for the detection of objects on images and for assessing grasp configurations. As an example, Jiang et al. [10] learned a rectangle representation of a grasp from hand-engineered features on RGB images, where the quality of the grasp was assessed with a support vector machine. Lenz et al. [11] have extended this approach and substituted the hand-engineered features as well as the quality determination of the grasp rectangles with deep learning models. Ogas et al. [12] have used a convolutional neural network to detect a given production piece on images of a monocular camera. For the determination of stable grasp points on the detected object shape they used the Hough transformation [13] and applied force closure to the extracted line-segments.

State-of-the-art grasping methods belong to the second approach and apply deep learning [14, 15, 16, 17] to train a visuomotor controller. Morrison et al. [16] predict for each pixel of a single depth image with a convolutional neural network the gripper width, gripper angle and grasp quality. Levine et al. [15] trained a deep neural network that predicts from camera images and a robot motor command the grasp success probability. An optimization algorithm is applied to determine the motor command that maximizes the grasp success probability. In order to train this network, 800 000 training data points had to be recorded. Like for most models that involve deep networks, the amount of data needed, in order to train the network successfully, is immense. Alternative approaches use a physics simulation of a robot arm and the objects to obtain the training data. The reality gap, which simply speaking is the difference between the simulation and the real world, is overcome by the principle of domain randomization [18]. The prevalent datasets that are used to train and evaluate convolutional neural networks for robotic grasp models are the Cornell [11] and Jacquard [19] datasets. Both datasets consist of RGB-D images of an object and grasp rectangles which represent the possible projection areas for a parallel-jaw gripper. The Jacquard dataset is completely generated in a simulation.

1.2 Problem statement and proposal

One problem that all of the state-of-the-art approaches share, is that the evaluation of a robotic grasp is only given in terms of grasp success probabilities. More precisely, these methods learn a mapping which assigns to each pixel on an object image a probability for a successful grasp. This assignment is done via a CNN, for which it is not clear which human-readable features lead to the prediction. This problem is known under the term “black box” and the solution strategies are summarized under the term explainable AI (XAI) [20, 21, 22]. As the decisions that are met by the neural networks are difficult to comprehend, the application in autonomous industrial environments is hardly possible.

Figure 1.

Flowchart of the proposed three-step model.

The contribution of this study is a three-step model to identify and evaluate grasp points. Therefore, the problem of robotic grasping is reformulated as finding the grasp postures on an image where the gripper can be lowered, closed and lifted. In contrast to the direct mapping from an image to a grasp success probability, the user of the model can identify the reasons of failed grasp attempts and, in addition, assess the quality of the predictions in each individual step. This assessment is possible since the user can at least partly evaluate the results in the visual domain based on his own experience. When the results coincide with his expectations, the trust of the user into the model increases. Furthermore, the proposed three-step model profits both from the advantages of using a simulation approach and deep neural networks. Figure 1 shows a flowchart of the proposed model. In a first step a predefined number of random grasp candidates is generated on an object image. All grasp points, where the gripper can be lowered on the table while enclosing an object part, are determined by the Lowering Model. Therefore a neural network (NN) is trained to generate an image of the gripper for parameters $\alpha$ (gripper rotation) and $s$ (gripper opening width). A Pareto based genetic algorithm is used to determine whether a pair of parameters $(\alpha,s)$ exists such that the gripper can be lowered without collision. In the second step, a convolutional neural network (CNN) is used which predicts the physical interaction between the gripper and the object in the image domain (Closing Model). Finally, a model is learned that predicts the grasp points where the object is liftable (Lifting Model). This model is a simple CNN that classifies the images predicted by the Closing Model into liftable, when the object remains between the gripper tips, and not liftable, when the object slips out of the gripper during lifting. An object is graspable if there is at least one grasp point where the gripper can be lowered, closed and lifted. When a grasp attempt fails, the explanation can be directly determined from the responses of the three steps. When a grasp attempt fails after the first step, the region of the object targeted for grasping was too large for the gripper. When a grasp attempt fails after the second step, the object was pushed out during the closing movement of the gripper due to the local object geometry. Finally, when the grasp attempt failed after the third step, the grasp point was not close enough to the centre of mass of the object.

Another advantage of this new approach over the existing methods is, that the dynamics between the gripper and the object is taken directly into account in the Closing Model. To the best of our knowledge there are no grasping models where the object movement due to the closing movement of the gripper is directly predicted. There is research on predicting object movements in pushing tasks directly in the image domain [23, 24, 25], but not in the realm of grasping. In [26, 27] the effect of a force acting on an object in an image is predicted in terms of a vector field but without the prediction in the image domain.

1.3 Outline

In the following, first the materials and methods, i.e. the underlying robot simulation and used object database, are described. Then a description of the algorithm for the determination of stable grasp points is given. Afterwards, the training and evaluation of the Lowering, Closing and Lifting Model are explained. Finally, the performance of the chained Overall Model is presented in detail, and the predicted grasp points from the Overall Model are compared to the grasp points from the analytical force closure algorithm to show and verify that the determined grasp points are stable grasp points.

2. Robot simulation and object database

The gripper under consideration is a parallel-jaw gripper with the gripper opening width $s$ and the orientation $\alpha$ measured with respect to the gripper coordinate frame (see Fig. 3). For the collection of the training data, the robot simulator V-REP [29] is used with the integrated bullet physics engine [30]. The parallel-jaw gripper has a maximal opening width of 5 cm. Since the focus of this study is the interaction between the gripper and the object, the inverse kinematics of an attached robot arm is supposed to be given. Therefore, the robot arm is not considered in the simulation. The object database consists of prismatic object shapes which are generated from binary (household) item images, by triangulating the boundaries of the edge-filtered images and extruding the resulting faces in 3D. By this simplification, the objects considered here all have the same height at each location and are assumed to be rigid with a uniform mass distribution and a constant friction coefficient. To increase the variability of the object shapes, 10 000 objects were created from bitmap images of household items and artificial random polygon shapes in three different sizes. On the left image of Fig. 2, a hammer that was built this way is positioned on a table. The prismatic objects are placed on a table, one object at a time. Two orthographic cameras are installed in this setup: the ceiling camera and the gripper camera (Fig. 2). The ceiling camera is centred above the table and has a resolution of 512 $\times$ 512 pixels. The maximal gripper opening width of 5 cm corresponds to 51 pixels in this image. The gripper camera is centred between the gripper tips. The right image of Fig. 2 shows an example of the image obtained by the gripper camera. The gripper camera has a resolution of 1024 $\times$ 1024 pixels.

Figure 2.

Left: V-REP simulation of the gripper in pre-grasp position. The position and orientation of the world coordinate system is indicated by the grey circle and the coordinate axes. The orthographic gripper camera and the orthographic ceiling camera, centred above the table are shown as grey boxes. Right: Image from the orthographic gripper camera in the pre-grasp position. Adapted from [28].

Figure 3.

Parallel-jaw gripper with gripper parameters $\alpha$ and $s$ .

A grasp is executed perpendicular to the table surface at the world coordinates $(w_{x},w_{y},w_{z})$ with gripper parameters ( $\alpha$ , $s$ ). For prismatic object shapes, the z-coordinate of a grasp $w_{z}$ is constant. For the following sections, these definitions are made: A grasp candidate is a pixel location ( $x$ , $y$ ) in an image, which is evaluated for being a graspable object location or not. A grasp point is a verified grasp candidate (thus, also a pixel location). A grasp posture is the parameter tuple consisting of the grasp point and the corresponding gripper parameters: $(\alpha,s,x,y)$ . The obtained pixel coordinates ( $x$ , $y$ ) of the grasp points can be transformed to the corresponding world coordinates $(w_{x},w_{y})$ by the known intrinsic parameters of the ceiling camera.

3. Three-step model for stable grasp point detection

[ht] DetermineStableGrasps.Input: Binary orthographic image of an object. Output: A set of grasp postures and causes for failure. [1] $i m a g e, g r a s p C a n d i d a t e s$ $g r a s p P o s t u r e s, g r a s p R e a s o n i n g$ DetermineStableGrasps $i m a g e$ 1 internal structure of $graspCandidates:[x_{i},y_{i}]$ $graspCandidates\leftarrow\textsc{randomGraspCandidates}(image,n)$ generate n random grasp candidates $i=1:n$ for all grasp candidates… $imagePatch\leftarrow\textsc{getImagePatch}(image,graspCandidates[i])$ get image patch 2 internal structure of $relGraspPosture:[\alpha,s,dx,dy]$ $lowerable,relGraspPosture\leftarrow\textsc{LoweringModel}(imagePatch)$ 2 transform the grasp candidate by dx,dy to obtain the x,y pixel coordinates of the grasp posture 2 internal structure of $graspPostures[i]:[\alpha_{i},s_{i},x_{i},y_{i}]$ $graspPostures[i]\leftarrow\textsc{transform}(graspCandidates[i],relGraspPosture)$ $l o w e r a b l e$ if lowering was successful proceed 3 transform the object image with the grasp posture to gripperImage and objectImage $gripperImage,objectImage\leftarrow\textsc{imageTransform}(image,graspPostures[% i])$ 3 obtain gripperImage and objectImage after closing the gripper: gripperImageCl, objectImageCl $enclosable,gripperImageCl,objectImageCl\leftarrow\textsc{ClosingModel}(% gripperImage,objectImage)$ $e n c l o s a b l e$ if object remained between the gripper proceed $liftable\leftarrow\textsc{LiftingModel}(gripperImageCl,objectImageCl)$ $l i f t a b l e$ if lifting was successful… $graspReasoning[i]=0$ …grasp was successful $graspReasoning[i]=3$ grasp failed at Lifting step

$graspReasoning[i]=2$ grasp failed at Closing step $graspReasoning[i]=1$ grasp failed at Lowering step

$g r a s p P o s t u r e s, g r a s p R e a s o n i n g$ all graspPostures, where graspReasoning is 0, are valid

In this section the proposed algorithm is described. The pseudo-code is given in Algorithm 3. This algorithm is used to determine stable grasp postures ( $\alpha_{i}$ , $s_{i}$ , $x_{i}$ , $y_{i}$ ) from an object image via three steps ( $i$ is the index for the finally generated postures, also including the unsuccessful ones). The input of the algorithm is a binary orthographic image of the object, taken from the ceiling camera. In this object image, $n$ grasp candidates are randomly chosen. For each grasp candidate, an image patch is extracted. The image patch is the input of the Lowering Model. The Lowering Model determines if the gripper can be lowered and returns a relative grasp posture ( $\alpha$ , $s$ , $d x$ , $d y$ ). This relative grasp posture describes the gripper parameters $\alpha$ , $s$ and the necessary relative translation $d x$ , $d y$ of the original grasp candidate for successfully lowering the gripper. The grasp posture is determined by applying the relative translation $d x$ , $d y$ to the grasp candidate (Algorithm 3, line 10). If the gripper can be lowered, the grasp posture is used to transform the object image from the ceiling camera to an image of the gripper camera. The gripper parameters $s,\alpha$ are used to determine the corresponding gripper image (Algorithm 3, line 12). The transformed gripper image and transformed object image are the inputs of the Closing Model. The Closing Model predicts if the object is pushed out during closing the gripper and returns the according gripper image and object image after the closing movement. If the Closing Model predicts that the object remains between the gripper tips, the predicted gripper image and object image are fed into the Lifting Model. The Lifting Model classifies this input into liftable vs. not liftable. If the Lifting Model classifies the input as liftable, a successful grasp posture has been found. If one of the three steps fails, the reason is saved in the grasp reasoning array (Algorithm 3, line 19, 21, 23). The overall result of the algorithm is a set of grasp postures and, for every rejected grasp candidate, a causal attribution why this grasp candidate is not suited for successful grasping.

Thus, there are three possible causes why a chosen grasp candidate does not lead to a stable grasp. The first reason is that there is no gripper parameter configuration where the gripper can be lowered onto the table without colliding with the object. This possibly means that the object is too big for the used gripper at the grasp location. The second reason is that for the given grasp candidate, the object was pushed out of the gripper during closing due to the local shape of the object. Finally, the last reason for a failed grasp is that the object falls out of the gripper during lifting, because of the distance between the grasp candidate and the centre of mass of the object. The independent evaluation of these three steps allows the attribution of grasping failures to distinct causes in contrast to competing holistic approaches.

Figure 19 shows a sample workflow for one object image. The red dots on the object indicate the randomly chosen grasp candidates, the green dots the grasp candidates that have been translated by $d x$ , $d y$ as a result of the Lowering Model. For each positive grasp candidate of the Lowering Model, the following transformations are successively applied:

1.
translate the object on the orthographic camera image such that the grasp candidate (translated by $(dx,dy)$ ) is at the centre of the image
2.
rotate the object around the centre by $-\alpha$
3.
predict the gripper projection for ( $\alpha_{i},s_{i}$ ) $=$ (0, $s_{\textit{opt}}$ ) and add it to the orthographic camera image
4.
scale the image by a factor of 5.3 to make the transformation from the ceiling camera to the gripper camera
5.
centre crop a region of 250 $\times$ 250 pixels and rescale the image to 64 $\times$ 64 pixels
6.
separate the image into the gripper and object channels by pixel intensities

The last step yields the input images of the Closing Model.

In the following sections, the implementation of the Lowering, Closing and Lifting Model are described in detail, before the Overall Model is evaluated in an end-to-end fashion.
3.1 Step one: The Lowering Model

The first step of grasping is to lower the gripper tips on the table without colliding with the object. Therefore, a binary orthographic image is taken from the ceiling camera. In this binary image, grasp candidate are chosen randomly on the object shape. Since the size of the objects is restricted, the maximum number of grasp candidate is limited to 30. Furthermore, to guarantee an even distribution of the points, the grasp candidate are forced to have a distance of at minimum 5 pixels which corresponds to 1 cm. For each grasp candidate an image patch is cropped with a size of 51 $\times$ 51 pixels such that the grasp candidate is at the centre of the image patch. To determine if there is a collision between the gripper tips for a given gripper configuration ( $\alpha_{i}$ , $s_{i}$ ) and an image patch, the image of the gripper tip projection for the parameter configuration has to be determined. It is performed through an extreme learning machine (ELM, [31]).

Figure 4.

Structure of the extreme learning machine for the gripper image prediction.

3.1.1 Predicting the gripper image

An extreme learning machine (ELM) is a single-hidden-layer feedforward neural network. By using only one hidden layer and keeping the weights between input and hidden layer fixed, the network training can be formulated as a simple matrix problem, in contrast to the backpropagation algorithm for networks with more than one hidden layer. This makes the networks’ training time extremely fast. For a detailed description, it is referred to the original work of Huang [31]. In the case of the gripper image prediction, the input of the ELM is the gripper parameter tuple ( $\alpha_{i},s_{i}$ ). The output of the ELM is the vectorized image of the gripper projection. The number of output units is therefore determined by the size of the image as depicted in Fig. 4. For training data generation, $\alpha_{i}$ is varied in steps of 2 ${}^{\circ}$ in [0 ${}^{\circ}$ , …, 18 ${}^{\circ}$ ] and $s_{i}$ in steps of two pixels in [0px, …, 30px], leading to a total number of $N=$ 1456 patterns. For simplicity, the corresponding 51 $\times$ 51 image of the gripper projection is computed by a simple rotation and translation operation on the two projection rectangles with the centre pixel as the centre of rotation. These images can also be obtained by simulation, where the motor commands are set such that $\alpha$ and $s$ have the target values, taking a snapshot from the gripper camera. This would lead to the same results but would be more time-consuming. As activation function $f_{a}(.)=\tanh(.)$ is chosen. The initial weights $w_{ij}$ are randomly drawn from a uniform distribution in the interval [ $-$ 0.95, 0.95]. Let $\textbf{t}_{i}\in\!R^{L}$ denote the vectorized gripper image, with $L=2601,i\in 1,\ldots,N$ . Then the training data for the ELM is a $N\times L+2$ matrix, with rows ( $\alpha_{i}$ , $s_{i}$ , $\textbf{t}_{i}$ ). To determine the number of hidden units $\hat{N}_{\textit{opt}}$ , 5-fold cross-validation [32] is used and yields $\hat{N}_{\textit{opt}}=70$ . Let the thresholded output be denoted by $o_{i}^{t}$ . Then the error used for cross-validation between the desired and the predicted image is computed as

$\displaystyle d(o_{i}^{t},t_{i})=\sum_{k=1}^{L}|o_{i}^{t}(k)-t_{i}(k)|,$ (1)

which is the number of wrongly predicted pixels. The overall error for one validation set is then the average distance over the total number of patterns (S)

$\displaystyle\textit{Err}=\frac{\sum_{i=1}^{S}d(o_{i}^{t},t_{i})}{S}.$ (2)

The performance of the ELM is also tested on novel data. Since $\alpha_{i}$ and $s_{i}$ have been varied in steps of two for training data generation, every odd parameter configuration in between, e.g. $\alpha=3^{\circ}$ and $s=7$ , is unknown to the model. Therefore, the test set is the dataset consisting of tuples ( $\alpha_{i}$ , $s_{i}$ , $g_{i}$ ) with odd $\alpha$ and $s$ . The error on the whole test set, consisting of 1350 patterns, is $\textit{Err}=$ 22.1. This means that on average 22.1px are predicted wrongly. The maximal number of wrongly predicted pixels on the test dataset is 26px. Figure 5 shows the ground truth, the predicted gripper image and the difference image for the maximal number of wrongly predicted pixels. Also visual inspection shows that the error is small, therefore one can conclude that the goal of the gripper prediction model is achieved.

Figure 5.

Gripper image of maximal error (26px) in test patterns for $\alpha=101^{\circ}$ and $s=29$ . Left: computed gripper image, centre: predicted image, right: difference image between computed and predicted image.

3.1.2 The optimization problem

Figure 6.

Schematic view of the optimization problem of lowering the gripper.

After training the ELM and obtaining the image patches for an object with the grasp candidate at the centre, the task of finding the collision-free gripper parameters can be formulated as optimization problem as depicted in Fig. 6: for each image patch with the grasp candidate at the centre, find the parameters ( $\alpha_{i}$ , $s_{i}$ ) such that the predicted gripper image from the ELM and the object part on the image have no intersection. Let $G_{i}(\alpha_{i},s_{i})$ denote the set of pixels where the gripper image is 1 and $I_{n}$ the set of pixels where the chosen image patch $n$ is 1. Then all optimal values for $\alpha_{i}$ and $s_{i}$ satisfy the condition that the intersecting set of $G_{i}(\alpha_{i},s_{i})$ and $I_{n}$ is empty: $G_{i}(\alpha_{i},s_{i})\cap I_{n}=\varnothing$ , which is the same as demanding that the cardinal number $f_{c}(\alpha_{i},s_{i})=|{G_{i}(\alpha_{i},s_{i})\cap I_{n}}|$ has to be zero. $f_{c}$ is the cost function of the optimization problem, for now depending only on the parameters $\alpha_{i}$ and $s_{i}$ . Furthermore, the gripper is supposed to approach the object as closely as possible, that is, $s$ has to be chosen as small as possible. This is expressed by an additional cost function $f_{s}=s$ . Since the grasp candidates have been chosen randomly, it often occurs that the centre of the image patch does not coincide with the centre of the object shape and prevents the algorithm from finding a solution for $\alpha_{i}$ and $s_{i}$ . Therefore, small translations $d x$ and $d y$ of the image patch within [ $-$ 6px, 6px] are allowed for. The effect is shown in the top row of Fig. 7. In order to prevent the algorithm from finding solutions where the gripper is lowered next to the object without enclosing the object, it is demanded by a constraint that in the centre of the image an object part needs to be located. An example of a result without and with constraint is given in the bottom row of Fig. 7.

Figure 7.

Top: Illustration of the effect of the translation $(dx,dy)$ , left: without translation the gripper cannot be lowered since the maximum gripper opening width is reached, right: with translation the gripper can be lowered. Bottom: Effect of the constraint that an object part is supposed to be in the centre of the image. Left: Possible solution without the constraint, that an object part has to be between the gripper tips, right: solution with the constraint. The according grasp candidates are marked as red dots.

Figure 8.

Example for the approximation of the Pareto front by GDE3, NSGA-II and NSGA-III for the shown image patch. The image for the gripper tips determined from the last population of GDE3 is exemplary shown.

In summary, the Lowering Model is a bi-objective optimization problem with the two objective functions $f_{c}(\alpha_{i},s_{i},d_{x},d_{y})$ and $f_{s}$ , depending on four parameters, and one constraint. The valid intervals for the parameters are $s\in$ [0, 30px], $\alpha\in$ [0 ${}^{\circ}$ , 180 ${}^{\circ}$ ] and $dx,dy\in$ [ $-$ 6px, 6px]. The region of interest on the Pareto front is defined by $f_{c}=0$ , expressing that the gripper should not collide with the object. In order to make the Lowering Model less sensitive to noise on the image, especially the object boundary, all image patches are lowpass filtered with a Gaussian ( $\sigma=$ 1px) and thresholded to a binary image again before the optimization algorithm is applied.

In the following, three different popular Pareto multiobjective optimization algorithms [33, 34] are compared in order to determine the solutions of the Lowering Model: generalized differential evolution 3 (GDE3) [35], non-dominated sorting genetic algorithm II (NSGA-II) [36] and non-dominated sorting genetic algorithm III (NSGA-III) [37]. Starting from an inital generation of vectors in the domain of the problem, the goal of all algorithms is to iteratively find the generation members that approximate the Pareto front [38] in an evenly distributed fashion. The Pareto front is the unknown set of solutions of the optimization problem, where the goal of one objective function can only be improved by worsening the result of the other objective [33]. The compared algorithms can be separated into two groups: while the goal of GDE3 and NSGA-II is to approximate the Pareto front [33] in a uniform fashion, NSGA-III uses predefined reference vectors as additional information, in order to force the focus on the most interesting part of the approximated Pareto front. The Matlab toolbox PlatEMO [39] and its implementation of these algorithms was used to obtain the presented results. The parameter configurations of all algorithms are given by: population size $N=150$ , number of objectives $M=2$ , number of variables $D=4$ . At maximum 10 050 function evaluations are performed for each algorithm. Since a valid solution of the Lowering Model must satisfy that $f_{c}=0$ , the optimal solution from the final generation is determined by the strict dominance [38]: The member $m$ of a population is said to be strictly dominated by another member $m*$ if the values of both objective functions $f_{c}$ and $f_{s}$ are strictly smaller for $m*$ . If there is no member in the population that dominates a given member $m*$ , the member $m*$ is said to be non-dominated. The evaluation steps that are performed to obtain the optimal solution with this principle are:

from the last population obtain the solutions with $f_{c}=0$

from the solutions of step 1, find the non-dominated solution with minimal $f_{s}$

Table 1

Confusion matrices of the optimizers for 343 image patches

GDE3			NSGA-II			NSGA-III
$n=$ 343	Act. Pos.	Act. Neg.	$n=$ 343	Act. Pos.	Act. Neg.	$n=$ 343	Act. Pos.	Act. Neg.
Pred. Pos.	203	0	Pred. Pos.	205	0	Pred. Pos.	204	0
Pred. Neg.	3	137	Pred. Neg.	1	137	Pred. Neg.	2	137

Table 2

Confusion matrices of the optimizers for 343 noisy image patches

GDE3			NSGA-II			NSGA-III
$n=$ 343	Act. Pos.	Act. Neg.	$n=$ 343	Act. Pos.	Act. Neg.	$n=$ 343	Act. Pos.	Act. Neg.
Pred. Pos.	206	24	Pred. Pos.	206	23	Pred. Pos.	206	22
Pred. Neg.	0	113	Pred. Neg.	0	114	Pred. Neg.	0	115

This procedure also ensures that the minimal possible value for the gripper opening width $s$ is found. As the minimal value of the gripper opening width $s$ changes with the currently processed image patch, a general preference region for $s$ cannot be given. In order to pull the solution to the smallest $s$ value in NSGA-III, the preference values for $s$ have therefore been set to values between zero and one. Figure 8 shows an example of the approximated Pareto front, found by GDE3, NSGA-II and NSGA-III for the shown image patch. Exemplary the gripper projection of the optimal solution found by the described evaluation steps on the last population of GDE3 is shown. The performance of all optimizers is measured as the prediction error on 343 image patches. The set of image patches can be divided into two disjunctive sets. The first contains all patches where at least one parameter tuple exists for which there is no intersection between gripper and object ( $f_{c}=0$ ), and for which the corresponding gripper opening width ( $s_{i}$ ) is greater than 0. The second set contains all image patches with either collision between gripper and object ( $f_{c}>0$ ) or without collision, but with a closed gripper ( $f_{c}=0$ and $s_{i}=0$ ). Image patches belonging to the first category are labelled positive because the gripper can be lowered in an open position without collision, otherwise they are labelled negative.

To generate ground truth data for the assessment of the prediction quality of the Lowering Model, systematic search is carried out. For each image patch, all parameter configurations of the gripper are tested within a dense search grid whether they lead to a collision or not.1

The confusion matrices of the optimizers (obtained after two runs for each optimizer) on all 343 image patches are shown in Table 1. All confusion matrices are assessed with Matthews correlation coefficient (MCC) [40]. All three optimizers perform very well, with $\textit{MCC}_{\textit{GDE}3}=$ 0.98, $\textit{MCC}_{\textit{NSGA-II}}=$ 0.99 and $\textit{MCC}_{\textit{NSGA-III}}=$ 0.99. The difference is that NSGA-II has only one false negative, while GDE3 and NSGA-II have three and two false negatives, respectively. These results show that NSGA-II is the best optimizer for the Lowering Model, although only by a very slight margin. It is not observed that the optimizers produce solutions that lie outside the domain of the four parameters. This is due to the chosen constraint, which limits the translation $d x, d y$ , and the fact that the gripper opening width $s$ is also minimized. The number of runs per optimizer was limited to two to keep the overall number of evaluations and the associated computational costs within feasible limits. One has to be aware that using different implementations of these algorithms like in the popular MOEA Framework [41], Platypus [42] or jMetal-Java [43] may yield different results, as was shown in a thorough analysis by Rostami [44]. According to the authors of this study this is mainly due to different interpretations of the same algorithm, in case some important information on the details are missing in the original description of the algorithm. The second reason are inaccuracies for the sake of more elegant software design.

Another important note for assessing the results is the No Free Lunch Theorem [45], which states that the performance of all optimizers is equally good if averaged over all possible given problems. More precisely, there needs to be a specific reason why an optimizer performs especially well on a problem [46]. In the optimization problem discussed here, a solution is only valid, when one objective function is zero ( $f_{c}=0$ ). In contrast to GDE3, NSGA-II and NSGA-III take the elitist approach for the selection criterion of the next population. Solutions with $f_{c}=0$ are therefore preserved when they are found, without sacrificing them for the diversity of the population. It is hypothesized that this is the reason why the NSGA-II/III approach is more appropriate for the presented optimization problem.

Object shapes distorted by noise

In order to determine the impact of noise on the performance of the optimizers, noisy image patches are generated. The noise generation process is intended to simulate errors in the identification of the object borders during image segmentation, e.g. caused by changing lighting conditions, shadows or partly reflecting surfaces. For this purpose, dilation, erosion and closing operations [47] are used to morph the original object shape. This leads to distorted object boundaries: the object appears thinner or slightly thicker than the original object. For the dilation, erosion and closing operations, a 3 $\times$ 3 filter matrix was used. The following steps were carried out to transform each original image patch to its noisy variant:

at each pixel position of the original image patch perform a dilation or erosion with a probability of 0.5

perform a closing operation on the image patch obtained from step 1

perform an erosion on the image patch obtained from step 2 to further deform the object

Figure 9.

Example of a morphed image.

An example of the effect, obtained by this procedure, is depicted in Fig. 9. The confusion matrices of the optimizers on the noisy data are shown in Table 2. As can be seen in comparison to the results on the undistorted image patches in Table 1, on average 23 image patches are false positives. The reason for this classification result is that at object regions, where the object was made thinner through image morphing, the optimizer finds a solution that is not valid. This is also the explanation why the false negative results disappear. The resulting MCC values are: $\textit{MCC}_{\textit{GDE}3}=$ 0.86, $\textit{MCC}_{\textit{NSGA-II}}=$ 0.87 and $\textit{MCC}_{\textit{NSGA-III}}=$ 0.87. In conclusion, all of the three optimizers appear to be equally sensitive to the influence of noise on the object segmentation process.

3.1.3 Computational complexity

The overall computational complexity of the Lowering Model in the inference step is determined by the computational complexity of the ELM and the computational complexity of the NSGA-II optimizer (which is chosen here because it shows the best performance). Since an ELM consists of two layers (not counting the input), the complexity is given by $O(L^{2})$ , where $L$ is the output size, i.e. the number of pixels in the predicted gripper image. We assume that the size of the hidden layer is proportional to $L$ , therefore the resulting complexity is $O(L^{2})$ [48]. The computational complexity of NSGA-II for one generation is given by $O(MN^{2})$ , where $M$ is the number of objectives and $N$ the population size [36]. In our application, $M$ is fixed. Combining NSGA-II (ignoring constant $M$ ) and the ELM, the overall complexity for each generation in the optimization process is $O(N^{2}+NL^{2})$ . The term $NL^{2}$ arises because the ELM has to be evaluated for each of the $N$ population members.

3.2 Training data generation for the Closing and Lifting Model

The positive grasp postures ( $\alpha_{i}$ , $s_{i}$ , $x_{i}$ , $y_{i}$ ) found by the Lowering Model are used to create the training data for the Closing Model and Lifting Model. Therefore the coordinates ( $x_{i}$ , $y_{i}$ ) of the grasp postures are transformed to the world coordinate system, yielding ( $w_{x},w_{y},w_{z}$ ). The gripper is moved to ( $w_{x},w_{y},w_{z}$ ) and the gripper is rotated by $\alpha_{i}$ . The gripper opening width is set to $s_{i}$ (see Fig. 10a). Then the gripper is lowered and closed in a uniform way. During the closing movement of the tips, an image sequence is recorded with the gripper camera. An image sequence obtained by this procedure is shown in Fig. 10c. If the object remained between the tips, the gripper is lifted (see Fig. 10b). If the object remains between the gripper tips after lifting, the ground truth of the grasp candidate is true for the closing and lifting process.

The orthographic gripper camera used for generating the data for the Closing and Lifting Model is fixed to the gripper. Therefore, the gripper has the 0-orientation in all image sequences. The resolution of the camera is 1024 $\times$ 1024. To reduce the image size, a region of 250 $\times$ 250 pixels is cropped at the centre and rescaled to 64 $\times$ 64 pixels. When the object is pushed out and the gripper opening width is 0 after the closing movement, the grasp candidate is said to be negative, otherwise positive. An example for a negative and positive grasp trial is shown in Fig. 10a and b. Figure 10c shows an example of an image sequence obtained by this method. The images are inverted and normalized, so that the pixels of the image that represent the gripper have a value of 0.7, the pixels that represent the object have a value of 1.0, and the background is 0. The different pixel intensities are used later to separate the images into two channels, one for the object and one for the gripper.

Figure 10.

a. Example of a grasp sequence in V-REP for a negative grasp candidate. b. Example of a grasp sequence in V-REP for a positive grasp candidate. c. Example of an image sequence obtained in simulation. First published in [28].

Figure 11.

CNN architecture used for the Closing Model. Net2Vis was used for visualization [49]. First published in [28].

3.3 Step two: The Closing Model

The goal of the Closing Model is to predict the physical interaction between object and gripper in the image domain while closing the gripper. The input of the Closing Model is an image of the object and gripper after lowering the gripper. The output is the image after closing the gripper and the resulting gripper opening width $s$ . If the resulting $s$ is greater than 0, the object remained between the gripper tips, otherwise the object was pushed out. A thorough analysis of different CNN architectures to learn the Closing Model can be found in the authors’ previous work [28]. The major findings are briefly described here to provide the context for the Overall Model in the end-to-end application. The first finding is that better results are obtained when the object and gripper are presented to the network in two separated channels. The separation also allows for a better evaluation of the predicted gripper opening width $s$ . Therefore, the input and output of the Closing Model have two channels. The CNN with the best performance is adapted from [50] and consists of four convolutional layers followed by four deconvolutional layers. It is depicted in Fig. 11. The kernel-sizes are chosen as (8 $\times$ 8), (6 $\times$ 6), (6 $\times$ 6), (4 $\times$ 4) and in reverse order for the deconvolutional part with a stride of 2 as in [50]. The number of filters are [64,128,128,128] for the convolutional layers and in reverse order for the deconvolutional layers. Each convolutional layer has RELU as an activation function.

Table 3
Confusion matrices of the Closing and Lifting Model

Closing Model			Lifting Model
$n=$ 5000	Act. Pos.	Act. Neg.	$n=$ 2000	Act. Pos.	Act. Neg.
Pred. Pos.	2384	23	Pred. Pos.	930	53
Pred. Neg.	116	2477	Pred. Neg.	70	947

Table 4

Confusion matrix of the Overall Model – individual results

$n=$ 16 000	Actual positive	Actual negative
Predicted positive	$TP_{Li\_T}\textbf{(1970)}$	$FP_{Li\_T}\textbf{(36)}+FP_{Li\_F}\textbf{(16)}$
Predicted negative	$FN_{Li\_T}\textbf{(2)}+FN_{Lo}\textbf{(5)}+FN_{Cl}\textbf{(604)}$	$TN_{Lo}\textbf{(8000)}+TN_{Cl}\textbf{(3557)}+TN_{Li\_F}\textbf{(424)}+TN_{Li% \_T}\textbf{(1386)}$

The method and the evaluation are described in more detail in the following: The CNN model is trained with 15 000 positive and 15 000 negative image pairs and evaluated on 2500 positive and 2500 negative image pairs. The results are evaluated with respect to two measures. The first measure is the Intersection over Union (IOU) [51] value for determining how well the object and gripper movements are learned. The second measure is the predicted gripper opening width in terms of a confusion matrix: if the gripper was not closed in prediction and physics simulation the result is true positive, if the gripper was closed in prediction and physics simulation the result is true negative. False positives and false negatives are defined analogously. This measure determines how good the grasp candidates are classified. However, since the resulting image of the Closing Model is a necessary input for the Lifting Model, the IOU values are more important in this analysis. The resulting confusion matrix values for 2500 positive and 2500 negative image pairs (test data) are given in Table 3 (left), yielding a Matthews correlation coefficient of 0.945. The IOU value is 0.982 (sum over gripper and object channel). This means that the prediction of the gripper and object movement as well as the classification results of the Closing Model are good. Figure 12 shows an example for a true negative (top row) and a true positive (bottom row) grasp candidate.

3.3.1 Computational complexity

The computational complexity of an inference step in a full CNN, as used in the Closing Model, is given by [2]:

$\displaystyle O\left(\sum_{l=1}^{d}n_{l-1}\cdot s_{l}^{2}\cdot n_{l}\cdot m_{l% }^{2}\right)$ (3)

where $d$ is the number of convolutional layers, $l$ the index of the convolutional layer, $n_{l}$ the number of filters, $n_{l-1}$ the number of input channels of the $l$ -th layer, $s_{l}$ the spatial size of the filter and $m_{l}$ the spatial size of the output feature map. The most relevant factor for the Closing Model is the number of pixels in the input and output images (denoted by $L$ in Section 3.1), which is proportional to $m_{l}$ in Eq. (3). Therefore, assuming that the presented CNN architecture works equally well for varying values of $L$ , the problem-specific complexity of the inference step in the Closing Model is $O(L^{2})$ .

3.4 Step three: The Lifting Model

The task of the Lifting Model is to predict for a two-channel object/gripper image, after closing the gripper, if the object falls out of the gripper or remains between the tips when the gripper is lifted. Since the object image after lifting is not needed for further processing steps, the Lifting Model is treated as classification task. This task is learned with a CNN since this is the state-of-the art method for image classification tasks. The input is a 2-Channel image and the output is computed through a sigmoid function. The output is thresholded for evaluation purposes to either 0 or 1. A standard CNN (Fig. 13) is used for classification in order to predict if the object falls out of the gripper during lifting. The training data are again gained from the V-REP simulation. The network is trained with 5000 positive and 5000 negative images and tested on 1000 positive and 1000 negative images. The resulting confusion matrix entries on the test data are given in Table 3 (right). The Matthews correlation coefficient of the confusion matrix is $\textit{MCC}=$ 0.88. Figure 14 shows some examples for positively and negatively classified images.

Figure 12.

Examples for the prediction of the Closing Model. Left column: input. Center column: prediction. Right column: ground truth.

Figure 13.

CNN architecture for classification of pre-grasp images into liftable and non-liftable. Net2Vis was used for visualization [49]. First published in [28].

Figure 14.

Examples for positively (top row) and negatively (bottom row) classified images by the CNN of the Lifting Model.

3.4.1 Computational complexity

The computational complexity of the inference step for the CNN in the Lifting model is mainly determined by Eq. (3). In analogy to the Closing Model, we argue here that the problem-specific complexity corresponds to $O(L^{2})$ . Forward propagation through the additional fully connected layer before the binary classification (Fig. 13) has itself a computational complexity of $O(L^{2})$ , assuming that the sizes of the fully connected layer and the preceding feature maps are proportional to the number $L$ of pixels in the input images. Therefore, the overall computational complexity of the inference step in the Lifting Model is $O(L^{2})$ .

4. Evaluation and validation of the the Overall Model

4.1 Performance evaluation

The evaluation of the performance of the Overall Model is done with respect to two measures. The first measure is the confusion matrix derived from the individual confusion matrices after each of the three evaluation steps. Figure 15 shows the graph which is used for the determination of the overall matrix. After the first step, the Lowering Model, the entries of its confusion matrix are obtained, denoted by $TP_{Lo},FP_{Lo},FN_{Lo},TN_{Lo}$ . Since NSGA-II yielded the best results (see Section 3.1.2) of the compared optimizers in terms of the confusion matrix, NSGA-II is chosen for the Lowering Model in this overall evaluation. In principle, both the true positive and false positive results could be used as input for the Closing Model. However, since we did not obtain any false positive results for the observed 16 000 grasp candidates, only the true positives of the Lowering Model are the input of the Closing Model. The true positives $TP_{Lo}$ are split into the entries of the confusion matrix of the Closing Model. They are denoted by $TP_{Cl}$ , $FP_{Cl}$ , $FN_{Cl}$ , $TN_{Cl}$ . The true positives and false positives of the Closing Model are fed into the Lifting Model. This leads to eight different pathways, where the index $Li\_T$ denotes the results from the true positives and $Li\_F$ the results from the false positives. The false positives from the Closing Model are the grasp candidates where the model wrongly predicts that the object is not pushed out during closing. Therefore, in these cases there is no possibility of a true positive or false negative prediction of the Lifting Model. This means that $TP_{Li\_F}$ and $FN_{Li\_F}$ are inherently both zero. The Overall Model is evaluated on 16 000 test grasp candidates.

Figure 15.

Graph for the determination of the confusion matrix for the Overall Model.

They are chosen such that in each step of the model half of the candidates are positive and half negative. Table 4 shows the obtained confusion matrix: The overall true positive results are the grasp candidates that are correctly classified as positive by the Lowering, Closing and Lifting Model ( $TP_{Li\_T}$ ). The overall true negatives are the sum of each individual true negative prediction ( $TN_{Lo}$ , $TN_{Cl}$ , $TN_{Li}$ ), where the true negatives of the Lifting model are the sum of $TN_{Li}=TN_{Li_{F}}+TN_{Li_{T}}$ . Similarly, the false negatives of the Overall Model are the sum of each individual false negative prediction. Finally, the false positives of the Overall Model are the false positives of the Lifting Model, which are $FP_{Li\_T}$ and $FP_{Li\_F}$ . The values in the resulting overall confusion matrix are given in Tables 4 (detailed) and 5 (summed up). The Matthews correlation coefficient of the overall confusion matrix is 0.84.

The Overall Model is also used for the interpretation of failed grasps. Therefore, the number of grasp candidates that are rejected for the wrong reason and the number of grasp candidates that are rejected for the correct reason are determined: The number of grasp candidates that have been rejected for the wrong reason is given by the sum of the false negative values for each prediction step $FN_{Lo}+FN_{Cl}+FN_{Li\_T}$ and the additional term $TN_{Li\_F}$ . The $TN_{Li\_F}$ are grasp candidates that have been rejected by the Lifting Model but should have been rejected by the Closing Model. With $FN_{O}:=FN_{Lo}+FN_{Cl}+FN_{Li\_T}+TN_{Li\_F}$ as the grasp candidates rejected by the Overall Model for the wrong reason and $TN_{O}=TN_{Lo}+TN_{Cl}+TN_{Li\_T}$ as the number of grasp candidates rejected for the right reason, the percentage of grasp candidates rejected for the wrong reason is

$\displaystyle E:=\frac{FN_{O}}{FN_{O}+TN_{O}}=0.074.$ (4)

This means that only 7.4% of the rejected grasp candidates are rejected for the wrong reason. Thus, the interpretation of failed grasps is in 92.6% of the cases correct and reliable.

4.2 Validation: Comparison with force closure

Table 5
Confusion matrix of the Overall Model – overall result

$n=$ 16 000	Actual positive	Actual negative
Predicted positive	1970	52
Predicted negative	611	13367

Figure 16.

Example of two contact points $c_{1}$ and $c_{2}$ that are in force closure since the dashed connecting line lies in both friction cones. Contact points $c_{3}$ and $c_{4}$ are not in force closure since the connecting line lies only in one friction cone.

In order to validate the stability of the predicted points, the grasp points determined by the Overall Model are compared to the grasp points determined by force closure [52]. Since data from a physics simulation have been used for the training of the individual models, the comparison with force closure also validates V-REP as a simulation tool. The force closure algorithm used here for comparison is an analytical method for finding force closure grasps of a 2D rigid object with two hard-finger point contacts with friction. When the gripper tips are in contact with the object, they exert forces on the object. These forces are called inner forces. When the inner forces can balance external forces and torques applied to the object, the object is said to be in force closure ([53], p. 223). The normal force $\vec{F}_{N}$ at a contact point and the angle $\alpha$ given as $\alpha=\text{atan}(\mu)$ define the friction cone in 2D, where $\mu$ is the static friction coefficient. If the resulting force $\vec{F}$ of two contact points lies inside the cone there is no slipping. In [52], it was shown that a planar grasp with two hard finger contacts with friction is in force closure if the line joining the two contact points lies in both friction cones. This is illustrated in Fig. 16: On the boundary (dashed line) of the planar object two contact points $\vec{c_{1}}$ and $\vec{c_{2}}$ are given. The two friction cones are known when the normal and tangent vectors $\vec{N_{1}}$ , $\vec{N_{2}}$ , $\vec{T_{1}}$ , $\vec{T_{2}}$ of the boundary shape at the contact points and the friction coefficient $\mu$ are given. The dotted line is the joining line between $\vec{c_{1}}$ and $\vec{c_{2}}$ . Since it is located in both friction cones, the grasp is in force closure. In contrast, the two contact points $\vec{c_{3}}$ and $\vec{c_{4}}$ are not in force closure since the joining line lies only in the friction cone of contact point $c_{4}$ . A more detailed description including proofs can be found in [52, 53], pp. 232. The claim that the joining line of two contact points lies in both friction cones is formulated mathematically as

$\displaystyle(\vec{c_{2}}-\vec{c_{1}})\times(\vec{N_{1}}-\mu\vec{T_{1}})>=0$ $\displaystyle(\vec{c_{2}}-\vec{c_{1}})\times(\vec{N_{1}}+\mu\vec{T_{1}})<=0$ (5) $\displaystyle(\vec{c_{1}}-\vec{c_{2}})\times(\vec{N_{2}}-\mu\vec{T_{2}})>=0$ $\displaystyle(\vec{c_{1}}-\vec{c_{2}})\times(\vec{N_{2}}+\mu\vec{T_{2}})<=0$

where $\times$ is the 2D cross product, which is the determinant of the two vectors, see [54]. The first two inequalities check if the line lies in the friction cone of contact point $\vec{c_{1}}$ , that is if the line is on the right side of the left boundary of the friction cone (determinant is greater or equal to 0) and on the left side of the right boundary of the friction cone (determinant is smaller or equal to 0). Inequalities 3 and 4 check the same for the friction cone at contact point $\vec{c_{2}}$ . Given a parametrization of the boundary of a planar object, all combinations of two point contacts that are in force closure can be determined by checking that all inequalities 4.2 hold. The steps for finding the points that are in force closure for the orthographic object images are:

Figure 17.

Application of the force closure algorithm to an orthographic object image. The object shape is indicated by the dashed grey line.

parametrize the boundaries of the object on the orthographic camera image with B-splines [55].

compute the tangent and normal vectors for every third pixel-sized point on the parametrized boundaries, with $\mu=$ 0.5.

for all pairs of points on the boundaries, check if the force closure inequalities (4.2) are fulfilled, and save the pairs of boundary pixels that fulfil the inequalities as contact points being in force closure.

eliminate all contact point pairs where the distance is greater than $s_{\textit{max}}$ .

for the remaining pairs of contact points, the midpoints of the joining lines of each pair are the grasp points determined by force closure.

An example for the enumerated steps is given in Fig. 17. The B-spline parametrization of the boundary is indicated by the grey dashed line, and the tangent and normal vectors by the black and green arrows, respectively. The determined grasp points are indicated by the grey dots. Since the force closure approach assumes point contact, but the three-step model uses rectangular gripper tips, the grasp points cannot be compared directly. Instead, it is verified that the grasp points from the Overall Model are a subset of the grasp points computed by force closure. In order to determine the final grasp points from the Overall Model, the object translation and rotation predicted by the Closing Model has to be determined. Therefore, three image properties are determined for each thresholded object channel before closing and after closing: the centre of mass and the major and minor axis. From these three properties the translated grasp candidate is determined.

Figure 18.

Example for grasp points found by the Overall Model (left) and the force closure approach (right). The grasp points are indicated by the grey dots.

Figure 19.

Schematic view of the end-to-end application of the three-step model.

The Overall Model is evaluated on 50 objects with 420 grasp candidates to determine the liftable grasp candidates. For these grasp candidates, the translation predicted by the Closing Model is determined and applied to the grasp candidates determined by the Lowering Model, yielding the predicted grasp points of the Overall Model. A predicted grasp point is said to be a stable grasp point if at a maximum distance of 5.5px (corresponds to 0.5 cm) at least one grasp point from force closure is detected. From 100 remaining grasp candidates after the Lifting Model, 89% are also in accordance with the grasp candidates determined by force closure, while 11% are not. These 11% correspond to the false positive classified grasp candidates. This means that all true positive grasp candidates from the Overall Model have an equivalent in the force closure grasp points. Figure 18 shows an example of the grasp points found by the Overall Model in comparison to the force closure approach. As can be seen in this figure, there are always more grasp points found by force closure since point contact is assumed. For the shown object, there is only one feasible grasp candidate that is found by the Overall Model.

5. Discussion and outlook

The three-step model presented in this study was successfully trained to determine from an orthographic camera image grasp points where the object can be grasped. Failed grasp attempts can directly be explained through the outcome of each of the three steps. The first step, the Lowering Model is realized as an optimization task with respect to a predicted gripper projection and an object image. In the second step, the Closing Model predicts the interaction between the gripper and the object while closing the gripper. In the third step, the Lifting Model classifies the predicted images of the Closing Model into liftable and non-liftable. In an end-to-end application, the input of the Overall Model is an image of the object, the output is a set of stable grasp points and a set with rejected grasp candidates (plus the information at which step they were rejected). The MCC values of the submodels amount to 0.98, 0.84, and 0.88, respectively, and of the Overall Model to 0.84. In case of grasp-point rejection, the correct cause is determined in 92.6% of the cases.

Compared to the ML-based wholistic approaches to robotic grasping which are mentioned in the introduction section [14, 15, 16, 17], the proposed three-step model allows for a causal attribution why a specific grasp candidates is not suitable. This an important advantage and makes such an approach potentially more useful for industrial applications. In our work, a successful implementation of this approach was shown for single flat objects on a workplate in combination with a robot arm with two-finger gripper and two orthographic cameras, one of them positioned centrally between the gripper tips. To transfer this approach and setting to the real world, the main challenge is to reliably detect the object shape in the camera images. For this purpose, many ML-based algorithms for object shape recognition exist [3, 4].

As was shown for the Lowering Model, noise in the image acquisition process that distorts the object shapes, influences the classification performance only to a minor extent. Since the object images are further downscaled for the Closing Model and Lifting Model, small distortions of the object boundaries will have an even smaller effect on the performance of these two models as well as on the performance of the Overall Model.

The large amounts of required training data for deep learning were generated in simulation instead of the real world. This has many obvious advantages like cost and time savings, but also pitfalls like the deviations between the simulation and physical reality (which we assessed via the successful validation with the force closure method). We want to point out an additional advantage of using simulations which comes to play in our study: In simulation, additional data can be acquired which is not easily available in a real-world setting. In the presented approach, this concerns the camera placed between the gripper tips to record the training data for the Closing Model even when the gripper is nearly or completely closed. This would not be possible in this straightforward way in the real world because of occlusions. Does this prevent the transfer to the real world? In training yes, but – and this is the crucial point – not during the inference phase during application. In this phase, only the camera image before closing the gripper is required (as input to the Closing Model).

In the current version of the three-step model, the Closing Model and the Lifting Model are implemented by CNNs. This is a straightforward design decision since CNNs are state-of-the-art in the areas of image-to-image regression and image classification. However, especially in the field of classification there are many competing machine learning algorithms which may be promising alternatives. Recent developments are for example neural dynamic classification algorithms [56], neural ensemble methods with only a small set of hyperparameters [57], or finite element machine classifiers where the whole training set is modeled as a probabilistic manifold [58]. Especially neural dynamic classification achieves competitive results in image classification tasks (shown in [56] for the MNIST datase [59]).

As the optimization approach for the Lowering Model has the biggest impact on the computational costs in the inference step according to our analysis of the computational complexity of the submodels, it is also desirable to reduce the computational effort for the Lowering Model. This could be achieved by using interactive preference articulation for the parameters to be determined [60]. In contrast to NSGA-II/-III, this approach allows to adapt the region of interest on the Pareto front during the optimization process.

The next planned steps of our research are the following: (1) Adapt the model to a real-world setting in the application phase and adjust the preceding training process in simulation as necessary. (2) Work with generic object shapes in 3D, allowing for arbitrary objects: For this purpose, depth cameras have to be used, and the submodels have to be changed accordingly. The main challenge is the handling of uncertainties on the depth images due to changes in perspective and occlusions. For this reason, the Closing Model has to be implemented on the basis of artificial neural networks with probabilistic properties. (3) Work with full-color images instead of object shapes to recognize objects more specifically. In this way, the predictions of the trained Closing and Lifting Models could account for an uneven mass distribution and varying friction coefficients.

Thus, there are still open research questions before the proposed approach can be applied to arbitrary real-world settings. Nevertheless, the contribution of our paper is to put forward the three-step approach which allows for clear causal attributions why specific grasp candidates are not suitable, and why an object may be completely non-graspable. This is highly valuable information for any industrial application in which ML-based systems often are avoided as long as they are a complete “black box”. In contrast, we could demonstrate that our proposed Overall Model can be successfully trained and applied to a specific setting in robotic grasping and gives the user the promised insights.

6. Conclusions

In this study, it was proposed to divide the task of robotic grasping into three simpler tasks that allow for the interpretation of failed grasp attempts. The models that have to be learned to accomplish these three tasks are the Lowering Model, the Closing Model and the Lifting Model. Overall, the evaluation of each single model shows with respect to classification performance very good results. The evaluation of the end-to-end application of the Overall Model shows that the causal attribution of failed grasps is reliable in 92% of the cases. Furthermore, the classification performance of the Overall Model is only slightly worse than the individual classification performances. Comparison with the force closure algorithm shows that all grasp points identified by the Overall Model as final true positives are stable with respect to external torques and wrenches. This is an important validation of the Overall Model and the underlying simulation approach using V-REP and the bullet physics engine.

Footnotes

The grid for the systematic search is: $s\in$ [0px, 30px] with $\Delta s=$ 1px, $\alpha\in$ [0 ${}^{\circ}$ , 180 ${}^{\circ}$ ] with $\Delta\alpha=$ 1 ${}^{\circ}$ and $dx,dy\in$ [ $-$ 6px, 6px] with $\Delta x=\Delta y=$ 1px.

Acknowledgments

This work was supported by the EFRE-NRW funding programme “Forschungsinfrastrukturen” (grant no. 34.EFRE-0300119).

We are also thankful to Prof. Ralf Möller from Bielefeld University for many helpful discussions and his overall support.

References

Caldera

Rassau

Chai

. Review of deep learning methods in robotic grasp detection. Multimodal Technologies and Interaction. 2018; 2(3): 57.

Sun

. Convolutional Neural Networks at Constrained Time Cost. arXiv preprint arXiv:14121710. 2014.

Zhao

Zheng

. Object Detection with Deep Learning: A Review. arXiv:180705511 [cs]. 2019 Apr; ArXiv: 1807.05511. Available from: http://arxiv.org/abs/1807.05511.

Liu

Ouyang

Wang

Fieguth

Chen

Liu

, et al. Deep learning for generic object detection: a survey. International Journal of Computer Vision. 2020 Feb; 128(2): 261–318. Available from: doi: 10.1007/s11263-019-01247-4.

Rubert

Kappler

Morales

Schaal

Bohg

. On the relevance of grasp metrics for predicting grasp success. In: 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS); 2017. pp. 265–272. ISSN: 2153-0866.

Larsen

Kim

Kupke

Schuster

. Automatic path planning of industrial robots comparing sampling-based and computational intelligence methods. Procedia Manufacturing. 2017 Dec; 11: 241–248.

Martínez-Del-Amor

Pérez-Hurtado

Martínez-Del-Amor

Zhang

Neri

Pérez-Jiménez

. A membrane parallel rapidly-exploring random tree algorithm for robotic motion planning. Integrated Computer-Aided Engineering. 2020; 27(2): 121–138. Available from: https://nottingham-repository.worktribe.com/output/3656358.

Wang

Lian

Zhao

. Vision-based robotic grasping from object localization, object pose estimation to grasp estimation for parallel grippers: a review. Artificial Intelligence Review. 2021 Mar; 54(3): 1677–1734. ArXiv: 1905.06658. Available from: http://arxiv.org/abs/1905.06658.

Bicchi

Kumar

. Robotic grasping and contact: A review. In: Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No. 00CH37065). vol. 1. IEEE; 2000. pp. 348–353.

10.

Jiang

Moseson

Saxena

. Efficient grasping from rgbd images: Learning using a new rectangle representation. In: 2011 IEEE International Conference on Robotics and Automation. IEEE; 2011. pp. 3304–3311.

11.

Lenz

Lee

Saxena

. Deep learning for detecting robotic grasps. The International Journal of Robotics Research. 2015; 34(4-5): 705–724.

12.

Ogas

Avila

Larregay

Moran

. A Robotic Grasping Method using ConvNets. In: 2019 Argentine Conference on Electronics (CAE); 2019. pp. 21–26.

13.

Illingworth

Kittler

. A survey of the hough transform. Computer Vision, Graphics, and Image Processing. 1988; 44(1): 87–116. Available from: https://www-sciencedirect-com-443.web.bisu.edu.cn/science/article/pii/S0734189X88800331.

14.

Redmon

Angelova

. Real-time grasp detection using convolutional neural networks. In: 2015 IEEE International Conference on Robotics and Automation (ICRA). IEEE; 2015. pp. 1316–1322.

15.

Levine

Pastor

Krizhevsky

Ibarz

Quillen

. Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection. The International Journal of Robotics Research. 2018; 37(4-5): 421–436.

16.

Morrison

Corke

Leitner

. Closing the loop for robotic grasping: A real-time, generative grasp synthesis approach. arXiv preprint arXiv:180405172. 2018.

17.

Mahler

Liang

Niyaz

Laskey

Doan

Liu

, et al. Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics. arXiv:170309312 [cs]. 2017 Aug; Available from: http://arxiv.org/abs/1703.09312.

18.

Tobin

Fong

Ray

Schneider

Zaremba

Abbeel

. Domain randomization for transferring deep neural networks from simulation to the real world. In: 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE; 2017. pp. 23–30.

19.

Depierre

Dellandréa

Chen

. Jacquard: A Large Scale Dataset for Robotic Grasp Detection. arXiv:180311469 [cs]. 2018 Sep; ArXiv: 1803.11469. Available from: http://arxiv.org/abs/1803.11469.

20.

Gunning

Stefik

Choi

Miller

Stumpf

Yang

. XAI-Explainable artificial intelligence. Science Robotics. 2019; 4(37). Available from: https://robotics.sciencemag.org/content/4/37/eaay7120.

21.

Chakraborty

Tomsett

Raghavendra

Harborne

Alzantot

Cerutti

, et al. Interpretability of deep learning models: A survey of results. In: 2017 IEEE SmartWorld, Ubiquitous Intelligence Computing, Advanced Trusted Computed, Scalable Computing Communications, Cloud Big Data Computing, Internet of People and Smart City Innovation (SmartWorld/SCALCOM/UIC/ATC/CBDCom/IOP/SCI); 2017. pp. 1–6.

22.

Murdoch

Singh

Kumbier

Abbasi-Asl

. Definitions, methods, and applications in interpretable machine learning. Proceedings of the National Academy of Sciences. 2019; 116(44): 22071–22080. Available from: https://www-pnas-org-s.web.bisu.edu.cn/content/116/44/22071.

23.

Finn

Goodfellow

Levine

. Unsupervised Learning for Physical Interaction through Video Prediction. arXiv (2016). arXiv preprint arXiv:160507157. 2016.

24.

Schenck

Hasenbein

Möller

. Detecting affordances by mental imagery. In: Proceedings of the 12th International Conference on Adaptive Behaviour (SAB 2012), Workshop on Artificial Mental Imagery in Cognitive Systems and Robotics. Citeseer; 2012. pp. 15–32.

25.

Kloss

Schaal

Bohg

. Combining learned and analytical models for predicting action effects. CoRR. 2017; abs/1710.04102. Available from: http://arxiv.org/abs/1710.04102.

26.

Mottaghi

Bagherinezhad

Rastegari

Farhadi

. Newtonian Image Understanding: Unfolding the Dynamics of Objects in Static Images. arXiv preprint arXiv:151104048. 2015.

27.

Mottaghi

Rastegari

Gupta

Farhadi

. “What happens if …” Learning to Predict the Effect of Forces in Images. arXiv preprint arXiv:160305600. 2016.

28.

Schwan

Schenck

. Visual Movement Prediction for Stable Grasp Point Detection. In: International Conference on Engineering Applications of Neural Networks. Springer; 2020. pp. 70–81.

29.

Copellia. Copellia Robotics homepage; 2021. Accessed: 2021-03-28. https://www.coppeliarobotics.com/.

30.

bullet3. bullet3 physics engine; 2021. Accessed: 2021-03-28. https://github.com/bulletphysics.

31.

Huang

Zhu

Siew

. Extreme learning machine: theory and applications. Neurocomputing. 2006; 70(1-3): 489–501.

32.

Raschka

. Model Evaluation, Model Selection, and Algorithm Selection in Machine Learning. arXiv:181112808 [cs, stat]. 2020 Nov; ArXiv: 1811.12808. Available from: http://arxiv.org/abs/1811.12808.

33.

Ngatchou

Zarei

El-Sharkawi

. Pareto Multi Objective Optimization. In: Proceedings of the 13th International Conference on, Intelligent Systems Application to Power Systems; 2005. pp. 84–91.

34.

Das

Mullick

Suganthan

. Recent advances in differential evolution – an updated survey. Swarm and Evolutionary Computation. 2016; 27: 1–30. Available from: https://www-sciencedirect-com-443.web.bisu.edu.cn/science/article/pii/S2210650216000146.

35.

Kukkonen

Lampinen

. GDE3: the third evolution step of generalized differential evolution. In: 2005 IEEE Congress on Evolutionary Computation. vol 1; 2005. pp. 443–450. ISSN: 1941-0026.

36.

Deb

Pratap

Agarwal

Meyarivan

. A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Transactions on Evolutionary Computation. 2002 Apr; 6(2): 182–197.

37.

Deb

Jain

. An evolutionary many-objective optimization algorithm using reference-point-based nondominated sorting approach, Part I: solving problems with box constraints. IEEE Transactions on Evolutionary Computation. 2014 Aug; 18(4): 577–601.

38.

Price

Storn

Lampinen

. Differential evolution: a practical approach to global optimization. Springer Science & Business Media; 2006.

39.

Tian

Cheng

Zhang

Jin

. PlatEMO: a MATLAB platform for evolutionary multi-objective optimization [educational forum]. IEEE Computational Intelligence Magazine. 2017 Nov; 12(4): 73–87.

40.

Tharwat

. Classification assessment methods. Applied Computing and Informatics. 2020 Jan; 17(1): 168–192. Available from: doi: 10.1016/j.aci.2018.08.003.

41.

MOEA. MOEA Framework; 2021. Accessed: 2021-03-28. http://moeaframework.org/.

42.

Platypus. Platypus Framework; 2021. Accessed: 2021-03-28. https://github.com/Project-Platypus/Platypus.

43.

Durillo

Nebro

. jMetal: a Java framework for multi-objective optimization. Advances in Engineering Software. 2011 Oct; 42(10): 760–771. Available from: https://www-sciencedirect-com-443.web.bisu.edu.cn/science/article/pii/S0965997811001219.

44.

Rostami

Neri

Gyaurski

. On algorithmic descriptions and software implementations for multi-objective optimisation: a comparative study. SN Computer Science. 2020 Aug; 1(5): 247. Available from: doi: 10.1007/s42979-020-00265-1.

45.

Wolpert

Macready

. No free lunch theorems for optimization. IEEE Transactions on Evolutionary Computation. 1997 April; 1(1): 67–82.

46.

Al-Dabbagh

Neri

Idris

Baba

. Algorithmic design issues in adaptive differential evolution schemes: Review and taxonomy. Swarm and Evolutionary Computation. 2018 Apr; 43.

47.

Gonzalez

Woods

. Digital Image Processing. Pearson; 2018.

48.

Taghavi

Shoaran

. Hardware Complexity Analysis of Deep Neural Networks and Decision Tree Ensembles for Real-time Neural Data Classification. In: 2019 9th International IEEE/EMBS Conference on Neural Engineering (NER); 2019. pp. 407–410. ISSN: 1948-3554.

49.

Bäuerle

van Onzenoodt

Ropinski

. Net2Vis – A Visual Grammar for Automatically Generating Publication-Tailored CNN Architecture Visualizations. arXiv preprint arXiv:190204 394. 2019.

50.

Guo

Lee

Lewis

Singh

. Action-Conditional Video Prediction using Deep Networks in Atari Games. arXiv preprint arXiv:150708750. 2015.

51.

Rezatofighi

Tsoi

Gwak

Sadeghian

Reid

Savarese

. Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression. arXiv:190209630 [cs]. 2019 Apr; ArXiv: 1902.09630. Available from: http://arxiv.org/abs/1902.09630.

52.

Nguyen

. Constructing force-closure grasps. The International Journal of Robotics Research. 1988 Jun; 7(3): 3–16. Available from: doi: 10.1177/027836498800700301.

53.

Murray

Sastry

. A Mathematical Introduction to Robotic Manipulation. CRC Press; 1994.

54.

Ponce

Stam

Faverjon

. On computing two-finger force-closure grasps of curved 2D objects. The International Journal of Robotics Research. 1993 Jun; 12(3): 263–273. Available from: doi: 10.1177/027836499301200305.

55.

Boor

. A Practical Guide to Splines. Springer New York; 1978.

56.

Rafiei

Adeli

. A new neural dynamic classification algorithm. IEEE Transactions on Neural Networks and Learning Systems. 2017 Dec; 28(12): 3074–3083.

57.

Alam

KMR

Siddique

Adeli

. A dynamic ensemble learning algorithm for neural networks. Neural Computing and Applications. 2020 Jun; 32(12): 8675–8690.

58.

Pereira

Piteri

Souza

Papa

Adeli

. FEMa: a finite element machine for fast learning. Neural Computing and Applications. 2019; 32: 6393–6404.

59.

Lecun

Bottou

Bengio

Haffner

. Gradient-based learning applied to document recognition. Proceedings of the IEEE. 1998 Nov; 86(11): 2278–2324.

60.

Rostami

Neri

. Progressive preference articulation for decision making in multi-objective optimisation problems. Integrated Computer-Aided Engineering. 2017 Jul; 24(4): 1–21.

A three-step model for the detection of stable grasp points with machine learning

Abstract

Keywords

1. Introduction

1.1 State of the art

1.2 Problem statement and proposal

2. Robot simulation and object database

3.2 Training data generation for the Closing and Lifting Model

Table 3 Confusion matrices of the Closing and Lifting Model

4. Evaluation and validation of the the Overall Model

4.1 Performance evaluation

Table 5 Confusion matrix of the Overall Model – overall result

6. Conclusions

Footnotes

Acknowledgments

References

Table 3
Confusion matrices of the Closing and Lifting Model

Table 5
Confusion matrix of the Overall Model – overall result