Segmentation and Location Computation of Bin Objects

2.1. Image Acquisition and Prepreocessing

The stereo images of the occluded bin objects are captured using the stereo cameras, direct lighting of the bin is avoided to reduce brightness and albedo effects. Objects in the bin, partially occlude each other and have different intensity levels due to their location. The captured images are monochromatic 640 × 480 pixels size. The acquired images are pre-processed to improve the efficiency of the segmentation process. The preprocessing involves two stages i) image resizing and ii) noise filtering. Images are resized to minimize the processing time and to improve the efficiency of the system without significant loss of information of the objects. A image size of 128 × 96 pixels is found to be suitable for the current application. Since the topmost object will be the one without occlusions and the one with higher intensity levels in comparison to the rest of the objects, filtering techniques are applied to smooth out the intensity of the objects and to enhance the edges. The images are filtered using a regional filter and a mask. This masked filter, filters the data in the image with the 2-D linear Gaussian filter and a mask the same size as the original image. This filter returns an image that consists of filtered values for pixels in locations where the mask contains ones and unfiltered values for pixels in locations where the mask contains zeroes. The above process smoothens the intensity of the image around the objects. The resulting filtered image is then subjected to segmenting techniques as detailed in the following section.

2.2 Bin Image Segmentation

Bin image segmentation involves identifying the top most object from the cluster of objects in the bin for pick up. Segmentation of occluded objects has been dealt by Rahardja and Kosaka (Krisnawan Rahardja & Akio Kosaka. 1996), who developed a stereo based algorithm to find simple visual clues of complex objects, this algorithm depends on human assistance. In contrast to this we propose an automatic segmentation algorithm which can segment partially occluded bin objects. Since all the objects are partially occluded except the topmost object, separating the topmost object can be done using the grey value of the object. A histogram of the bin stereo images displays the grey levels of the image. Segmentation using binary thresholding is possible by identifying the pixels of grey levels higher than a threshold value which is assumed to relate to the topmost object, as the topmost object has a brightness level higher than the other objects in the bin. A suitable threshold segmenting only the topmost object is to be computed.

For real-time bin picking, automatic determination of threshold value is an essential criterion. To determine this threshold value an algorithm is proposed which uses the grey levels of the image from the histogram of both the stereo images to compute the threshold. The proposed algorithm is as follows:

Segmentation Algorithm:

Step 1:

The histogram is computed from the left and right gray scale images for a bin value of 0 to 255.

Counts a (i), i=1,2,3,…,256

contains the number of pixels with a gray scale value of (i-1) pixels for the left image.

Counts b(i), i=1,2,3,…,256

contains the number of pixels with a gray scale of (i-1) for the right image.

Step 2:

Compute the logarithmic weighted gray scale value of the left and right image as

t a ( i ) = l o g ( c o u n t a ( i ) ) ∗ ( i − 1 )

(1)

t b ( i ) = l o g ( c o u n t b ( i ) ) ∗ ( i − 1 )

(2)

where i = 1,2,3,…,255

Step 3:

Compute the logarithmic weighted gray scale

t a m = 1 256 ∑ i = 1 256 t a ( i )

(3)

t b m = 1 256 ∑ i = 1 256 t b ( i )

(4)

Step 4:

The threshold T is the minimum value of ‘tam’ and ‘tbm’.

Threshold of both the stereo images are computed separately and the min value of the two thresholds is applied as the threshold to both the images.

The topmost object is segmented and converted into a binary image. The performance of the proposed algorithm was compared with the more popular global thresholding algorithm proposed by Otsu. Fig.1 shows the results of the comparison of both the methods.

OPEN IN VIEWER

Fig. 1

Comparison of thresholding methods

3. Object Location

The next phase is to compute the location of the object with respect to the bin area. To pick up the topmost object the robot must be provided with the location co-ordinates of the object midpoint [x, y and z co-ordinates]. The x and y co-ordinates are determined from the 2D segmented image of the object, whereas the z co-ordinate which is the depth or distance co-ordinate requires the 3D representation. To find the z co-ordinate a neural network and stereo image approach is proposed which trains a neural network to compute the distance or z co-ordinate of the object from its stereo images. Two neural nets namely radial basis function nets and feedforward nets are compared in computing the z co-ordinate. The object features of the stereo images are used as the input data and the distance of the object from gripper is used as the output data to train the neural network. The object features are extracted using singular value decomposition [SVD]. The following sections elaborate on SVD and feature extraction process.

3.2 Feature Extraction

To pick up the topmost object, the location of the object is to be computed. Since only one object is present in the segmented image, finding the centroid of the object will provide the x and y coordinates of the object midpoint, however it is also important to know the z coordinate [distance] to move the gripper for pick up process. Stereo feature matching is one of the popular methods for distance computation. In this research we present a method to compute the distance from the added stereo images of the object, called unified stereo imaging, wherein the stereo images of the object are added for feature extraction. The features of the added image varies with respect to the distance of the object, this aspect is used to train the neural network. To optimise the computational time edge images of the added image is used in the feature extraction process. Singular value features are extracted using SVD, for training the network singular values are extracted for various distance of the object. Fig. 2. shows the flow diagram for object feature extraction. Singular values feature having values below ‘1’ are ignored as few prominent singular value features of the object are sufficient to represent the object distance (Zi-Quan Hong, 1991). Hence ten significant features of the object image are fed to a neural network to train the network for object distance computation.

OPEN IN VIEWER

Fig. 2.

Flow Diagram for Object Feature Extraction

3.3 Radial Basis Function Network Architecture

An extremely powerful type of feedforward artificial neural network for pattern recognition is the Radial Basis Function [RBF] Network, which differs strongly from the multiple layer networks in the activation functions. The architecture consists of three layers: an input layer, a hidden layer of J basis functions and an output layer of K linear output units. The activation values of the hidden units are calculated as the closeness of the input vector X_p to an vector μ _j associated with hidden unit Z_j. The activation is given as

Z j , p = e − ∥ x p − μ j ∥ 2 2 σ j 2

(5)

where σ_j and μ_j are respectively the standard deviation and mean of the basis function. The final output is calculated as

Y k , p = ∑ j = 1 J w k j Z j , p

(6)

Fig. 3 illustrates a RBF Network architecture. The μ _j vector is the weight vector to the j-th hidden unit. (Basis function) and σ j 2 is the bias to the j-th hidden unit also known as the spread factor. In the case of classification problems an RBF network finds the centroid of data clusters and uses this centroid as the centers of the Gaussian density function

OPEN IN VIEWER

Fig.3

Radial Basis Function Network Architecture

3.4 Simple Feedforward Network Architecture

The simple feedforward neural network architecture consists of three layers. 10 singular values are fed to the network as input data. The hidden layer is chosen to have 5 neurons and the output consists of 1 neuron, which represents the object distance. The hidden and input neurons have a bias value of 1.0 and are activated by bipolar sigmoid activation function. The initial weights for the network are randomized between −0.5 and 0.5 and normalized. The initial weights that are connected to any of the hidden or output neuron are normalized in such a way that the sum of the squared weight values connected to a neuron is one. This normalization is carried out using equation (7), which is used to implement the weight updating (S.N. Sivanandam & M. Paulraj, 2003).

w 1 j ( n e w ) = w 1 j ( o l d ) w 1 j 2 + w 2 j 2 + … + w n j 2 , j = 1 , 2 , 3 , … ,

(7)

where

n - number of input units

p - number of hidden units

A sum squared error criteria as defined by equation (8) is used as a stopping criteria while training the network. The sum-squared tolerance defined in equation (8) is fixed as 0.01. The network is trained by the conventional back propagation procedure (Laurene Fausett, 1990). The cumulative error is the sum squared error for each epoch and is given by:-

Sum squared error = ∑ p = 1 p ∑ k = 1 m ( t k − y k ) 2

(8)

where

t_k is the expected output value for the k^th neuron,

y_k is the actual output value for the k^th neuron,

m is the total number of output neurons, and

p is the total number of input neurons.

In the experimental phase 53 images were acquired. The bin images are pre-processed to smooth the intensity level of the object. The pre-processed images are segmented to extract the topmost object in the bin. The segmented left and right images are added and the edge of the added image is extracted. The ‘x’ and ‘y’ co-ordinates are computed from the left segmented object image, which is considered as the reference image, by finding the centroid of the reference image. To compute the ‘z’ co-ordinate, the singular value features are extracted from the added edge image using SVD. 10 singular values of an image are fed as input and the object distance is fed as output to the neural networks. The feature data is used to compare the performance of the two networks.

The RBF network is trained with 40 sample data, the spread factor σ_j is chosen as 3.2. The network trained in 1.407 seconds. Subsequently the network is tested for 53 samples. The radial basis function network successfully computed the distance for 47 samples out of 53 samples, resulting in computation efficiency of 88.6%. The training parameters and results are tabulated in Table 1.

The simple feedforward network is trained with 40 images samples using the back propagation training procedure. In the testing phase the network is tested with 53 sample data. The proposed method is found to successfully compute the distance for 49 images with a success rate of 92.45% and error tolerance of 0.001. Table 2. shows the training parameters and test results of the Neural Network.

OPEN IN VIEWER

Table 2.

Training Parameters of the Simple Feedforward Network

Nework Parameters	Values
No. of input neurons	10
No. of Output neurons	1
Spread Factor	3.2
Training time	1.407 seconds
Training Samples	40
Test Samples	53
Success Rate	88.6%

In the real time experimentation phase, the developed stereo vision system is interfaced with an Adept SCARA 600 Robot. A vacuum gripper is used for pick and place operation. A bin with seven partially occluded objects is used for testing. The Adept SCARA Robot is tested for real time bin picking using the object location co-ordinates [‘x’, ‘y’ and ‘z’ co-ordinates] computed by the stereo vision system. The bin picking system is designed to systematically pick the topmost object one at a time. The process is repeated till all the objects in the bin are picked. At an aeverage, six out of the seven objects were picked and placed succesfully with a bin pick and place performance of 85.7%.

A bin picking system using stereo vision sensors for object localization is presented. Algorithms for segmentation of partially occluded bin objects and location of topmost object is proposed. The results of two netwroks, RBF and feedforward nets are compared in evaluating the Z co-ordinate of object midpoint. The feedforward net is found to be more suitable for the current application. The system is experimentally verified and realtime bin picking results are presented. Real time tests validate the applicability of the proposed algorithms for bin picking. The major constraint of the proposed system was poor segmentation in the presence of albedo effects and uneven brightness in certain parts of the bin. Optimal lighting conditions are essential to derive satisfactory results. Future work will include optimising lighting conditions and improving the network performance.