Fruit Detachment and Classification Method for Strawberry Harvesting Robot

2.1. Materials

2.1.2 Strawberry Harvesting Robot

The strawberry harvesting robot (shown in Fig.1) was developed by Laboratory of Agriculture Engineering, Image Processing, Faculty of Agriculture, University of Miyazaki (Yongjie Cui & M Nagata, 2006, 2007). This robot consisted of XYZ three-axial moving stage (1); pneumatic end effector (2); global camera (3); local camera (4) and controller PC (5).

OPEN IN VIEWER

Fig. 1.

Image of strawberry harvesting robot

In order to improve the speed of operation, two-camera imaging system was designed for strawberry recognition. One camera called global camera(SONY DXC-151A, 640× 480)was installed on the top of robot frame, which could capture the image consisting of 8∼10 strawberries; the other camera (called local camera: ELMO EC-202 II, 640× 480)was installed on the end effector and could move with the XYZ moving stage. The image taken by local camera often consisted of one or two strawberries. The global camera had a larger view field than local camera and was responsible for the general location of fruits, while the local camera could capture the image of strawberries with high resolution and was responsible for fruit stem detection and quality judgment. This imaging system could decrease fruit searching time and improve harvest speed effectively. A frame grabbing board (PHOTRON FDM-PCI MULTI) was responsible for the image capturing from these two cameras. A PC (SHARP PC-SJ145) was utilized to execuate image processing task. Four fluorescent tubes (TOSHIBA FL20S. D-EDL-D65) were installed around the robot frame as lighting source, which could decrease the brightness disturbance caused by the natural light from outside. A ring-shaped fluorescent lamp (NATIONAL FCL9EX-N) was installed around the local camera to provide the stable lighting envrionment for fruit location and quality judgement.

The procedure of strawberry harvest was listed below:

1)

The global camera located the general position of every strawberry in its image;

2)

With the position coordinates provided by global camera, controller PC sent instruction to XYZ moving stage and made it move to the related position. Then local camera installed on the end effector captured the image of strawberries;

3)

Blob based algorithm was utilized to locate the strawberry in the central part of local image;

4)

Inertia principal axis of singular strawberry blob was calculated to provide the pose information of fruit. Then quality judgment algorithm would evaluate the ripeness of fruit;

5)

If fruit met the harvest standard, robot would execute strawberry harvest operation according to the extracted stem position coordinates.

2. 2 Stem Detection Method for Harvesting Robot

2.2.1 OHTA Color Space based Image Segmentation Algorithm

Segmentation of fruit form image was the foundation of stem detection. In order to extract red strawberries in image, OHTA color space was selected. This color space was introduced by Ohta (Ohta Yu-Ichi & Kanade Takeo, 1980), who analyzed more than 100 color features which were thus obtained during segmenting eight kinds of color pictures and found a set of orthogonal color features. Compared with other traditional color spaces, the conversion from RGB to OHTA color space is linear and computation inexpensive. To the imaging system of strawberry harvesting robot, the transfomation time from RGB to OHTA color space was about 50ms, which make it suitable for real-time image processing. OHTA color space has two different kinds of expression as shown in equation (1). In this paper, Fruit segmentation was achieved by a threshold based algorithm with the I'₂ feature (R-B).

{ I 1 = ( R + G + B ) ∕ 3 I 2 = ( R - B ) ∕ 2 I 3 = ( 2 G - R - B ) ∕ 4 , { I 2 ' = R - B I 3 ' = ( 2 G - R - B ) ∕ 2

(1)

The image taken by global camera was transformed from RGB color space to OHTA color space firstly, then threshold based method was selected for image segmentation. The threshold value could be set with software interface, which would make the segmentation algorithm work well even if the illumination intensity was different. The approximate center coordinates of every strawberry could be achieved by calculating the average coordinate of the pixels in row and column.

Because farmers usually utilize white or black thin sheets to cover the planting bed, so fruit segmentation experiment was performed on white and black background. Planting bed was covered with white and black plastic sheets, images of 20 samples were taken on different background with local camera. The pixels in the image were classified into four groups: red, unripe, green and background. The average value of R-B for each part was calculated, calculation results of 20 samples on different background were shown in Fig.2.

OPEN IN VIEWER

Fig. 2.

R-B value of four parts of 20 sample images on different background

Fig.2 (a) showed average R-B value of four parts of 20 sample images on white background, Fig.2 (b) showed the average R-B value of four parts of 20 sample images on black background. As shown in Fig.2 (a) and (b): the red part had the maximum R-B value; the background had the minimum R-B value; the variation of background value was relatively constant compared with other three parts; the difference between background and other three parts was very clear; so OHTA color space based segmentation algorithm could work well on black and white background. The segmentation result of image taken by local camera was shown in Fig.3.

OPEN IN VIEWER

Fig. 3.

Image segmentation result with OHTA color space

2.2.2 Blob based Singular Strawberry Detection Algorithm

Because of the randomness of strawberry distribution, there often existed more than one strawberries in the image of local camera, so a detection algorithm to extract complete singular strawberry from image background was necessary.

A blob based singular strawberry recognition algorithm was designed. The image taken by local camera was transformed from RGB color space into OHTA color space, and fruit parts were segmented from the image; an image morphology algorithm was introduced to erase image noises, then blob algorithm was utilized to detect fruit parts in the image; the central blob was defined as strawberry that could be harvested, consequent image processing was executed only within the range of central blob. The procedure and result of strawberry detection was shown in Fig.4.

OPEN IN VIEWER

Fig. 4.

Strawberry detection result with blob aglorithm

2.2.3 Inertia Principal Axis based Pose Judgment Algorithm

Because the inertia principal axis and mass center were inherent properties of object, and the direction of its principal axis often represented the orientation information, so the pose of strawberry was approximately represented with the inertia principal axis of strawberry blob. The principal axis was a line that passed through the mass center of the object, around which the first order moment of this object was minimum. The segmented binary image was recognized as a two dimensional inertia object, and two order central moments of binary image were calculated to get the inertia principal axis of fruit. The stem part could be found along the principal axis. The arc tangent value of inertia principal axis' orientation angle could be calculated with equation (2), which was introduced in (Sabyasachi Dey & Bhargab B, 2002):

θ = 1 2 t g - 1 ( 2 m 11 m 20 - m 02 ) .

(2)

in which θ was the orientation angle of inertia principal axis, m₂₀, m₀₂ and m₁₁ were the two order central moments of image. The value of central moments could be calculated with equation (3).

m i j = Σ Σ ( x - x ̄ ) i ( y - y ̄ ) j f ( x , y ) .

(3)

in which f(x,y) was the function of binary image.

f ( x , y ) = { 1 , i f ( p i x e l ( x , y ) = o b j e c t ) 0 , i f ( p i x e l ( x , y ) = b a c k g r o u n d )

(4)

(x̄, ȳ) could be calculated with equation (5)

x ̄ = ∑ ∑ x f ( x , y ) ∑ ∑ f ( x , y ) y ̄ = ∑ ∑ y f ( x , y ) ∑ ∑ f ( x , y )

(5)

The head and rear of fruit were defined according to the length and color feature along the longer principal axis. The ends of principal axis with green pixels were potential head parts. According to the fruit shape investigation, the distance from head to fruit mass center was shorter than that from rear to fruit mass center. So the head part of fruit was defined as the shorter semi-axis of principal axis with green pixels on it. The opposite end of principal axis was defined as the rear of fruit.

After the judgment of head and rear, a searching area was defined on the head of strawberry to execute stem detection task. Because the direction of stem did not coincide with the direction of principal axis, so a trapezoid shaped range was selected to search more space than rectangle area. The trapezoid range was classified into two parts along the direction of principal axis, the mass center of green pixels in every part was calculated separately, the connected line of two centers represented the position information of stem. The procedure and result of stem detection algorithm was shown in Fig.5.

OPEN IN VIEWER

Fig. 5.

Procedure and result of strawberry stem detection

To a fruit harvesting robot, the measurement of distance from detected fruit to the robot's end effector was difficult. While this robot was developed for harvesting strawberries growing on the ground, the height of planting bed was relatively fixed in a green house, so the distance from fruit to the robot was relatively constant, and this parameter could be defined beforehand in the system calibration process. With the extracted image X-Y coordinates, robot could control the manipulator to execute harvest operation.

2.3 HSI Color Space based Fruit Ripeness Judgment Method

Ripeness judgment method was developed for selective harvest of fruit, in other words, the robot only picked strawberry whose ripeness was 80% above. Fruit's ripeness was defined as the color ratio between the red distribution area and the whole fruit surface area, which could be calculated with equation (6):

C = R ∕ A

(6)

in which C represented color ratio, R represented the area of red pixels in fruit image and A represented the area of strawberry in the image, which could be calculated with the sum of segmented unripe part and red part.

Because it was difficult to distinguish the unripe part and green leaf in OHTA color space, so HSI color space was selected. This color space was claimed to be the closet approximation to human interpretation of color, the hue feature was independent of illumination variety. In HSI space, color is represented by hue angle from 0° to 360°, so threshold method could be used to segment red pixels in the image. Red pixels were marked blue in the range of central blob, the detection result was shown in Fig.6.

OPEN IN VIEWER

Fig. 6.

Red area of strawberry detection result

2.4 Histogram Matching based Fruit Shape Judgment Method

Histogram matching method had been proved robust to changes of the object's orientation, scale, or the viewing position by (Bernt Schiele, 2000), and was utilized for strawberry shape judgment in this paper: Shape judgment result was given by calculating and comparing the similarity between the inspected sample's shape histogram and standard template's shape histogram for every grade, inspected sample would be assigned to the grade whose templates had the biggest similarity with it.

There existed two major factors influencing strawberry's shape judgment result: side symmetry and rear roundness of fruit. In order to evaluate shape feature comprehensively, two varieties of shape histogram representing fruit's side and rear feature were built respectively. Side symmetry feature was represented by normalized side diameter histogram and rear roundness feature was represented by normalized rear radius histogram.

In the image, the central axis of fruit could be defined as the line from fruit's mass center to the mass center of lower part of trapezoid area. In the polar coordinate system defined by central axis and mass center of fruit area, the coordinates of fruit's edge points could be given with the values of angle and related radius. The angles lied in the set [120°, 240°] were defined as rear range and the angles distributed in the set (60°,120°), (240°, 300°) were defined as side range. The shape histograms extraction process was shown in Fig.7.

OPEN IN VIEWER

Fig. 7.

Shape histogram of normal shape and ill-shaped strawberries

Histogram similarity measure was a function defined on pairs of histograms to indicate the degree of resemblance of related fruit features. In the calculation of similarity, histogram features were regarded as multidimensional space vectors and the simliarity could be represented with the distance between them. In order to reduce calculation consumption, histogram intersection method was selected (Richard O. Duda & Peter E. Hart, 2000).

The similarity S(i, j) between two histograms h _i , h _j could be calculated using equation (7):

S ( i , j ) = ∑ k = 1 m min ( h i ( k ) , h j ( k ) )

(7)

in which m was the number of histogram bins;

h _i was the histogram of inspected sample

h _j was the histogram of template sample

S(i, j) was the similarity between two histograms h _i , h _j .

It ranged from 0 to 1.

In the evaluation of shape grade, the side similarity S _s and rear similarity S _r were compared separately, fruit sample was assigned to the grade whose side similarity and rear similarity were both larger than those of the other grades. This judgment method could avoid the samples with good side symmetry and poor rear roundness being misclassified as normal shaped fruit.

3.1 Strawberry Harvest Performance Experimentt

In order to test the harvest performance, growing environment of strawberry was simulated in laboratory. With 100 strawberries harvested from farmers, the performance of strawberry harvest was tested in the simulated growing environment. The average weight of fruit was 15.7 g; the average diameter of stem was 1.7mm; the average length of fruit was 42.6mm; the average width of fruit was 29.4mm. These strawberries (ten samples a time) were placed on the planting bed (1.3m × 0.5m). Five∼Six strawberries were placed along the long side of planting bed, the fruits were distributed randomly to simulate the real growing environment (the distance between plants was about 0.1m, the distance between branches was about 0.2m). The stem recognition process (from image capturation to stem detection) was timed with the library function Get Tick Count() of WindowsAPI, the fruit harvesting process was timed manually. Experiment result was shown in table.1.

OPEN IN VIEWER

Table 1.

Performance of strawberries harvest

Tested Items	Test Results
Stem recognition time	0.95s
Average harvesting time	30s
Average stem length	8mm
Injuring rate	5%
Detection failure rate	7%

It only took about 1 second for the controller PC to give the stem location information, while the developed trial mechanism slowed the harvest process. With optimized harvesting mechanism being able to improve the speed of moving stage and harvesting end-effector, the harvest speed could achieve 10 samples per minute. The average stem length was 8mm, and only 5% of inspected samples were injured during detachment process. The injury was mainly caused by ill-shaped fruit that made principal axis algorithm failed to judge the head of fruit. The stem detection algorithm failed to locate 7% of strawberries because of the occlusion caused by leaves or the strawberries were so close that the blob algorithm could not separate them. So in the future, some artificial intelligence algorithms should be added to improve the harvest performance.

3.2 Strawberry On-line Quality Judgment Experiment

In order to test the selective harvest performance, 50 strawberries (40 above 80% maturity and 10 below 80% maturity) were selected manually. These samples were placed specifically so that the stem detection method could work correctly. The experiment proceeded on both black and white background, experiment result was shown in table 2.

OPEN IN VIEWER

Table 2.

Under harvest rate on black and white plastic sheet

Plastic sheet color	Number of picked samples		Under harvest rate
	ripeness above 80%	ripeness below 80%
White	38	0	5%
Black	40	0	0%

The result showed that the robot could harvest mature strawberries on black background without mistakes, while missed two samples on white background. The white plastic sheet reflected more light than black sheet, and caused some bright spots appearing on the fruit surface, which would influence the color ratio calculation result. While robot did not pick samples whose ripeness below 80%, which met the requirements of farmers.

60 strawberries (45 Grade one (normal shaped)and 15 Grade two (ill-shaped)) were selected manually for fruit shape judgment experiment. 5 normal shaped samples and 5 ill-shaped samples were utilized as templates, average similarity between inspected sample histogram and 5 templates histograms from each grade was calculated for fruit classification. The samples were placed specifically so that the stem detection method could work correctly. The experiment proceeded on the black and white plastic sheet, experiment result was shown in table 3.

OPEN IN VIEWER

Table 3.

Shape classification result with histogram matching method

Plastic sheet color	Samples grade	Samples classified into each grade		Accuracy
		Grade one	Grade two
White	Grade one	36	4	90%
	Grade two	0	10	100%
Black	Grade one	37	3	92.5%
	Grade two	1	9	90%

From the experiment result, we could see that the histogram matching based shape judgment method could achieve 90% for grade one strawberries and 100% for grade two strawberries on white background, while 92.5% for grade one strawberries and 90% for grade two strawberries on black background. The misclassification error was mainly caused by image segmentation error and there existed no obvious difference on the black and white background. Experiment result proved that this method had high accuracy in shape judgment and was appropriate for field application.