A transformer-based mask R-CNN for tomato detection and segmentation

Abstract

Fruit detection is essential for harvesting robot platforms. However, complicated environmental attributes such as illumination variation and occlusion have made fruit detection a challenging task. In this study, a Transformer-based mask region-based convolution neural network (R-CNN) model for tomato detection and segmentation is proposed to address these difficulties. Swin Transformer is used as the backbone network for better feature extraction. Multi-scale training techniques are shown to yield significant performance gains. Apart from accurately detecting and segmenting tomatoes, the method effectively identifies tomato cultivars (normal-size and cherry tomatoes) and tomato maturity stages (fully-ripened, half-ripened, and green). Compared with existing work, the method has the best detection and segmentation performance for these tomatoes, with mean average precision (mAP) results of 89.4% and 89.2%, respectively.

Keywords

Mask R-CNN Swin Transformer tomato detection instance segmentation

1 Introduction

Traditional harvesting is labor-intensive and time-consuming, and the process is gradually being automated. Harvesting robots are increasingly used in agricultural production, where they can perform some of the most labor-intensive harvesting tasks in orchards [1]. Intelligence is an inevitable direction for harvesting robots [2], and vision-based fruit detection is a vital task for them, as it is the front-end perception system before subsequent grasping. However, the vision system is challenging due to factors such as fruit overlapping, occlusion, and illumination variation [3]. Accordingly, fruit detection has been explored by many researchers over the past several decades. Many methods use handcrafted features to detect fruit, and supervised methods based on machine learning are also widely applied in fruit detection.

Handcrafted features are often used to compute fruit class membership. A number of researchers have used color-based segmentation to detect fruit with distinct colors, sometimes integrating additional features such as shape and texture [4]. Plebe and Grasso [5] combined a hue saturation value (HSV) color space with adaptive shape analysis to recognize oranges. Huirong et al. [6] used the red-blue (R-B) chromatic aberration information of images to identify oranges. Kelman and Linker [7] proposed a shape-based method for apple detection, with Canny filters to identify edges in images. Edges corresponding to 3D convex objects were then detected using preprocessing operations and convexity tests. Linker et al. [8] combined color, texture, and edge shape features to detect green apples, achieving 85% accuracy for green apple recognition under natural lighting. Zhao et al. [9] investigated a tomato recognition algorithm based on image feature fusion, and used wavelet transformation to fuse the a*- and I-component extracted from the L*a*b* and YIQ color spaces. Moreover, algorithms such as the circular Hough transform (CHT) and random Hough transform (RHT) have been widely used to fit circular and oval shapes around possible regions in an image to segment fruits [10 –12].

Many studies have applied machine learning techniques to fruit detection in agriculture. Ji et al. [13] used a support vector machine (SVM) to improve the accuracy and efficiency of apple recognition after extracting the color and shape features. Liu et al. [14] trained an SVM classifier using histogram of gradient (HOG) descriptors to detect tomatoes. Bulanon et al. [15] used k-means clustering to detect red apples. Zhao et al. [16] identified tomatoes by extracting Haar-like features of grayscale images and classifying them using AdaBoost. Wu et al. [17] proposed an automatic algorithm for the recognition of ripening tomatoes, using a weighted relevance vector machine (RVM) classifier to classify image blocks based on color and texture features.

While traditional machine learning has brought great improvements to fruit detection, these methods require handcrafted features and are usually tedious. Deep learning enables automatic feature extraction, which is highly adaptable in a variety of fruit recognition scenarios [18]. Fruit detection algorithms based on deep learning include two-stage and single-stage detectors.

Two-stage detectors (region proposal followed by classification and detection) include the region-based convolutional neural network (R-CNN) [19], Faster R-CNN [20], and Mask R-CNN [21]. Bargoti and Underwood [22] used Faster R-CNN to detect mangoes and apples in the orchard background with an F1 score over 0.9. Sa et al. [23] explored the early and late fusion method of fusing multimodal (RGB and NIR) information, and applied the Faster R-CNN model with multi-vision sensors to detect fruits and vegetables, improving the detection accuracy of sweet peppers to 83.8%. According to [24], Mask R-CNN is improved by combining ResNet and DenseNet to make it more suitable for segmentation of overlapping apples. Yu et al. [25] used Mask R-CNN to generate segmented images of strawberry fruits, and proposed a visual localization method for strawberry picking points.

Single-stage detectors have a more concise structure, which removes the region proposal stage. Representative methods are You Only Look Once (YOLO) [26 –28] and the single shot MultiBox detector (SSD) [29]. For the operation and visual positioning of a banana robot, Wu et al. [30] proposed an improved YOLOv5 model for multi-target recognition of bananas. To detect and locate oil-seeded camellia fruit, YOLO-Oleifera was developed as a fruit detection model [31]. Wang et al. [32] applied YOLOv5 object detection network and DBSCAN point cloud clustering method to determine the location of litchi bunch fruits. Koirala et al. [33] developed a MangoYOLO framework, which achieved an F1 score of 0.968 and average accuracy of 98.3% in mango detection. Yuan et al. [34] proposed a method based on SSD to detect cherry tomatoes.

In the case of tomato detection, the current methods have two main limitations. The detection results of a rectangular bounding box cannot match the tomatoes exactly, and these methods are unable to distinguish cultivars or ripeness, and most can only identify normal-sized ripe or green tomatoes [35 –37].

Vision Transformer algorithms applied to image feature extraction were proposed with good results [38]. Inspired by this, we propose a Transformer-based Mask R-CNN structure, making it more suitable for the identification of tomato varieties and ripeness, and apply to tomato detection and segmentation. The remainder of this paper is organized as follows. Section 2 specifies the method of implementation by introducing feature extraction, generation of RoIs, and the multi-branch network structure, also describes the model training and loss function. Experiments and their results are presented in Section 3. We describe our conclusions in Section 4.

2 Methods

Mask R-CNN [21] is an object detection and instance segmentation algorithm. An extension of Faster R-CNN [20], it adds a mask branch at the end of the model to predict the class of each pixel based on the original classification and regression. We propose a Transformer-based Mask R-CNN structure and apply it to the vision system of a tomato harvesting robot so that it can effectively detect and segment various tomatoes. We replace the backbone network with the Swin Transformer [38] network for feature extraction. The feature maps obtained through the backbone network are used as input to the RPN network, which produces candidate object bounding boxes as regions of interest (RoIs). After obtaining the features of RoIs, the fully connected layers (FC) and fully convolutional network (FCN) are used to generate the classification scores of objects, bounding boxes, and segmentation masks. The model structure is shown in Fig. 1.

Fig. 1

Overall structure of improved Mask R-CNN.

2.1 Feature extraction (Swin Transformer)

Computer vision modeling has long been dominated by convolutional neural networks (CNNs), such as VGG [39], GoogleNet [40], ResNet [41], DenseNet [42], and EfficientNet [43]. In contrast, the most popular network in natural language processing (NLP) is Transformer [44], which is designed for sequence modeling and transformation tasks, as it focuses on modeling long-range dependencies in data.

We use a new vision Transformer, called Swin Transformer, as a backbone network for feature extraction. This backbone uses a hierarchical Transformer whose representation is computed by shifting windows. By restricting the self-attentive computation to non-overlapping local windows while allowing cross-window connections, the shift-window scheme brings higher efficiency. This hierarchy has the flexibility to model at different scales with linear computational complexity associated with image size. These qualities allow the Swin Transformer to better preserve the features of tomato images at multiple scales.

To achieve the best speed-accuracy trade-off, we use a tiny version of the Swin Transformer (Swin-T). The model structure, as shown in Fig. 2, has four steps, each with similar repeating units. In step 1, the input H × W × 3 RGB image is partitioned into non-overlapping patches, each considered a token with size 4 × 4, feature dimension 4 × 4 × 3 = 48, and number $\frac{H}{4} \times \frac{W}{4}$ . The feature dimensions of the divided patches are mapped to dimension C by a linear embedding layer, and sent into Swin Transformer blocks consisting of two Transformer blocks with modified self-attentive computation. To produce a hierarchical representation, the number of tokens is reduced by patch merging layers as the network goes deeper. In step 2, the first patch merging layer merges the features of each set of 2 × 2 adjacent patches and applies a linear layer on the connected features in 4C dimension. The number of tokens is reduced by a multiple of 2 × 2 = 4 with downsampling, and the output dimension is set to 2C. The Swin Transformer block is applied for feature transformation, and the resolution is kept at $\frac{H}{8} \times \frac{W}{8}$ . Steps 3 and 4 repeat the patch merging and feature transformation process, with respective output resolutions of $\frac{H}{16} \times \frac{W}{16}$ and $\frac{H}{32} \times \frac{W}{32}$ . In the structure of Swin-T, we set the number of channels C of the hidden layer to 96 in step 1, and the numbers of layers of the four steps to {2, 2, 6, 2}, respectively. These steps produce a hierarchical representation with the same feature map resolution as typical convolutional networks.

Fig. 2

Structure of Swin Transformer.

The Swin Transformer block is the core of each step. Figure 3 shows the structure of two consecutive Swin Transformer blocks. The input feature z^l-1 is normalized by LayerNorm (LN) and learned by window multi-head self-attention (W-MSA), and a residual operation obtains the output features ${\hat{z}}^{l}$ . Next, a LayerNorm (LN), an MLP, and a residual connection obtain the output feature z^l. W-MSA denotes window-based multi-head self-attention using regular window partitioning configurations. It has a linear relationship between computational complexity and image size, which greatly reduces the complexity of the model. The second block is similar to the first, where shifted-window multi-head self-attention (SW-MSA) is used instead of W-MSA in the computation of the features. These two Swin Transformer blocks are calculated as follows: $\begin{matrix} {\hat{z}}^{l} = W - MSA (LN (z^{l - 1})) + z^{l - 1} \\ z^{l} = MLP (LN ({\hat{z}}^{l})) + {\hat{z}}^{l} \\ {\hat{z}}^{l + 1} = SW - MSA (LN (z^{l})) + z^{l} \\ z^{l + 1} = MLP (LN ({\hat{z}}^{l + 1})) + {\hat{z}}^{l + 1} \end{matrix} .$ (1)

Fig. 3

Two Swin Transformer blocks.

The output feature z^l of the first block is used as the input feature of the second block, whose output feature is z^l+1. SW-MSA is a multi-headed self-attentive module with shifted window configuration, which eliminates the lack of information exchange between non-overlapping windows in the W-MSA layer. The shifted window partition method introduces the connection between adjacent non-overlapping windows in the previous layer, which greatly increases the perceptual field and improves model performance. As a result, the Swin Transformer significantly outperforms the CNN model with less time latency, with the best speed-accuracy trade-off.

2.2 Generation of RoIs (RPN) and RoIAlign

The output of the Swin-T backbone network is used as the input of the RPN network, and a series of region proposals are generated. Generally, the tomato images in the dataset have differences in size, due mainly to the distance from which they were photographed. There are also differences in the shapes of normal-sized tomatoes and cherry tomatoes, resulting in a large variation in the horizontal-to-vertical ratio of tomato exposure. Therefore, we designed anchor scales of 4 × 4, 8 × 8, 16 × 16, 32 × 32, and 64 × 64, and aspect ratios of 1:1, 1:2, and 2:1 for random combination, so each point on the feature map corresponds to 15 anchors of different sizes and widths, which correspond to the original map to basically cover all possible tomato targets. The region proposal network (RPN) filters from the huge number of anchors generated and adjusts the position to get the proposal, which is mapped to the feature map obtained by the backbone network, called Regions of Interest (RoIs).

The subsequent fully connected network requires a fixed dimensionality of features, but each RoI corresponds to a different feature size. RoIAlign pools the RoIs to a fixed dimensionality. Unlike the RoI Pooling used in Faster R-CNN, RoIAlign retains all the floating points. The values of many sample points are obtained by bilinear interpolation, and these are pooled for maximum value to get the final value. RoIAlign achieves better performance due to the use of sample points and the preservation of floating points.

2.3 Object detection and instance segmentation (FC/FCN)

After RoIAlign, the object detection and instance segmentation results are generated by a multi-branch prediction network. One branch employs a fully connected network (FC) to get the prediction of classification and bounding box regression, and the other, the mask branch, employs a fully convolutional network (FCN) to predict image segmentation. FCN uses convolution and deconvolution to build an end-to-end network. It convolves and pools the image to reduce the size of its feature map, and performs deconvolution to continuously improve the resolution of the feature map through interpolation. Finally, FCN determines the category of each pixel.

2.4 Model training and loss function

Due to the small number of images in the tomato dataset, we introduce a pre-trained model based on the ImageNet-1K [45] dataset, which contains 1.28 million training images and 50000 validation images from 1000 classes. The pre-training model extracts the general features of all classes from ImageNet, and the parameters of the model can be adjusted. We use Swin-T as the backbone network, initialized with pre-trained weights on ImageNet with the AdamW [46] training optimizer. An initial learning rate of 0.0001, weight decay of 0.05, and batch size of 2 are used. The training schedule is 36 epochs, with the learning rate decayed by 10× at epochs 27 and 33.

We use multi-scale training for data augmentation during training. This is the setting of different image input scales, randomly selected from multiple scales during training, where the input image is zoomed to the selected scale and fed into the network. Although each iteration only uses one scale, it is different in each iteration. Multi-scale training can effectively enhance object detection accuracy, which increases the robustness of the network with no additional computational effort. The input image size is adjusted with random levels and scale jitter, where the scale of the short edge is randomly sampled from 480 to 800 pixels, and the scale of the long edge is fixed at 1333 pixels.

The training loss has two components. The first is loss of RPN, with both classification and regression components, and the second is training loss of the multi-branch prediction network. The loss function is $L_{total} = L_{RPN} + L_{multi - branch}$ (2) where L_total represents all losses, L_RPN represents the training loss of RPN, and L_multi-branch represents the training loss of the multi-branch network. The RPN has two main roles. One is to select the proposal. It is not required to subdivide the class of foreground, so it is a binary classification using cross-entropy loss at this stage. The other is preliminary coordinate correction for the anchors belonging to the foregrounds. L_RPN includes both classification and regression losses: $\begin{matrix} L_{RPN} = \frac{1}{N_{cls 1}} \sum_{i} L_{cls} (p_{i}, p_{i}^{*}) \\ + λ_{1} \frac{1}{N_{reg 1}} \sum_{i} p_{i}^{*} L_{reg} (t_{i}, t_{i}^{*}) \end{matrix}$ (3) where $\sum_{i} L_{cls} (p_{i}, p_{i}^{*})$ represents the classification loss of anchors, and $\sum_{i} p_{i}^{*} L_{reg} (t_{i}, t_{i}^{*})$ represents the regression loss. p_i is the true class of each anchor, and $p_{i}^{*}$ is its predicted class. t_i is a vector representing the four parameterized coordinates of the predicted bounding box to measure the regression loss, and $t_{i}^{*}$ is that of the ground-truth box associated with a positive anchor. The classification and regression losses are normalized by N_cls1 and N_reg1, respectively, and λ balances the two parts of the losses. The classification uses cross-entropy loss, and regression uses the smooth_L1 function, $L_{cls} = - log [p_{i}^{*} p_{i} + (1 - p_{i}^{*}) (1 - p_{i})]$ (4) $L_{reg} (t_{i}^{*}, t_{i}) = {smooth}_{L 1} (t_{i}^{*} - t_{i})$ (5) ${smooth}_{L 1} (x) = {\begin{matrix} 0.5 x^{2}, if | x | < 1 \\ | x | - 0.5, otherwise \end{matrix} .$ (6) Apart from the loss of RPN, the training loss L_multi-branch of the multi-branch network has the three branches of classification, regression, and mask loss,

$\begin{matrix} L_{multi - branch} = \frac{1}{N_{cls 2}} \sum_{i} L_{cls} (p_{i}, p_{i}^{*}) \\ + λ_{2} \frac{1}{N_{reg 2}} \sum_{i} p_{i}^{*} L_{reg} (t_{i}, t_{i}^{*}) \\ + γ \frac{1}{N_{mask}} \sum_{i} L_{mask} (s_{i}, s_{i}^{*}) \end{matrix}$ (7)

$\begin{matrix} L_{mask} (s_{i}, s_{i}^{*}) = - (s_{i}^{*} log (s_{i}) \\ + (1 - s_{i}^{*}) log (1 - s_{i})) \end{matrix}$ (8)

The first two parts of the multi-branch loss are calculated in the same way as RPN, with the addition of L_mask to represent the loss of segmentation. The hyperparameters λ and γ balance the training losses of the regression and mask branches. $s_{i}^{*}$ and s_i respectively represent the mask binary matrices from the prediction and ground-truth labels. The mask branch has a Km²-dimensional output for each RoI, which encodes K binary masks of resolution m × m, one for each of the K classes. The sigmoid function is applied to each pixel on the mask that is sent to the cross-entropy loss. The average of all pixel losses is taken as the loss of the mask branch.

3 Experiment and results

Experiments were implemented with the PyTorch framework [47] and mmdetection [48], an open source, PyTorch-based object detection toolbox. We trained and tested models with an Nvidia RTX 2070 for GPU acceleration, Intel Core i7-9700k CPU, and 32 GB memory.

3.1 Dataset

An instance segmentation dataset of tomatoes at different maturity stages from Laboro Tomato [49] is used. A total of 804 RGB images of tomatoes were taken from Ide tomato farm in Fujisawa, Kanagawa Prefecture, Japan. The image format is JPG, and two cameras with different resolutions were used to capture the images. One part has 305 tomato images captured with a 3024 × 4032 pixel camera, and the other part has 499 tomato images captured with a 3120 × 4160 pixel camera. These tomatoes were grown in greenhouses and include normal-sized and cherry tomatoes of various maturity stages. Images were taken from different camera angles and under different lighting conditions. This better simulates the operation of the harvesting robot vision system.

The Laboro Tomato dataset has six categories. All tomatoes are divided into two categories according to cultivar (normal size and cherry tomato), and also into three categories depending on the stage of ripening (fully ripened, half-ripened, and green). Fully ripened tomatoes are red on 90% or more of the surface and are ready to be harvested. Half-ripened tomatoes are greenish and need time to ripen. Their surface is 30% to 90% red. Green tomatoes are almost completely green or white, with less than 30% red. Figure 4 shows the categories of tomatoes. The number of tomatoes labeled in each category is shown in Table 1.

Fig. 4

Images of six categories of tomatoes. a. Fully ripened, normal size; b. half-ripened, normal size; c. green, normal size; d. fully ripened cherry tomato; e. half-ripened cherry tomato; f. green cherry tomato.

For the experiments, we selected 643 tomato images for training and 161 test images for model evaluation. All images were detected and segmented.

Table 1

Number of labeled tomatoes with each category in the dataset

Tomato cultivar	Stage of ripening	Number of labeled tomatoes
Normal size	Fully-ripened	420
	Half-ripened	636
	Green	1854
Cherry tomato	Fully-ripened	1251
	Half-ripened	1020
	Green	4596
Total		9777

3.2 Evaluation metrics

We adopted the mean average precision (mAP) metric to quantitatively measure the performance of Transformer-based Mask R-CNN. Both box mAP and mask mAP were evaluated. The value of AP can be calculated by precision and recall. The sample can be divided into the four types of true positive (TP), false positive (FP), true negative (TN), and false negative (FN) for binary classification.

Precision is the overlap area of the predicted area and ground-truth divided by the total predicted area, and recall is the overlap area divided by the area of all ground-truth. Precision (P) and recall (R) are defined as $P = \frac{TP}{TP + FP}$ (9) $R = \frac{TP}{TP + FN} .$ (10)

The precision-recall (P-R) curve can be obtained by using the precision ratio as the vertical axis and the recall ratio as the horizontal axis. AP is the average precision, which represents the area enclosed by the point on the P-R curve and the coordinate axis, and the selection of this point requires balancing the values of precision and recall so that the AP value is maximized. It is calculated as

$AP = \frac{\sum Precision}{N},$ (11) where N is the number of pictures. We adopted the AP metric, which averages APs across intersection over union (IoU) thresholds, from 0.5 to 0.95, with an interval of 0.05. For box AP and mask AP, we also report AP50 and AP75 at IoU thresholds of 0.5 and 0.75, respectively. The APs of each class are independent of each other, and we average the APs of each class to get the mAP. IoU is the area of overlap between predicted and ground-truth areas divided by their area of union, representing the accuracy of detection, calculated as

$IoU = \frac{S_{overlap}}{S_{union}}$ (12) We use GFLOPs to measure model complexity of the model, where FLOPs are numbers of floating point operations, and a GFLOP is one billion FLOPs.

3.3 Ablation study

We performed ablation experiments to verify the performance of our model. Experiments were initialized with the appropriate pre-training model. The long and short edge scales of images were resized to 1333 and 800 pixels, respectively, without changing the aspect ratio. The experimental results are summarized in Table 2.

Table 2
The ablation experiment results

Model mAP^box ${mAP}_{50}^{box}$ ${mAP}_{75}^{box}$ mAP^mask ${mAP}_{50}^{mask}$ ${mAP}_{75}^{mask}$ GFlops Params

(%) (%) (%) (%) (%) (%) (M)

Mask R-CNN with ResNet50 66.8 83.5 75.3 69.3 82.8 77.0 258.27 43.77

Mask R-CNN with ResNet101 67.9 83.5 76.2 69.6 82.8 76.9 334.34 62.76

Mask R-CNN with swin-T 68.4 86.4 77.5 71.1 86.3 79.8 261.91 47.4

Improved Mask R-CNN 73.0 89.4 82.2 74.4 89.2 83.3 261.91 47.4

Model	mAP^box	${mAP}_{50}^{box}$	${mAP}_{75}^{box}$	mAP^mask	${mAP}_{50}^{mask}$	${mAP}_{75}^{mask}$	GFlops	Params
Mask R-CNN with ResNet50	66.8	83.5	75.3	69.3	82.8	77.0	258.27	43.77
Mask R-CNN with ResNet101	67.9	83.5	76.2	69.6	82.8	76.9	334.34	62.76
Mask R-CNN with swin-T	68.4	86.4	77.5	71.1	86.3	79.8	261.91	47.4
Improved Mask R-CNN	73.0	89.4	82.2	74.4	89.2	83.3	261.91	47.4

In the first row, we evaluate the performance of the original Mask R-CNN using ResNet50 as the backbone, which is the baseline of our method. The second row shows the performance of Mask R-CNN, which uses a ResNet101 backbone. Compared with the baseline, the performance improvement with ResNet101 is not significant, but the parameters and GFLOPs of the model increase substantially. In the third row, we use Swin-T as the backbone. Compared to the baseline, we see 2.9% and 3.5% gains in terms of mAP^box and mAP^mask (IoU=0.5), respectively, with slightly higher parameters and GFLOPs. In the fourth row, we use multi-scale training to resize the input so that the scale of the short edge is between 480 and 800 pixels, while the scale of the long edge is 1333 pixels. The results indicate the effectiveness of multi-scale training. Overall, our approach provides a significant performance improvement over the baseline, with slightly more parameters.

Fig. 5

Prediction results of improved Mask R-CNN. a. and b. Predicted results for normal-sized tomatoes. c. and d. Predicted results for cherry tomatoes.

3.4 Comparison with other methods

We also compare the proposed Mask R-CNN model with five typical object detection and instance segmentation methods: (1) Yolo v3 [28], (2) SSD [29], (3) Faster R-CNN [20], (4) Cascade Mask R-CNN [50], and (5) Hybrid Task Cascade (HTC) [51]. The first two of these methods perform the prediction and loss calculation of boxes only once, i.e., they have one-stage detection architectures. Other methods use a two-stage detection architecture, which takes advantage of area proposals to significantly improve localization accuracy. Both Cascade Mask R-CNN and HTC continuously optimize the results by cascading multiple detectors. Each defines positive and negative samples based on different IoU thresholds, with the output of the former detector serving as the input to the latter. Table 3 compares the performance of each model.

Table 3
Comparison with different methods

Model Backbone mAP^box ${mAP}_{50}^{box}$ ${mAP}_{75}^{box}$ mAP^mask ${mAP}_{50}^{mask}$ mAP ${mAP}_{75}^{mask}$ GFlops Params

(%) (%) (%) (%) (%) (%) (M)

Yolo v3 DarkNet53 57.1 75.5 65.5 / / / 193.95 61.55

SSD VGG16 58.6 78.1 67.4 / / / 345.5 25.12

Faster R-CNN ResNet50 66.5 83.1 75.4 / / / 206.69 41.15

Cascade Mask R-CNN ResNet50 70.0 83.6 77.5 70.0 83.1 77.2 389.22 76.81

HTC ResNet50 70.8 84.6 78.2 71.6 84.6 78.6 391.81 76.95

Improved Mask R-CNN Swin-T 73.0 89.4 82.2 74.4 89.2 83.3 261.91 47.4

Model	Backbone	mAP^box	${mAP}_{50}^{box}$	${mAP}_{75}^{box}$	mAP^mask	${mAP}_{50}^{mask}$	mAP ${mAP}_{75}^{mask}$	GFlops	Params
Yolo v3	DarkNet53	57.1	75.5	65.5	/	/	/	193.95	61.55
SSD	VGG16	58.6	78.1	67.4	/	/	/	345.5	25.12
Faster R-CNN	ResNet50	66.5	83.1	75.4	/	/	/	206.69	41.15
Cascade Mask R-CNN	ResNet50	70.0	83.6	77.5	70.0	83.1	77.2	389.22	76.81
HTC	ResNet50	70.8	84.6	78.2	71.6	84.6	78.6	391.81	76.95
Improved Mask R-CNN	Swin-T	73.0	89.4	82.2	74.4	89.2	83.3	261.91	47.4

From Table 3, it is observed that our improved Mask R-CNN with Swin Transformer is superior to other methods. Its detection and segmentation accuracy can reach 89.4% and 89.2%, respectively, which is at the IoU threshold of 0.5. The prediction results of our method are also shown in Fig. 5.

The first three methods only predict the bounding box without predicting the segmentation mask. Compared with Cascade Mask R-CNN and HTC, our model has higher detection accuracy with fewer parameters and GFLOPs. This indicates that our model is able to accurately detect and segment tomatoes from different varieties and growth stages. Our model achieves the best speed-accuracy trade-off, which is essential for tomato harvesting robots.

4 Conclusions

We presented a fruit detection and segmentation algorithm for tomatoes using Transformer-based Mask R-CNN. Our model can effectively detect normal-sized tomatoes and cherry tomatoes, and determine whether tomatoes are fully-ripened, half-ripened, or green. The model can accurately segment tomatoes in orchards. Based on that, harvesting robots can accurately identify specific types of tomatoes, so as to automatically pick fully-ripened tomatoes, for example. There are wider applications in tomato production based on our study, including harvest forecasts based on tomato maturity, pesticide spraying at specific ripening stages, and temperature control in greenhouses according to ripeness.

In developing this system, we replaced the backbone network based on the Mask R-CNN model. The Swin Transformer for feature extraction provides more improvements over CNNs. We investigated multi-scale training methods to further improve performance. Compared with other methods, ours had the best detection and segmentation performance for all kinds of tomatoes, with detection accuracy of 89.4% and segmentation accuracy of 89.2%. Our method has a small number of parameters and GFLOPs, for the best speed-accuracy trade-off. However, there are still many facts that need improvement. We will explore the detection of tomatoes under different occlusion areas by using more metrics. Also future work will integrate the proposed algorithm with a practical harvesting robot. We will compress the model by methods such as network pruning to achieve light weight and acceleration.

Footnotes

Acknowledgments

This work was supported in part by the Key Research and Development Project of Shandong Province under Grant 2019JZZY010706 and in part by the TaiShan Industrial Experts Programme under Grant tscy20200303.

References

Keren Kapach , Ehud Barnea , Rotem Mairon , Yael Edan , OhadBen-Shahar , Computer vision for fruit harvesting robots–state ofthe art and challenges ahead, International Journal ofComputational Vision and Robotics 3(1-2) (2012), 4–34.

Cheng’en Li , Yunchao Tang , Xiangjun Zou , Po Zhang , Junqiang Lin , Guoping Lian , Yaoqiang Pan , A novel agricultural machineryintelligent design system based on integrating image processing andknowledge reasoning, Applied Sciences 12(15) (2022), 7900

Hussin

, Rizon Juhari

, Wei Kang

N.G.

, Ismail

R.C.

and Kamarudin

, Digital image processing techniques for objectdetection from complex background image, Procedia Engineering 41 (2012), 340–344.

Gongal,

, Suraj Amatya , Manoj Karkee , Zhang

, Karen Lewis, , Karen Sensors and systems for fruit detection and localization: A review, Computers and Electronics in Agriculture 116 (2015), 8–19.

Alessio Plebe , Giorgio Grasso , Localization of spherical fruitsfor robotic harvesting, Machine Vision and Applications 13(2) (2001), 70–79.

Xu Huirong , Ye Zunzhong , Ying Yibin , et al. Identification ofcitrus fruit in a tree canopy using color information [j], Transactions of the Chinese Society of Agricultural Engineering 5 (2005).

Eliyahu Efim Kelman , Raphael Linker , Vision-based localisation ofmature apples in tree images using convexity, BiosystemsEngineering 118 (2014), 174–185.

Raphael Linker , Oded Cohen , Amos Naor , Determination of thenumber of green apples in rgb images recorded in orchards, Computers and Electronics in Agriculture 81 (2012), 45–57.

Yuanshen Zhao , Liang Gong , Yixiang Huang , Chengliang Liu , Robusttomato recognition for robotic harvesting using feature imagesfusion, Sensors 16(2) (2016), 173.

10.

Subhajit Sengupta , Won Suk Lee , Identification and determinationof the number of immature green citrus fruit in a canopy underdifferent ambient light conditions, Biosystems Engineering 117 (2014), 51–61.

11.

Denis Stajnko , Jurij Rakun , Michael Blanke et al. Modelling applefruit yield using image analysis for fruit colour, shape andtexture, European Journal of Horticultural Science 74(6) (2009), 260.

12.

Mohd Fadzil Abdul Kadir , Nurul Ain Najihah Yusri , Mohamed Rizon , Abd Rasid bin Mamat , Mokhairi Makhtar , Azrul Amri Jamal , Automatic mango detection using texture analysis and randomisedhough transform, Applied Mathematical Sciences 9(129) (2015), 6427–6436.

13.

Wei Ji , Dean Zhao , Fengyi Cheng , Bo Xu , Ying Zhang , Jinjing Wang , Automatic recognition vision system guided for apple harvestingrobot, Computers & Electrical Engineering 38(5) (2012), 1186–1195.

14.

Guoxu Liu , Shuyi Mao , Jae Ho Kim , A mature-tomato detectionalgorithm using machine learning and color analysis, Sensors 19(9) (2019), 2023.

15.

Duke Bulanon

, Takashi Kataoka , Hiroshi Okamoto , Shun-ichi Hata, , Development of a real-time machine vision system for the appleharvesting robot, in IEEE, SICE 2004 Annual Conference 1(2004), 595–598.

16.

Yuanshen Zhao , Liang Gong , Bin Zhou , Yixiang Huang , ChengliangLiu , Detecting tomatoes in greenhouse scenes by combining adaboostclassifier and colour analysis, Biosystems Engineering 148 (2016), 127–137

17.

Jingui Wu , Baohua Zhang , Jun Zhou , Yingjun Xiong , Baoxing Gu , Xiaolong Yang , Automatic recognition of ripening tomatoes bycombining multi-feature fusion with a bi-layer classificationstrategy for harvesting robots, Sensors 19(3) (2019), 612.

18.

Anand Koirala , Kerry Walsh

, Zhenglin Wang , Cheryl McCarthy , Deep learning–method overview and review of use for fruit detectionand yield estimation, Computers and electronics in agriculture 162 (2019), 219–234.

19.

Ross Girshick , Jeff Donahue , Trevor Darrell , Jitendra Malik Rich feature hierarchies for accurate object detection and semantic segmentation, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2014), 580–587.

20.

Shaoqing Ren , Kaiming He , Ross Girshick , Jian Sun Faster r-cnn: Towards real-time object detection with region proposal networks, arXiv preprint arXiv:1506.01497, 2015.

21.

Kaiming He , Georgia Gkioxari , Piotr Dollár , Ross Girshick Mask r-cnn, in Proceedings of the IEEE International Conference on Computer Vision (2017), 2961–2969.

22.

Suchet Bargoti , James Underwood Deep fruit detection in orchards, in 2017 IEEE International Conference on Robotics andAutomation (ICRA). IEEE (2017), 3626–3633.

23.

Inkyu Sa , Zongyuan Ge , Feras Dayoub , Ben Upcroft , Tristan Perez , Chris McCool , Deepfruits: A fruit detection system using deep neuralnetworks, sensors 16(8) (2016), 1222.

24.

Weikuan Jia , Yuyu Tian , Rong Luo , Zhonghua Zhang , Jian Lian , Yuanjie Zheng , Detection and segmentation of overlapped fruits basedon optimized mask r-cnn application in apple harvesting robot, Computers and Electronics in Agriculture 172 (2020), 105380.

25.

Yang Yu , Kailiang Zhang , Li Yang , Dongxing Zhang , , , and , Fruit detectionfor strawberry harvesting robot in non-structural environment basedon mask-R-CNN, Computers and Electronics in Agriculture 163 (2019), 104846.

26.

Joseph Redmon , Santosh Divvala , Ross Girshick , Ali Farhadi You only look once: Unified, real-time object detection, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2016), 779–788.

27.

Joseph Redmon , Ali Farhadi Yolo9000: better, faster, stronger, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2017), 7263–7271.

28.

Joseph Redmon , Ali Farhadi Yolov3: An incremental improvement, arXiv preprint arXiv:1804.02767, 2018.

29.

Wei Liu , Dragomir Anguelov , Dumitru Erhan , Christian Szegedy , Reed , Cheng-Yang Fu , Alexander Berg

Ssd: Single shot multibox detector, in European Conference on Computer Vision. Springer (2016), 21–37.

30.

Fengyun Wu , Jieli Duan , Puye Ai , Zhaoyi Chen , Zhou Yang , XiangjunZou , Rachis detection and three-dimensional localization of cut offpoint for vision-based banana robot, Computers and Electronicsin Agriculture 198 (2022), 107079.

31.

Yunchao Tang , Hao Zhou , Hongjun Wang , Yunqi Zhang , Fruitdetection and positioning technology for a camellia oleifera c. abelorchard based on improved yolov4-tiny model and binocular stereovision, Expert Systems with Applications 211 (2023), 118573.

32.

Hongjun Wang , Yiyan Lin , Xiujin Xu , Zhaoyi Chen , Zihao Wu , Yunchao Tang , A study on long-close distance coordination controlstrategy for litchi picking, Agronomy 12(7) (2022), 1520.

33.

Anand Koirala , Walsh

K.B.

, Zhenglin Wang and McCarthy

, Deeplearning for real-time fruit detection and orchard fruit loadestimation: Benchmarking of ‘mangoyolo’, Precision Agriculture 20(6) (2016), 1107–1135.

34.

Ting Yuan , Lin Lv, , Fan Zhang , Jun Fu , Jin Gao , Junxiong Zhang , WeiLi , Chunlong Zhang , Wenqiang Zhang , Robust cherry tomatoesdetection algorithm in greenhouse scene based on ssd, Agriculture 10(5) (2020), 160.

35.

Zhi-Feng Xu , Rui-Sheng Jia , Yan-Bo Liu , Chao-Yue Zhao , Hong-MeiSun , Fast method of detecting tomatoes in a complex scene forpicking robots, IEEE Access 8 (2020), 55289–55299.

36.

Yue Mu , Tai-Shen Chen , Seishi Ninomiya , Wei Guo , Intact detectionof highly occluded immature tomatoes on plants using deep learningtechniques, Sensors 20(10) (2020), 2984.

37.

Jun Sun , Xiaofei He , Minmin Wu , Xiaohong Wu , Jifeng Shen , BingLu , Detection of tomato organs based on convolutional neural networkunder the overlap and occlusion backgrounds, Machine Vision andApplications 31 (2020), 1–13.

38.

Ze Liu , Yutong Lin , Yue Cao , Han Hu , Yixuan Wei , Zheng Zhang , Stephen Lin , Baining Guo Swin Transformer: Hierarchical vision Transformer using shifted windows, arXiv preprint arXiv:2103.14030, 2021.

39.

Karen Simonyan , Andrew Zisserman Very deep convolutional networks for large-scale image recognition, arXiv preprint arXiv:1409.1556, 2014.

40.

Christian Szegedy , Wei Liu , Yangqing Jia , Pierre Sermanet , Reed , Dragomir Anguelov , Dumitru Erhan , Vincent Vanhoucke , Andrew Rabinovich Going deeper with convolutions, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2015), 1’9.

41.

Kaiming He , Xiangyu Zhang , Shaoqing Ren , Jian Sun Deep residual learning for image recognition, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2016), 770–778.

42.

Gao Huang , Zhuang Liu , Laurens Van Der Maaten , Kilian QWeinberger Densely connected convolutional networks, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2017), 4700–4708.

43.

Mingxing Tan , Quoc Le Efficientnet: Rethinking model scaling for convolutional neural networks, in International Conference on Machine Learning. PMLR (2019), 6105–6114.

44.

Ashish Vaswani , Noam Shazeer , Niki Parmar , Jakob Uszkoreit , LlionJones , Aidan Gomez

, Lukasz Kaiser , Illia Polosukhin Attention is all you need, arXiv preprint arXiv:1706.03762, 2017.

45.

Jia Deng , Wei Dong , Richard Socher , Li-Jia Li , Kai Li , Li Fei-Fei Imagenet: A large-scale hierarchical image database, in 2009 IEEE conference on computer vision and pattern recognition. IEEE (2009), 248–255.

46.

Diederik Kingma

, Jimmy Ba Adam: A method for stochastic optimization, arXiv preprint arXiv:1412.6980, 2014.

47.

Adam Paszke , Sam Gross , Francisco Massa , Adam Lerer , James Bradbury , Gregory Chanan , Trevor Killeen , Zeming Lin , Natalia Gimelshein , LucaAntiga , Alban Desmaison andreas Kopf , Edward Yang , Zachary DeVito,Martin Raison , Alykhan Tejani , Sasank Chilamkurthy , Benoit Steiner , Lu Fang , Junjie Bai , Soumith Chintala Pytorch: An imperative style, high-performance deep learning library, in Advances in Neural Information Processing Systems 32, H. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alché-Buc, E. Fox and R. Garnett, Eds. pp. 8024–8035. Curran Associates, Inc., 2019.

48.

Kai Chen , Jiaqi Wang , Jiangmiao Pang , Yuhang Cao , Yu Xiong , Xiaoxiao Li , Shuyang Sun , Wansen Feng , Ziwei Liu , Jiarui Xu , Zheng Zhang , Dazhi Cheng , Chenchen Zhu , Tianheng Cheng , Qijie Zhao , Buyu Li , Xin Lu , Rui Zhu , Yue Wu, , Jifeng Dai , Jingdong Wang , Jianping Shi , Wanli Ouyang , Chen Change Loy , Dahua Lin MMDetection: Open mmlab detection toolbox and benchmark, arXiv preprint arXiv:1906.07155, 2019.

49.

Roman Trigubenko Laboro Tomato, Laboro.AI Inc, 2020.

50.

Zhaowei Cai , Nuno Vasconcelos Cascade r-cnn: Delving into high quality object detection, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2018), 6154–6162.

51.

Kai Chen , Jiangmiao Pang , Jiaqi Wang , Yu Xiong , Xiaoxiao Li , ShuyangSun , Wansen Feng , Ziwei Liu , Jianping Shi , Wanli Ouyang et al. Hybrid task cascade for instance segmentation, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2019), 4974–4983.