Tomato pest classification using deep convolutional neural network with transfer learning,fine tuning and scratch learning

Abstract

Pests are major threat to economic growth of a country. Application of pesticide is the easiest way to control the pest infection. However, excessive utilization of pesticide is hazardous to environment. The recent advances in deep learning have paved the way for early detection and improved classification of pest in tomato plants which will benefit the farmers. This paper presents a comprehensive analysis of 11 state-of-the-art deep convolutional neural network (CNN) models with three configurations: transfers learning, fine-tuning and scratch learning. The training in transfer learning and fine tuning initiates from pre-trained weights whereas random weights are used in case of scratch learning. In addition, the concept of data augmentation has been explored to improve the performance. Our dataset consists of 859 tomato pest images from 10 categories. The results demonstrate that the highest classification accuracy of 94.87% has been achieved in the transfer learning approach by DenseNet201 model with data augmentation.

Keywords

Agriculture tomato pest deep learning convolutional neural network pre-trained network transfer learning fine tuning scratch learning

1. Introduction

Agriculture is one of the main sources of human sustenance on earth. It does not only provide necessary food for human consumption and existence but also plays a major role in the economy of the country [1]. In India, almost 70% of the population depends on farming and it is second larger producer of agriculture products [2]. Further, India accounts for 7.39% of total global agricultural output [2]. The quality and quantity of agricultural production is affected by environmental parameters like rain, temperature and other weather parameters that are beyond the control of human beings. It is a matter of concern to safeguard crops from bio-aggressors such as pests and insects which are very dangerous for the overall growth of the crops [3]. In India, approximately 18% of the crop yield is lost due to pest attacks every year which is valued around 90,000 million rupees [4]. It is almost impossible to execute the appropriate pest control at the right time in the right place without gathering information about pest activity. Conventionally, sticky traps, and black light traps are being utilized for manual pest monitoring and detection in farms. However, these techniques are less effective and more prone to cause harm to environmental friendly insects. Manual pest monitoring techniques are time-consuming and subjective to the availability of a human expert to detect the same. As a preventive measure, farmers spray pesticides in bulk quantity which are hazardous to the ecosystem [5]. Therefore, a lot of research is being carried out all around the world for better methods of pest control than the use of chemical pesticides. A program named Integrated Pest Management (IPM) has been initiated as an alternative and effective approach to pest control since 1960 [6]. IPM consultants regularly monitor the environment by counting harmful pests on crops and apply control according to the actual localization of pests. However, IPM is also a time taking process.

In recent years, image processing technologies and robotics are widely used in agriculture to reduce the workload and time of farmers. Many researches have incorporated the concept of image processing for the classification of plant leaf diseases [7, 8] and pests of various crops like Rose [9], Rice [10], Cotton [11], Maize [12], Soybean [13], Sugarcane [14] and teagarden [15]. In this context, various approaches have been proposed in literature for detection and classification of agricultural pests such as whiteflies [16], aphids and thrips [5], Honey bees [17], wasps [18], leaf miner [19], rice plant hopper [20] and many more.

In this paper, we have considered the pest of tomato plant. Tomato occupies second highest agricultural product in the Indian economy [21]. Despite its production, its loss is more due to attacks of pests. Thus, protecting tomato from pest is crucial for improving crop quality and quantity. In literature, several researches have been carried out on identification of tomato leaf diseases [22, 23, 24, 25] and its quality evaluation [26]. However, there are a handful number of literature is available on classification of tomato pests. An approach for detection of borer insect which is commonly affecting the tomato plant has been proposed using morphological features [27] and cloud computing [28]. Tomato plants are infected with two types of virus named Tomato Spotted Wilt Virus (TSWV) and Tomato Yellow Leaf Curl Virus (TYLCV). These two viruses have been identified using support vector machine (SVM) in [29]. Moreover, due to the significant improvement in deep learning technology [30], it has also been applied in agriculture field. For example, convolutional neural network (CNN) based VGG16 [31] model and transfer learning approach [32] has been presented to detect tomato pests and diseases in [33]. The transfer learning model has reported an accuracy of 89%, slightly better than VGG16 model whose accuracy value is 88%. Another transfer learning approach on Google’s Inception-V3 model was presented to classify different types of tomato pests and diseases in [34] and reported 88.90% accuracy. In [35], the authors have presented a deep learning-based approach for classification of tomato plant diseases and pests. They have experimented with three architectures: the faster region-based CNN (Faster R-CNN), region-based fully convolution network (R-FCN), and single-shot multiplex detector (SSD) with various CNN based feature extractors such as Virtual Geometry Group (VGGNet) and Residual Network (ResNet). It has been reported that the best average precision of 85.98% has been achieved using R-FCN with ResNet50. In [36], an approach has been proposed to detect tomato whitefly and its predatory bugs using a deep CNN model. The result has been compared with hand-counted insects using the yellow sticky trap method. The average classification accuracy was reported as 87.40%. In [37], a comparative study of K-Nearest Neighbour (KNN), SVM, Multilayer Perceptron (MLP), Faster R-CNN, and SSD classifiers has been presented in distinguishing Bemisia Tabacii egg and Trialeurodes Vaporariorum egg tomato pest classes. The highest classification accuracy of 82.51% has been obtained using Faster-RCNN. Dawei et al. has presented a transfer learning based pest image classification approach using AlexNet model and reported a classification accuracy of 93.84% [38]. Recently, our group has presented tomato pest image classification approach using various pre-trained deep CNN models with transfer learning technique and reported a classification accuracy of 88.83% [39]. The following observations have been made from the literature on tomato pest image classification: (i) A handful number of research works have been done on tomato pest image classification, so there is a need to explore the image-based tomato pest classification tasks; (ii) The dataset used in most of the research works is a mix of tomato plant diseases and pests, which may not result in a robust and reliable model for tomato pest classification; (iii) The performance of deep learning models on tomato pest image classification is found to be higher than shallow models, which motivates us to do the analysis with various deep learning models.

Table 1
Details of tomato pest dataset

Class label	Class name	# images	Host damage
Pest 1	BactroceraLatifrons	80	Larvae feed on the fruit pulp
Pest 2	BemisiaTabacii	80	Piercing and sucking sap from the foliage of plants which causes weakening
Pest 3	ChrysodeixisChalcites	94	Feeds on foliage and fruit of vegetables
Pest 4	EpilachnaVigintioctopunctata	94	Adult and larva feed on plant leaves
Pest 5	HelicoverpaArmigera	92	It invades young fruits. The larger larva may bore into older fruits
Pest 6	IceryaAegyptiaca	80	Damages caused due to sap depletionshoots dry up and defoliation occurs
Pest 7	LiriomyzaTrifolii	88	Damages occurred due to mining into leaves.photosynthesis ability is reduced
Pest 8	NesidiocorisTenuis	91	Adult and larva feed on stems, leaves, and flowers, shedding of flowers and pods
Pest 9	SpodopteraLitura	80	It causes damage because of their vicious eating habits
Pest 10	TutaAbsoluta	80	It severely attacks apical bud,flowers on new fruit of host
Total Number of Images		859

Figure 1.

Sample image from each class of tomato pest.

In this work, we have presented the performance of 11 state-of-the-art deep CNN architectures in three configurations: transfer learning, fine-tuning and scratch learning. The 11 models used here are: ResNet50V2, ResNet101V2, ResNet152V2, InceptionV3, Xception, InceptionResNetV2, MobileNet, DenseNet121, DenseNet169, DenseNet201 and NASNetMobile. In addition, data augmentation technique has been applied to increase the size of our dataset and avoid overfitting. The contributions of this paper are as follows: (i) application of deep CNN model with three configurations transfers learning, fine-tuning and scratch learning on tomato pest classification are first of its kind; (ii) performance comparison of 11 state-of-the-art deep CNN models on tomato pest classification; (iii) investigation of effect of data augmentation technique on the performance of deep CNN model for classification of tomato pest images.

The reminder of the paper is structured as follows: In Section 2, we have described the methodology that consists of dataset collection and preparation, CNN model with three configurations and data augmentation technique. The experimental setup has been described in Section 3. We have been presented the results and discussion in Section 4. Finally, we conclude the paper with future scope in Section 5.

2. Methodology

2.1 Dataset collection and preparation

The dataset used in this study has been collected from online sources [40, 41, 42, 43, 44]. The dataset consists of 859 tomato pest images belonging to 10 classes. All the images are in RGB color space. The details of the dataset have been provided in Table 1. Bactrocera Latifrons [45] is a pest of solanaceous crops like potatoes, tomatoes, eggplant, capsicum and chillies. In general, it can be attacked as larvae either by parasitoids or by vertebrates eating fruit. Bemisia Tabaci [46] attacks more than 500 species of plants from 63 plant families. It can damage directly, indirectly or by virus transmission. Chrysodeixis Chalcites [47] is noticed with two silver spot and golden color. The major agricultural crop hosts of this insect pest are tobacco, tomato, cotton, cruciferae, legumes, corn, soybeans, potatoes, artichokes, greenhouse crops and cauliflower. Epilachna Vigintioctopunctata [48] is observed as a serious pest of Solanaceaeous crops. It results in total crop failure due to defoliation caused by these pests. Helicoverpa Armigera [45] is intercepted repeatedly at entry port and not detected easily. The most important crop hosts are cotton, pigeon pea, chickpea, tomato, sorghum and cowpea. Icerya Aegyptiaca [49] is a sap sucking insect. Damages to the host caused by sap depletion resulting leaf drop and stunted growth. LiriomyzavTrifolii [45] is a leaf-mining insect, commonly known as the serpentine leaf miner. It is highly polyphagous and has been recorded from 25 families. It is a major pest of ornamental and vegetable crops, including beans, capsicum, potatoes and tomatoes. Nesidiocoris Tenuis [50] feeds on solanaceous crops. It has significant contributions in controlling of greenhouse pests. Spodoptera Litura [45] is one of the important pests of agricultural crops. Damages caused due to voracious eating habits of its larvae, which leads to stripping of plants. Tuta Absoluta [45] is an insect pest that causes major losses to tomato and it affects in all growing stages of egg, larvae, pupa and adult. The impact of the pest includes severe yield loss reaching 100% in case of tomato crop. Hence it is required to address these pest’s activity to reduce the crop losses to minimal.

2.2 Deep convolutional neural network models

Deep learning models, especially convolutional neural networks (CNNs) have shown great success in image classification. CNNs are made up of learnable weights and biases. The architecture of a typical CNN structure can be explained with four main layers: convolutional layer, ReLU (Rectified Linear Unit) layer, pooling layer and fully connected layer. A kernel or filter is convolved with the input and passed through the non-linear activation function ReLU and generates feature map. The pooling layer helps to reduce the spatial size of the feature map and provides translation invariance property. In deep CNN model, convolution and pooling layers are stacked alternately followed by fully connected layers at the end, which connects every neuron in one layer to every neuron in another layer. Model is trained using well-known back propagation algorithm [51]. The final layer of CNN for classification is Softmax activation function that returns a probability distribution over the target class in a multiclass classification problem.

In this paper, we have explored 11 state-of-the-art deep CNN models which are ResNet (5OV2, 101V2, 152V2) [52], InceptionV3 [53], Xception [54], InceptionResNetV2 [55], MobileNet [56], DenseNet (121, 169, 201) [57] and NASNetMobile [58]. These models are usually trained on ImageNet dataset [59] which has 1.2 million images including 1000 categories. The input size of VGG16 is 224 $\times$ 224. It has 16 layers of stacked convolution and pooling operation. VGG19 is similar to VGG16 with more depth of total 19 layers. ResNet architecture proposed residual connections of identity mappings which propagates forward and backward signals directly from one block to another. The Inception network replacing residual connections with inception block. In each inception block, convolution filters and pooling are stacked upon each other for reducing the dimensionality of output of previous layer. Then concatenated along their channels prior to providing output. Xception is inspired by InceptionV3. In this case, a ‘depthwise separable convolution is used as an inception module. InceptionResNetV2 is the combination of inception architecture and residual connections. Another model named MobileNet is used for mobile vision applications. It is based on depthwise separable filters to deepen the network and reduce computation. MobileNetV2 makes the network more lightweight by using depthwise convolution to filter the feature maps in the intermediate layer. DenseNet takes dense blocks as a unit of basic modules. The dense block passes feature maps of all subsequent layers by adding a dimension of it unlike adding pixel values in ResNet. In NASNet architecture, a Neural Architecture Search (NAS) Framework was used to train the model. This architecture identifies repeated pattern of convolutional filter banks, where each convolutional cell has the same architecture but different weights, and nonlinearities to achieve state of the art results. A new regularization technique called “ScheduledDropPath” [57] has introduced in the NASNet models to drop out each path in the convolutional filter banks with a probability that is linearly increased over the course of training.

We have explored the performance of these 11 models on tomato pest dataset by training the model in three different approaches: Transfer Learning (TL), Fine Tuning (FT) and Scratch Learning (SL). In TL approach, a model is trained on large dataset and the trained weights are used for new classification problem having small dataset. During training in TL approach, all the layers of pre-trained model are frozen except last few fully connected layers. In this way, weights of all frozen layers are unchanged while training and only weights of unfrozen layers take part in training. Similar to TL approach, the model is loaded with pre-trained weights in FT approach. However, the weights of all the layers are updated while training in FT unlike TL approach. On the other hand, the model is initialized with random weights in SL approach and weights of all the layers are updated. The concept of deep CNN model with three configurations: TL, FT and SL are represented in Fig. 2. Further, the details of all 11 models are provided in Table 2.

Table 2
Details of deep CNN models in TL, FT and SL configuration

Deep CNN model	Input shape	# convolution layer	# pooling layer	Trainable parameter in TL	Trainable parameter in FT	Trainable parameter in SL
ResNet50V2	(224,224,3)	53	5	20,490	23,539,850	23,539,850
ResNet101V2	(224,224,3)	103	5	20,490	42,549,386	42,549,386
ResNet152V2	(224,224,3)	155	5	20,490	58,208,394	58,208,394
InceptionV3	(299,299,3)	94	14	20,490	21,788,842	21,788,842
Xception	(299,299,3)	40	5	20,490	20,827,442	20,827,442
InceptionResNetV2	(299,299,3)	244	6	15,370	54,291,562	54,291,562
MobileNet	(224,224,3)	27	1	10,010	4,241,986	4,241,986
DenseNet121	(224,224,3)	119	5	10,250	6,964,106	6,964,106
DenseNet169	(224,224,3)	168	5	16,650	12,501,130	12,501,130
DenseNet201	(224,224,3)	196	5	19,210	18,112,138	18,112,138
NASNetMobile	(224,224,3)	174	44	10,570	4,243,548	4,243,548

Figure 2.

Concept of training a deep CNN model in three configurations: (a) TL; (b) FT; (c) SL.

2.3 Data augmentation

The limitation of deep learning technique is overfitting due to small dataset. To prevent overfitting and generalize the model, data augmentation (DA) can be used to enhance the size of training dataset [60]. It includes techniques like geometric transformations, color space transformations, mixing images, adversarial training and meta-learning schemes. The geometric transformation is commonly used due to its simplicity. It involves transformations such as translation, rotation, scaling, flipping and shearing of the original image that can be represented mathematically as $y=wx+b$ where $y$ denotes augmented data, $x$ denotes input data, $w$ denotes increasing factor and $b$ is bias. Here, we focused on DA based on basic image manipulations such as flipping, rotation, zooming and shifting. We have used ImageDataGenerator function to perform DA which is a popular data augmentation library available in Keras. The details of this operation is explained in Section 3.

Table 3
OA obtained using 11 state-of-the-art models with and without DA in three configurations (TL, FT, SL) on tomato pest classification

Model	TL approach		FT approach		SL approach
	Without DA	With DA	Without DA	With DA	Without DA	With DA
ResNet50V2	90.92 $\pm$ 0.66	93.13 $\pm$ 1.53	70.34 $\pm$ 1.64	80.23 $\pm$ 3.80	57.55 $\pm$ 2.93	73.13 $\pm$ 5.83
ResNet101V2	92.32 $\pm$ 1.81	94.53 $\pm$ 1.25	62.79 $\pm$ 4.02	73.95 $\pm$ 4.44	62.09 $\pm$ 2.41	73.48 $\pm$ 1.86
ResNet152V2	91.50 $\pm$ 1.33	92.43 $\pm$ 2.78	59.18 $\pm$ 18.54	60.11 $\pm$ 17.76	59.65 $\pm$ 7.17	70.93 $\pm$ 4.51
InceptionV3	91.86 $\pm$ 1.98	93.02 $\pm$ 1.21	76.16 $\pm$ 9.30	78.83 $\pm$ 9.72	70.11 $\pm$ 5.86	75.46 $\pm$ 5.33
Xception	90.00 $\pm$ 2.02	91.39 $\pm$ 1.06	82.20 $\pm$ 6.48	86.27 $\pm$ 4.99	64.76 $\pm$ 9.23	75.93 $\pm$ 6.37
InceptionResNetV2	90.34 $\pm$ 2.61	91.39 $\pm$ 0.56	82.20 $\pm$ 7.04	84.76 $\pm$ 7.18	70.91 $\pm$ 5.15	77.41 $\pm$ 2.90
MobileNet	89.53 $\pm$ 2.36	90.11 $\pm$ 2.57	82.20 $\pm$ 5.90	84.65 $\pm$ 3.49	58.80 $\pm$ 5.23	69.30 $\pm$ 5.15
DenseNet121	91.86 $\pm$ 0.73	93.83 $\pm$ 1.93	79.53 $\pm$ 5.45	84.53 $\pm$ 2.45	67.79 $\pm$ 6.56	76.85 $\pm$ 5.83
DenseNet169	93.36 $\pm$ 2.36	93.83 $\pm$ 1.74	83.72 $\pm$ 4.59	86.27 $\pm$ 3.65	70.11 $\pm$ 3.82	75.11 $\pm$ 4.72
DenseNet201	94.18 $\pm$ 2.05	94.87 $\pm$ 2.67	81.86 $\pm$ 7.13	83.25 $\pm$ 6.90	71.39 $\pm$ 2.70	78.25 $\pm$ 3.13
NASNetMobile	89.05 $\pm$ 3.66	91.16 $\pm$ 2.30	31.04 $\pm$ 14.43	36.62 $\pm$ 13.01	46.74 $\pm$ 6.24	48.25 $\pm$ 9.30

3. Experimental setup

We have trained 11 state-of-the-art deep CNN models in three configurations: TL, FT and SL for tomato pest classification. We have resized the image size to input shape of each model. For example, images are resized to 224 $\times$ 224 for models like ResNet (5OV2, 101V2, 152V2), MobileNet, NASNetMobile, DenseNet (121, 169, 201) and 299 $\times$ 299 for models like InceptionV3, InceptionResNetV2 and Xception. The next experimental setup is to replace the last fully connected layer with 1000 neurons to fully connected layer with 10 neurons. Since training a deep CNN model on small dataset exhibits data overfitting, DA is used to boost the performance and prevent likelihood of overfitting. The transformations like flipping, cropping, scaling and rotation have been performed for DA. We have empirically set certain parameters such as: rotation range as 10, zoom range as 0.15, width shift range as 0.1, height shift range as 0.1, shear range as 0.15, horizontal flip is true and fill mode is ‘nearest’. The effect of DA has been analysed by presenting the results of all 11 models with and without DA. We have split the dataset into 70% training, 10% validation, and 20% test set. Each model is trained for 100 epochs with a batch size of 8, learning rate of 0.01 and Adam optimizer. Moreover, we have run our model for five trials ( $T$ ) due to variation in classification accuracy because of random partitioning of the train, validation, and test set. Finally, the overall accuracy (OA) has been calculated by averaging the accuracy of five trials. Also, we have shown standard deviation (STD) of accuracy in five trials which demonstrate the robustness of the model. All experiments have been performed in python 3.6 with Keras framework having a Tensorflow backend. The simulation was carried out in Google Colaboratory that provides Intel(R) Xeon(R) CPU @ 2.30GHz, 13GB RAM, and NVIDIA Tesla K80 GPU.

4. Result and discussion

The paper presents an exhaustive comparison of performance of 11 state-of-the-art deep CNN models with three approaches: TL, FT and SL on tomato pest classification. Moreover, DA has been applied to avoid overfitting. Table 3 shows the OA with STD obtained using 11 models with three approaches on tomato pest dataset. Further, the results have been shown with and without DA to validate the effect of DA. Following observations have been made from Table 3: (i) The highest OA has been achieved using DenseNet201 model among all considered models in TL and SL approach and in both the cases: with and without DA. Though, the DenseNet169 has achieved highest OA in FT, the difference in OA between DenseNet169 and DenseNet201 is very small. DenseNet169 has achieved 83.72% without DA and 86.27% with DA whereas DenseNet201 model has achieved OA of 81.86% without DA and 83.25% with DA. (ii) While comparing the performance of three approaches, it can be clearly observed that TL approach has produced better performance for all 11 models in both the cases: with and without DA. The rationale behind it is that the pre-trained weights in TL helped in better classification in 100 epochs whereas FT and SL approach might require more number of epochs to effectively train the model. (iii) The improved performance has been observed with DA in all 11 models and all three approaches. (iv) Overall, the highest OA of 94.87% has been obtained using DenseNet201 model with DA in TL approach and STD of 2.67% in this case shows robustness of the model.

Table 4
Other performance parameters obtained using DenseNet201 model with DA in TL configuration for tomato pest classification

Class	Class accuracy (%)	Precision (%)	Sensitivity (%)	Specificity (%)	F1-score
Pest 1	100	91.3	96.6	98.9	0.93
Pest 2	100	98.8	100	99.8	0.99
Pest 3	88.2	95.3	96.4	99.4	0.95
Pest 4	100	100	100	100	1.0
Pest 5	94.4	85.5	90.5	98.1	0.87
Pest 6	100	98.6	93.3	99.8	0.95
Pest 7	100	93.5	94.9	99.3	0.94
Pest 8	96	97.5	95.6	99.7	0.96
Pest 9	83.3	88.3	87.3	99.1	0.87
Pest 10	93.3	94.0	91.4	99.3	0.92

Table 5

Benchmarking of our approach on tomato pest classification with literature

Authors (year)	Data size	# classes	Methodology	Classification accuracy (%)
Shijie et al. (2017) [33]	7040	10 (7 diseases and 3 pest)	VGG16 with TL	89
Fuentes et al. (2017) [35]	5000	9 (7 diseases and 2 pest)	Faster R-CNN, R-FCN, SSD	85.98 (R-FCN)
Llorca et al. (2018) [34]	2779	5 (3 pests and 2 diseases)	InceptionV3 with TL	88.9
Nieuwenhuizen et al. (2018) [36]	6900	4 (pest)	Faster R-CNN	87.4
Gutierrez et al. (2019) [37]	4331	4 (2 pests and 2 their eggs)	KNN, MLP, Faster R-CNN, SSD	82.51 (Faster-RCNN)
Pattnaik et al. (2020) [39]	859	10 pests	DenseNet169 with TL	88.83
Our approach	859	10 pests	DenseNet201 with transfer learning	94.87

Figure 3.

Training performance using DenseNet201 model with DA in TL configuration: (a) validation loss vs. epochs; (b) validation accuracy vs. epochs.

Figure 4.

ROC curve obtained using DenseNet201 model with DA in TL configuration.

For more detailed analysis, we have also calculated the following parameters: class-wise accuracy, precision, sensitivity, specificity, and F1-score using DenseNet201 model with DA in TL approach as it has produced highest OA and shown in Table 4. It has been observed that five classes of pest (Pest1, Pest2, Pest4, Pest6 and Pest7) has been accurately classified i.e., class accuracy $=$ 100%. In addition, pest4 named ‘Epilachna Vigintioctopunctata’ is correctly classified with respect to all parameters. Besides this, the performance graphs in terms of loss and accuracy are depicted in Fig. 3 for DenseNet201 model in TL approach with DA. Figure 3a, illustrates the validation loss and Fig. 3b illustrates the validation accuracy with respect to epochs while training. A receiver operating characteristic (ROC) curve for 10 classes has been presented for DenseNet201 model in TL approach with DA in Fig. 4. We can observe that the curve for all 10 classes are on the upper left corner that validates the promising performance of the model.

Lastly, we have presented a benchmarking of our approach with recent studies on tomato pest classification in Table 5. It can be observed that most of the research work carried out on tomato pest image classification utilized a dataset that is mix of diseases and pest [32, 33, 34, 36]. Further, the number of pest classes considered in literature are very low. On the other hand, this paper presented a comprehensive analysis on tomato pest image classification on 10 classes of pest dataset using 11 state-of-the-art models in three configurations (TL, FT and SL). Moreover, the effect of DA has been demonstrated. Finally, DenseNet201 model in TL configuration with DA has achieved highest classification accuracy of 94.87% and outperformed the performance reported in literature.

5. Conclusion and future scope

We have presented an extensive comparative performance analysis of 11 state-of-the-art deep CNN models for tomato pest image classification. We have implemented these 11 models with three training approaches TL, FT, and SL. Moreover, we have augmented the pest dataset to avoid overfitting and validated the effect of data augmentation in the performance of deep CNN models. We made three conclusions here: (1) TL approach outperformed than other two approaches (FT and SL) because initial state of weights exploits a large amount of visual knowledge already learned by ImageNet dataset, (2) DenseNet201 model is outperformed than other considered models, (3) Data augmentation improves the performance of all considered models and hence, it can be a useful technique in case of insufficient data. In future, we would like to explore the augmentation with Generative Adversarial Network (GAN) technique on tomato pest classification tasks.

Declaration of interest statement

Authors declare no conflict of interest.

Funding

No funding has been received for this research.

References

Deng

Ding

(2008, August). Study on precision agriculture monitoring framework based on WSN. In 2008 2nd International Conference on Anti-counterfeiting, Security and Identification. IEEE. pp. 182-185.

Sectorwise GDP contribution of India: Available online: http://statisticstimes.com/economy/sectorwise-gdp-contribution-of-india.php (Accessed on 24

{}^{\text{th}}

April, 2020).

Gondal

Khan

Y.N.

(2015). Early Pest Detection from Crop using Image Processing and Computational Intelligence. In FAST-NU Research Journal.

Dhaliwal

G.S.

Jindal

Mohindru

(2015). Crop losses due to insect pests: Global and Indian scenario. Indian Journal of Entomology. 77(2), 165-168.

Cho

Choi

Qiao

C.W.

Kim

H.Y.

Uhm

K.B.

Chon

T.S.

(2007). Automatic identification of whiteflies, aphids and thrips in greenhouse based on image analysis. Red. 346(246), 244.

Xia

Lee

(2015). Detection of small-sized insect pest in greenhouses based on multifractal analysis. Optik-International Journal for Light and Electron Optics. 126(19), 2138-2143.

Shrivastava

V.K.

Pradhan

M.K.

Minz

Thakur

M.P.

(2019). Rice plant disease classification using transfer learning of deep convolution neural network. International Archives of the Photogrammetry, Remote Sensing & Spatial Information Sciences.

Kasturiwala

S.B.

Kasturiwale

H.P.

(2020). Image superresolution technique: A novel approach for leaf diseased problems. Intelligent Decision Technologies. (Preprint), 1-11.

Boissard

Martin

Moisan

(2008). A cognitive vision approach to early pest detection in greenhouse crops. Computers and Electronics in Agriculture. 62(2), 81-93.

10.

Faithpraise

Birch

Young

Obu

Faithpraise

Chatwin

(2013). Automatic plant pest detection and recognition using k-means clustering algorithm and correspondence filters. International Journal of Advanced Biotechnology and Research. 4(2), 189-199.

11.

Zhang

Hou

Zhao

(2013). Cotton pests and diseases detection based on image processing. Telkomnika. 11(6), 3445-3450.

12.

Sena

D.G.

, Jr Pinto

F.A.C.

Queiroz

D.M.

Viana

P.A.

(2003). Fall armyworm damaged maize plant identification using digital images. Biosystems Engineering. 85(4), 449-454.

13.

Souza

T.L.

Mapa

E.S.

dos Santos

Menotti

(2011, September). Application of complex networks for automatic classification of damaging agents in soybean leaflets. In 2011 18th IEEE International Conference on Image Processing. IEEE. pp. 1065-1068.

14.

Zhao

Liu

Yao

(2009, November). Study on image recognition of insect pest of sugarcane cotton aphis based on rough set and fuzzy C-means clustering. In 2009 Third International Symposium on Intelligent Information Technology Application. IEEE. Vol. 2, pp. 553-555.

15.

Samanta

R.K.

, & Ghosh

(2012). Tea insect pests classification based on artificial neural networks. International Journal of Computer Engineering Science (IJCES). 2(6), 1-13.

16.

Dey

Bhoumik

Dey

K.N.

(2016). Automatic detection of whitefly pest using statistical feature extraction and image classification methods. International Research Journal of Engineering and Technology. 3(9), 950-959.

17.

Batra

S.W.T.

(1988). Automatic image analysis for rapid identification of Africanized honey bees. Africanized honey bees and bee mites. 260-263.

18.

Watson

A.T.

O’Neill

M.A.

Kitching

I.J.

(2004). Automated identification of live moths (Macrolepidoptera) using digital automated identification System (DAISY). Systematics and Biodiversity. 1(3), 287-300.

19.

Dake

Chengwei

(2006, August). The support vector machine (SVM) based near-infrared spectrum recognition of leaves infected by the leafminers. In First International Conference on Innovative Computing, Information and Control-Volume I (ICICIC’06). IEEE. Vol. 3, pp. 448-451.

20.

Qing

Xian

D.X.

Liu

Q.J.

Yang

B.J.

Diao

G.Q.

Jian

T.A.N.G.

(2014). Automated counting of rice planthoppers in paddy fields based on image processing. Journal of Integrative Agriculture. 13(8), 1736-1745.

21.

Atherton

Rudich

(Eds.). (2012). The tomato crop: a scientific basis for improvement. Springer Science & Business Media.

22.

Patil

J.K.

Kumar

(2011). Color feature extraction of tomato leaf diseases. International Journal of Engineering Trends and Technology. 2(2), 72-74.

23.

Prince

Clarkson

J.P.

Rajpoot

N.M.

(2015). Automatic detection of diseased tomato plants using thermal and stereo visible light images. PloS One. 10(4), e0123262.

24.

Brahimi

Boukhalfa

Moussaoui

(2017). Deep learning for tomato diseases: Classification and symptoms visualization. Applied Artificial Intelligence. 31(4), 299-315.

25.

Mokhtar

El Bendary

Hassenian

A.E.

Emary

Mahmoud

M.A.

Hefny

Tolba

M.F.

(2015). SVM-based detection of tomato leaves diseases. In Intelligent Systems’ 2014. Springer, Cham. pp. 641-652.

26.

Gastélum-Barrios

Bórquez-López

R.A.

Rico-García

Toledano-Ayala

Soto-Zarazúa

G.M.

(2011). Tomato quality evaluation with image processing: A. African Journal of Agricultural Research. 6(14), 3333-3339.

27.

Prathibha

G.P.

Goutham

T.G.

Tejaswini

M.V.

Rajas

P.R.

Balasubramani

(2014). Early pest detection in tomato plantation using image processing. International Journal of Computer Applications. 96(12).

28.

Rupanagudi

S.R.

Ranjani

B.S.

Nagaraj

Bhat

V.G.

Thippeswamy

(2015, January). A novel cloud computing based smart farming system for early detection of borer insects in tomatoes. In 2015 International Conference on Communication, Information & Computing Technology (ICCICT). IEEE. pp. 1-6.

29.

Mokhtar

Ali

M.A.

Hassanien

A.E.

Hefny

(2015). Identifying two of tomatoes leaf viruses using support vector machine. In Information Systems Design and Intelligent Applications. Springer, New Delhi. pp. 771-782.

30.

Buntine

W.L.

(2020). Machine learning after the deep learning revolution. Frontiers Comput. Sci. 14(6), 146320.

31.

Simonyan

Zisserman

(2014). Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556.

32.

Talo

Yildirim

Baloglu

U.B.

Aydin

Acharya

U.R.

(2019). Convolutional neural networks for multi-class brain disease detection using MRI images. Computerized Medical Imaging and Graphics. 78, 101673.

33.

Shijie

Peiyi

Siping

(2017, October). Automatic detection of tomato diseases and pests based on leaf images. In 2017 Chinese Automation Congress (CAC). IEEE. pp. 2537-2510.

34.

Llorca

Yares

M.E.

Maderazo

(2018). Image-based pest and disease recognition of tomato plants using a convolutional neural network. In Proceedings of international conference technological challenges for better world.

35.

Fuentes

Yoon

Kim

Park

(2017). A robust deep-learning-based detector for real-time tomato plant diseases and pests recognition. Sensors. 17(9), 2022.

36.

Nieuwenhuizen

A.T.

Hemming

Suh

(2018). Detection and classification of insects on stick-traps in a tomato crop using Faster R-CNN. Proceedings of the Netherlands Conference on Computer Vision NCCV18. http://nccv18.nl/.

37.

Gutierrez

Ansuategi

Susperregi

Tubío

Rankić

Lenža

(2019). A Benchmarking of Learning Strategies for Pest Detection and Identification on Tomato Plants for Autonomous Scouting Robots Using Internal Databases. Journal of Sensors, 2019.

38.

Dawei

Limiao

Jiangong

Jiyue

Hongfei

Zhongzhi

(2019). Recognition pest by image-based transfer learning. Journal of the Science of Food and Agriculture. 99(10), 4524-4531.

39.

Pattnaik

Shrivastava

V.K.

Parvathi

(2020). Transfer Learning-Based Framework for Classification of Pest in Tomato Plants. Applied Artificial Intelligence. 1-13.

40.

Flickr, the online photo management and sharing application in the world. Photos of pests available online: (https://www.flickr.com/search/?text=helicoverpa%20armigera, (accessed on 10th Dec 2018).

41.

IPM Images, The Center for Invasive Species and Ecosystem Health, Last updated in 2018. Photos of pests available online: https://www.ipmimages.org/browse/Areathumb.cfm?area=63 (accessed on 18th Dec 2018).

42.

Insect Images, The Entomology Society of America and USDA Identification Technology Program, Last updated in 2018. Photos of pests available online: https://www.insectimages.org/search/action.cfm?q=spodoptera+litura (accessed on 22nd Dec 2018).

43.

The National Bureau of Agricultural Insect Resources (NBAIR), Insects in Indian Agro Ecosystem, 2013. Photos of pests available online: http://www.nbair.res.in/insectpests/Bactrocera-latifrons.php (accessed on 25th Dec 2018).

44.

The Tamil Nadu Agricultural University (TNAU), established in 1971. Pests of Tomato, available online: http://agritech.tnau.ac.in/crop_protection/crop_prot_crop_insect-veg_tomato.html (accessed on 5th Jan 2019).

45.

BactroceraLatifrons, HelicoverpaArmigera, LiriomyzaTrifolii, Spodopteralitura, TutaAbsoluta: https://www.cabi.org/isc/datasheet/8719, https://www.cabi.org/isc/datasheet/26757, https://www.cabi.org/isc/datasheet/30965, https://www.cabi.org/isc/datasheet/44520, https://www.cabi.org/isc/datasheet/49260.

46.

BemisiaTabaci: http://www.extento.hawaii.edu/kbase/Crop/Type/b_tabaci.htm.

47.

Chrysodeixischalcites: https://pest.ceris.purdue.edu/pest.php?code=ITBCQGA.

48.

EpilachnaVigintioctopunctata: https://www.plantwise.org/knowledgebank/datasheet/21518.

49.

IceryaAegyptiaca: https://www.plantwise.org/KnowledgeBank/datasheet/28426.

50.

Calvo

Bolckmans

Stansly

P.A.

Urbaneja

(2009). Predation by nesidiocoristenuis on bemisiatabaci and injury to tomato. BioControl. 54(2), 237-246.

51.

LeCun

Bengio

Hinton

(2015). Deep learning. Nature. 521(7553), 436-444.

52.

Zhang

Ren

Sun

(2016, October). Identity mappings in deep residual networks. In European conference on computer vision. Springer, Cham. pp. 630-645.

53.

Szegedy

Vanhoucke

Ioffe

Shlens

Wojna

(2016). Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 2818-2826.

54.

Chollet

(2017). Xception: Deep learning with depthwise separable convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 1251-1258.

55.

Szegedy

Ioffe

Vanhoucke

Alemi

A.A.

(2017, February). Inception-v4, inception-resnet and the impact of residual connections on learning. In the Thirty-First AAAI Conference on Artificial Intelligence.

56.

Howard

A.G.

Zhu

Chen

Kalenichenko

Wang

Weyand

Adam

(2017). Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861.

57.

Huang

Liu

Van Der Maaten

Weinberger

K.Q.

(2017). Densely connected convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 4700-4708.

58.

Zoph

Vasudevan

Shlens

Q.V.

(2018). Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 8697-8710.

59.

Krizhevsky

Sutskever

Hinton

G.E.

(2012). Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems. pp. 1097-1105.

60.

Building powerful image classification models using very little data By Francois Chollet, available online: https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html, accessed on 5th June 2020.

Tomato pest classification using deep convolutional neural network with transfer learning,fine tuning and scratch learning

Abstract

Keywords

1. Introduction

Table 1 Details of tomato pest dataset

2.1 Dataset collection and preparation

2.2 Deep convolutional neural network models

Table 2 Details of deep CNN models in TL, FT and SL configuration

Table 3 OA obtained using 11 state-of-the-art models with and without DA in three configurations (TL, FT, SL) on tomato pest classification

4. Result and discussion

Table 4 Other performance parameters obtained using DenseNet201 model with DA in TL configuration for tomato pest classification

Declaration of interest statement

Funding

References

Table 1
Details of tomato pest dataset

Table 2
Details of deep CNN models in TL, FT and SL configuration

Table 3
OA obtained using 11 state-of-the-art models with and without DA in three configurations (TL, FT, SL) on tomato pest classification

Table 4
Other performance parameters obtained using DenseNet201 model with DA in TL configuration for tomato pest classification