A study on plant recognition using conventional image processing and deep learning approaches

1 Introduction

In agriculture, plant species identification is used for weed detection [5], growth estimation, and plant disease classification [1]. Also, plants are used as medicines providing solutions to diabetes [4] and cardiovascular diseases [12]. In plant species recognition [2], leaf plays an important role compared to other parts like flower, seeds and stem. Computer vision techniques are utilized in automatic plant identification and recognition. Numerous mobile applications such as Pl@ntNet [11], leafsnap [13] are also developed.

Shape is one of the main characteristics to classify objects. Hu et al. [9] proposed a shape descriptor known as Multiscale distance matrix (MDM). MDM is a global based contour approach. MDM is invariant to rotation, translation, scaling, and bilateral symmetry. Decomposed Newton’s Method (DNM) and Maximum Margin Criterion (MMC) are applied for dimensionality reduction and further nearest neighbor (1NN) classifier is used. This method is tested for two datasets namely, Swedish Leaf Dataset and ICL (Intelligent Computing Laboratory) leaf dataset.

Zhao et al. [29] proposed a counting based shape descriptor to recognize simple and compound leaves. Independent-Inner Distance Shape Context (I-IDSC) measures the count of active shape pattern rather than considering matching features. Nearest neighbor classifier is used for classification. I-IDSC descriptor is tested over five datasets namely, Swedish leaf, ICL, Smithsonian, Plumbers Island, and their own leaf dataset formed from 54 species of Hong Kong.

Naresh et al. [18] proposed a modified Local Binary Pattern (LBP) for feature extraction and nearest neighbor classifier for medicinal plant classification. This method is tested with several standard leaf datasets and a collected dataset from Mysore, India. Tomar et al. [27] observed that the directed acyclic graph multi-class least square support vector machine (DAG-MLSTSVM) classifier performed better than artificial neural network and support vector machine. Prior to classification, leaf features are extracted based on shape and texture (21-d). Further, Hybrid Feature Selection (HFS) is carried out to identify the best features.

Ghazi et al. [6] applied transfer learning over LifeCLEF plant dataset with the help of pre-trained models like AlexNet, GoogleNet and VGGNet. For all these deep convolutional neural networks, fine tuning is performed and various parameters are analyzed after data augmentation. Parameters like batch size and number of iterations are analyzed. Lee et al. [14] discussed that Convolutional Neural Network (CNN) is used to learn leaf features and further gain knowledge based on selective features using Deconvolution Network (DN) approach. The authors observed that leaf veins help in more accurate plant identification than leaf shape. Learning features from hybrid local-global methods with deep learning performs better recognition than other techniques.

Sun et al. [23] proposed a 26-layer ResNet (Residual Network) model for plant identification. BJFU100 dataset is used and it consists of 10000 images of 100 ornamental plant species found in Beijing Forestry University campus. For experimental analysis, BJFU100 and Flavia datasets are utilized. In deep residual networks, 18, 26, 34, and 50 layers are considered. Amongst the four set of layers considered, ResNet26 outperformed the other three models. For experimental training, learning rate is set to 0.001. Flavia dataset accuracy (99.65%) result is compared with other approaches like Radial Basis Probabilistic Neural Network (RBPNN), Support Vector Machine (SVM), Deep Belief Network with dropout (DBN) and ResNet26. ResNet26 architecture produced an accuracy of 91.78% recognition rate for BJFU100 dataset. Barre et al. [3] developed a LeafNet, a CNN-based plant identification system. The LeafNet consisted of five sets of 2 convolutional layers and 1 max-pooling layer followed by 1 convolution, 1 max-pooling layer and 3 fully connected layers. LeafNet is tested over Leafsnap, Foliage and Flavia datasets.

Based on the extensive literature survey, it is identified that the reported work on plant recognition over Indian plant species are sparse. Also, numerous research works are carried out using the features such as shape, texture, color, morphological or physiological features. Reported works on plant species recognition using deep learning architecture are limited. Hence in this paper, an investigation is performed using traditional methods and deep learning architectures in order to achieve higher plant recognition rate.

2 Methodology

Two approaches are used for plant species recognition, namely, traditional method and deep learning methods as depicted in Fig. 1. The conventional image classification steps include preprocessing, feature extraction and classification using the machine learning classifiers. Feature extraction includes extraction of shape features, texture features and color features from leaves of plant images. These features are known as handcrafted features. Local Binary Pattern (LBP) [19] and Haralick texture features [7] are used to extract the texture information. Hu moments [10] are used for shape extraction and color channel statistics (mean and standard deviation of three color channels) for color information. To the extracted features, classification is carried out using machine learning techniques such as Logistic Regression (LR), K- Nearest Neighbour (KNN), Classification and Regression Tree (CART), Random Forest classifier (RF), linear discriminant analysis (LDA), Bagging Classifier (BC) and Naïve Bayes (NB) classifier [15, 16].

OPEN IN VIEWER

Fig.1

Plant classification methods.

In deep learning method, CNN based pre-trained models such as VGG 16, VGG 19, Inception-v3 and Inception-ResNet-v2 is used for feature extraction. These models have been trained on ImageNet dataset and their weights are free to use. ImageNet dataset contains about 1.2 million images. Pre-trained model weights are used as initial weights in deep learning architectures to extract the features from the input image. To the features extracted from pre-trained model, machine learning classification techniques is applied.

The Convolutional Neural Network [3] consists of several convolution layers, max pooling layer and a fully connected layer (FCL) as shown in Fig. 2. Convolutional layer is used as a feature extractor. Max pooling layer is used to reduce the dimension of the extracted feature vector. The fully connected layer is used to convert the feature map to 1-D feature vector and also, it is used as a classification layer. The deep learning models such as VGG 16, VGG 19, Inception-v3, and Inception-ResNet-v2 are developed based on the CNN principle.

OPEN IN VIEWER

Fig.2

Convolutional neural network.

VGG 16 [21] has 16 weight layers containing two sets of two convolution layers with max pooling, two sets of three convolution layers with max pooling followed by three fully connected layers. Similar to VGG 16, VGG 19 [21] has two sets of two convolution layers with max pooling, three sets of four convolution layers with max pooling followed by three fully connected layers. There are 138 and 144 million parameters in VGG 16 and VGG 19, respectively. In both the architectures, the width of layers starts from 64 and increases by 2 times after max pooling till 512 is reached. Also, both the models use ReLU as activation function in all layers and uses softmax for the final fully connected layer. Computational complexity of VGGNet is greater compared to other models.

‘Feature pooling’ is a concept used in Inception-v3 [25] that uses 1×1, 3×3 or 5×5 convolutions to collect maximum feature from each convolution. Inception-v3 uses stem (contains few convolutional layers and max pooling layers), few sets of filter concatenation and fully connected layers.

Inception-ResNet-v2 [24] merges both the concepts of Inception-v3 and ResNet [8] architectures. Inception-ResNet-v2 has stem as in Inception-v3 and Residual blocks as in ResNet model. But, inside every residual block, filter concatenation is carried out and their filter size varies for these residual blocks.

Three standard datasets (Folio [17], Swedish leaf [22], Flavia [28]) and one real-time dataset are used in the experimental studies. Swedish leaf dataset is a standard dataset prominent because of its clarity. Swedish leaf dataset contains 15 different classes with 75 images in each class. Totally, there are 1125 images in this dataset.

Flavia dataset contains 32 classes of leaves and 1907 images. Folio database has 637 images in 32 classes. These two datasets have uneven number of images in each class. Hence, for Flavia dataset 50 images are used in each class (1600 images). Similarly, for folio dataset 18 images in each class (576 images) are used. The Real-time dataset is named as Leaf12 dataset. Twelve plant species images are collected and each class contains 320 images. It is photographed under different illumination conditions, color backgrounds, viewpoints and orientations using a portable camera. The list of plants in Leaf12 dataset and their sample images are shown in Fig. 3.

OPEN IN VIEWER

Fig.3

Leaf12 samples.

Plant species recognition rate is determined using traditional methods and deep learning methods for three standard datasets (Folio, Flavia and Swedish leaf) and one real-time dataset (Leaf12). The implementation is carried out using Python language with the help of OpenCV package. For neural network, Keras package with theano as backend is used. Train and test size for analysis of results is set as 70% and 30%, respectively. The results of various datasets including Leaf12 dataset for traditional methods and pre-trained models are discussed in this section.

Plant species recognition is carried out by two approaches namely, traditional methods (feature extraction followed by classifier) and deep learning method (pre-trained models with machine learning classifiers). Four different datasets (Folio, Swedish, Flavia and Leaf12) are considered in this studies. From the experimental investigation, it is observed that the deep learning model yielded a higher accuracy compared to that of conventional methods for all datasets considered. Logistic regression classifier with pre-trained models resulted in an improved accuracy compared to that of other classifiers with pre-trained models. VGG 16 or 19 deep learning architectures with LR classifier resulted in higher accuracy compared with Inception-v3 and Inception-ResNet-v2. Maximum plant recognition rate obtained for different datasets are listed in Table 6 given below. Further, the accuracies can be improved by increasing the number of images using data augmentation methods.