Leaf disease classification with Multiple-model deep learning

3.1 MnLeaf framework

In this study, we present a sophisticated multiple-model neural network aimed at effectively classifying various leaf diseases, named MnLeaf framework. Our proposed methodology entails the integration of four distinct neural network architectures. Specifically, we employ the ResNext50 architecture to facilitate precise leaf disease detection in scenarios involving high-quality images. In cases where image quality is relatively lower, we deploy the Vision Transformer (ViT) architecture to ensure accurate disease classification. To address the challenges posed by noisy images, we introduce the EfficiencyNetB5 architecture, which serves as both a teacher and a student model. Additionally, we leverage the capabilities of the MobilenetV3 architecture to extract pertinent segmentation features, as visually depicted in Figure 2. Through this comprehensive approach, our study contributes to the advancement of leaf disease classification using cutting-edge neural network architectures and carefully curated datasets.

OPEN IN VIEWER

Fig. 2

Overall architecture of the proposed method.

The architecture of the multi-channel deep learning model is proposed to combine ResNeXt50, EfficientNet B5, and MobileNet V3 for feature extraction, and ViT layers for sequence prediction in four channels, as fully connected as illustrated in Figure 2. Each channel or head consists of a stack of Conv1D layers, followed by max-pooling layers. The Conv1D layers, configured with 128 filters, directly map and abstract the sensor inputs to extract features. Feature extraction is achieved through convolutional operators applied to kernels, and feature maps are computed as described in Table 3 below. To enable feature extraction at different resolutions, we employed two layers, ResNeXt50 and ViT, for the Conv1D layers of corresponding channels. The outputs from each Conv1D layer are then fed into max-pooling layers, reducing the size of learned features by summarizing them into separate elements without compromising accuracy. The EfficientNet B5 layer has an output configuration of 128 units for the Conv1D layers and max-pooling layers. The advantage of this is that the EfficientNet B5 layer is well-suited for learning the input image noise, leading to improved image classification. This model helps minimize overfitting and enhances model accuracy. Finally, the MobileNet V3 layer is employed to learn segmentation features for identifying the objects to be classified, in this case, leaves and leaf diseases. The outputs from the four channels are flattened and then concatenated within the model. They pass through three fully connected layers, capable of generating interpretable features into different classes. The final output of the model is from a dense layer with a softmax activation function to compute the probability distribution over the predicted operational classes.

In the next section, we will present in detail the used features and the architecture of the four networks.

To streamline and enhance the quality of input records, we performed three preprocessing steps (show in Figure 2), including data dimension reduction, adding noise to the data, and generating segmentation images. Firstly, we applied several preprocessing techniques to reduce the data dimension. For instance, we employed PyTorch functions to reduce the image size from 512×512 to 224×224 while retaining essential features. Additionally, we utilized downsampling techniques in convolutional neural networks (CNNs) to further reduce the data dimension.

After reducing the data dimension, we introduced image noise using the GrandInversion [63] technique to add noise. GrandInversion utilizes ResNet-50 as a backbone to introduce random noise and generate a natural-looking image. GrandInversion reconstructs the input from gradients as part of an optimization process. The addition of noise results in two images: (i) an origin image; and (ii) an image with noise.

Finally, we carried out the generation of segmentation images to isolate individual objects, with a focus on objects affected by diseases, such as plant leaves. We employed a segmentation technique [64] developed by Facebook’s research team. This technique effectively separates all objects within the image, after which we processed the data to select only the diseased plant leaves and classify them accordingly. The primary objective of our segmentation approach is to generate binary masks for each leaf of the plant, for each type of disease on the leaves, and associate it with a specific plant based on real agricultural field images. Therefore, we perform simultaneous segmentation of individual leaf samples and plants. This enables us to determine the shape and size of each leaf, as well as the condition of disease on the leaves, which is highly suitable for conducting morphological analysis.

3.3 ResNeXt50

The conventional approach to enhance model accuracy involves deepening or expanding the network architecture. However, as the number of hyperparameters (such as channels, filter sizes, etc.) increases, challenges in network design and computational costs also escalate. Building upon the foundation of ResNet, ResNeXt integrates the block-stacking strategy of ResNet with the grouped convolutions of the Inception structure. This amalgamation creates a synthesis approach that elevates the accuracy of the recognition model without amplifying its complexity. Through this strategy, ResNeXt achieves improved classification efficacy by employing a simplified topology, eliminating the need for additional parameters. Experimental results demonstrate that ResNeXt can achieve superior classification performance while utilizing a more straightforward architecture, thereby avoiding the necessity for parameter augmentation. Table 1 presents the detailed architecture of the proposed network.

The proposed architecture in Table 2, referred to as ViT, is specifically devised for the purpose of classifying objects with correlations between similar classes and dissimilar classes. While global classification can be achieved through relatively simple global cues, achieving finer classification necessitates highly localized discriminative regions. In this study, to capture these subtle distinguishing features, prototypes have been applied to condensed image patches. This approach aims to differentiate between categories at a more detailed level, allowing for more precise classification by focusing on localized features.

The scaled dot-product attention serves as a pivotal element within the Multi-Head Self Attention layer (MHSA) [51] of the Transformer architecture. In the MHSA, a set of queries Q ∈ ℝ N × d , keys K ∈ ℝ N × d , and values V ∈ ℝ N × d is first generated through appropriate projections. Subsequently, the query vector q ∈ ℝ d is compared to each individual key vector within K. The resultant output is determined by the weighted summation of a collection of N value vectors v, with the weighting guided by the similarity scores derived from this matching process. This mechanism is commonly referred to as the scaled dot-product attention mechanism:

Attention ( Q , K , V ) = Softmax ( QK T d ) V(1)

To mitigate the issue of exceedingly small gradients and enhance training stability, every element within the QK ^T matrix is multiplied by the constant factor 1 d , which effectively rescales the values into a more standardized range.

We synergistically combined the method of proportional scaling to optimize parameter efficiency and training speed for EfficientNetB5 models. In addition, we integrated the concept of Noisy Student training to implement a semi-supervised learning approach that capitalizes on self-training and knowledge distillation principles, as presented in Fig. 2. This involves training a teacher model on labeled images to produce pseudo labels for unlabeled images. Subsequently, a student model is trained using a blend of labeled and pseudo-labeled images. Notably, this process employs student models of equal or larger size and introduces controlled noise to the student during the learning phase, resulting in further performance enhancement.

3.6 MobilenetV3

In this section, we introduced the utilization of MobileNetV3 as the foundational network architecture for the mobile semantic segmentation task. Concurrently, we employed specialized segmentation heads known as R-ASPP, as initially proposed in [xxx]. R-ASPP represents a streamlined adaptation of the Atrous Spatial Pyramid Pooling module [xxx], focusing on two distinct branches encompassing a 1 × 1 convolution and a global-average pooling operation. To harness richer features, we implemented atrous convolution within the final block of MobileNetV3. Additionally, we integrated a skip connection from low-level features to ensure the incorporation of finer-grained details.

3.7 Fusion

Given an input image, after passing through four models, we obtain a predictive label that amalgamates distinctive feature characteristics from each deep learning model. Tailoring the fusion process according to the method we propose is expected to enhance the accuracy in classifying diseased tree leaves. We have synthesized these processes into an algorithm outlined in Table 3.

Finally, our model has gained significant improvements in performance compared to the previous version at almost no computational cost. In detail, the new non-linearity activation called h-swish has fixed some weakness of its base by replacing the computationally expensive sigmoid with a piecewise linear analogue (RELU6):

swish ( x ) = x · σ ( x )(2) h - swish ( x ) = x RELU 6 ( x + 3 ) 6(3)

4.1 Dataset

The model is trained and evaluated using expansive datasets: the Cassava Leaf Disease Classification dataset, comprising 15,000 images, and the TLU Leaf Disease dataset (show in Fig. 3), comprising 2,500 images. The TLU Leaf Disease dataset comprises 5 labels (bamboo leaf disease, verticillium wilt, leaf scorch, turning yellow, plant wilting), each containing 500 images. In Figure 3, each row corresponds to 3 sample images of the same label. Both datasets encompass five distinct labels corresponding to different diseases.

OPEN IN VIEWER

Fig. 3

Samples images from the own collected dataset (TLU-Leaf).

The experimental outcomes were evaluated utilizing metrics including accuracy, F1 score, precision, recall, Kappa, and the confusion matrix. These metrics can be computed as follows:

precision = TP TP + FP(4)

recall = TP TP + FN(5)

accuracy = TP + TN TP + TN + FP + FN(6)

F 1 = 2 × precision × recall precision + recall(7)

Kappa = 2 × ( T 3 - F 4 ) ( T 1 + T 2 + 2 × T 3 + F 1 + F 2 + F 3 )(8) where TP is the true positive, TN is the true negative, FP is the false positive, and FN is the false negative number, T₁ = TP × FN, T₂ = TP × FP, T₃ = TP × TN, F₁ = FN × TN, F₂ = FP × FP, F₃ = FP × TP, and F₄ = FN × FP.

We also computed the Area Under the Receiver Operating Characteristic Curve (AUC) as a metric for assessing diagnostic accuracy.

We conduct training for our models using a synchronous training configuration on four Tesla M10 GPUs. We employ the PyTorch framework alongside the standard TensorFlow, incorporating a momentum of 0.9. The initial learning rate is set at 0.01, coupled with a batch size of 32 (equivalent to 8 images per chip). A learning rate decay rate of 0.001 is applied every three epochs. To enhance regularization, we integrate a dropout rate of 0.8 and an L2 weight decay of 1e-5. Our image preprocessing protocol aligns with the approach detailed in the backbone framework. Additionally, we integrate exponential moving averaging with a decay factor of 0.9999. Notably, all convolutional layers within our architecture incorporate batch-normalization layers, with an average decay rate of 0.99.

4.4 Experiment setup

We design our extensive empirical study to answer the following five key research questions (RQs):

RQ1: How much does MnLeaf improve deep learning performance compared to classical deep learning methods?

In RQ2, we carried out a total of six distinct scenarios, which are outlined in Table 4. For RQ1, we showcase the experiments conducted on the three foundational network baselines. Furthermore, for both RQ3 and RQ4, we employed the two synthetic datasets to assess the performance of the models in the presence of noise. The results will be averaged over experimental runs on two datasets.