Pollen grains detection of four plant species of Yucatan using deep learning

Abstract

Yucatan has a variety of plant species of melliferous importance. The honey produced in Yucatan has several special properties that make it one of the most demanded internationally. Analyzing the pollen grains present in honey is essential to determine its quality and identify its plants of origin. This study is a time-consuming process that must be carried out by highly trained palynologists. In this work, we propose an improved model based on a fully convolutional neural network for the automatic detection of pollen grains in microscopic images of four plant species of Yucatan to contribute to the analysis of the honey designation of origin.

Keywords

Pollen analysis object detection palynology deep learning

1 Introduction

The state of Yucatan, located in the southeast of Mexico, houses a wide variety of plant species essential for honey production. Yucatan’s honey has distinctive characteristics, such as color, flavor, and aroma, that make it exceptional, and therefore, increase its popularity and demand in the international markets. Pollen grains can be found in honey, and thanks to their rich morphological diversity between species, they can be analyzed to assess the quality of the honey as well as determine the botanical origin of the plant species that produced it [3]. However, the process of identifying and counting pollen grains requires the expertise of trained specialists through visual inspection using microscopes. This task may be slow and take much time to complete. Therefore, this work proposes to automate the identification of pollen grains by using deep learning models to accelerate and improve the accuracy of the process.

Currently, there are several computational methods for pollen grains classification using machine learning and pattern recognition techniques [18]; however, there are few works focused on their automatic detection (localization and classification). For instance, in [7], Martínez et al. used second-order texture feature extraction techniques from the gray level co-occurrence matrix (GLCM) at multiple scales. These methods were applied on microscopic images of pollen grains. The extracted features served as input data for an artificial neural network used in classification. This approach yielded an accuracy rate of 88% for correct classification. However, this classification is only based on a reduced set of features that may not be sufficient for proper differentiation between species. On the other hand, in [5], He et al. proposed an unsupervised method for the automatic analysis of pollen grains in unlabeled images. Their database consists of 650 pollen granule images collected from honey samples. They used a convolutional neural network based on the YOLO structure [12] to generate a set of bounding boxes for every object detected in the image. Based on the bounding box coordinates, image fragments of potential grains were extracted. Subsequently, a convolutional neural network, incorporating the VGG-16 architecture [15], automatically extracts features from these image fragments. Finally, the K-means algorithm is selected to recognize similar groupings based on the extracted features. This method aims to identify clusters of pollen grains with analogous features in unlabeled images. This method does not classify image objects by species but instead focuses on grouping objects based on their similarities. In [9] Murkute classifies pollen grains by an ensemble of fully convolutional neural networks (FCNNs) based on the EfficientNet architecture [16]. These models were trained and tested on the publicly available Pollen-13K database [1], with 13,000 images of pollen grains. Using this approach, the models achieved an accuracy of 97.28% for classifying individual pollen grains. This method can be used only for single object classification, this is, images containing one pollen granule only. If multiple models are running in an ensemble, the response time will increase. Additionally, this study has two layers of fully connected neural networks at the end of each module that are based on EfficientNet. Unfortunately, this slows down the inference process even further.

In this work, we introduce a more efficient deep learning model based on the architectures Res2Unet [10] and YOLOv4 [2] to achieve real-time detection of pollen grains of four beekeeping-relevant species in the state of Yucatan (see Fig. 2).

2 Methodology

2.1 Dataset

The database used in this study consists of microscopic images obtained from the Laboratorio de Ciencia de Alimentos del Campo Experimental Mocochá del Centro de Investigación Regional del Sureste. The dataset comprises 370 images of pollen grains derived from four plant species of beekeeping importance in the state of Yucatan. The plant species are shown in Fig. 1, and are the following: Box Catzín (Acacia gaumeri), Chaká (Bursera simaruba), Jabín (Piscidia piscipula), and Tajonal (Viguiera dentata). Table 1 shows detailed dataset distribution for all pollen grain images. All images were acquired using an optical Motic microscope equipped with 10×, 40× and 100× objectives. The images were taken in RGB format and had a resolution of 3664 × 2748 pixels. The database is available at http://tinyurl.com/5n7bk3yr. In order to speed up the training process and reduce the GPU memory occupied, the dimensions of these images were reduced by 25% of their original size, resulting in a final size of 916 × 687 pixels.

Fig. 1

Pollen grains samples of the four species used in this work. (a) Box catzín; (b) Chaká; (c) Jabín; (d) Tajonal.

Table 1

Pollen images distribution for training, validation and test subsets

Class	Training (80%)	Validation (10%)	Test (10%)	Pollen grain count	Total images
Box catzín	76	9	11	265	96
Chaká	100	12	14	334	126
Jabín	41	5	6	601	52
Tajonal	76	9	11	426	96

Fig. 2

YOLO-based model architecture diagram.

2.2 Deep learning models

There exists a diverse range of neural network architectures that are specifically designed for object detection. Among these architectures, the YOLO (You Only Look Once) network [12] has gained popularity due to its fast detection speed and comparable accuracy to other complex networks that require several seconds to perform detection. Consequently, the YOLO network is well-suited for real-time object detection applications. The YOLO neural network is a single-stage convolutional model that addresses the object detection problem by treating it as a regression problem. Since its first release, there have been multiple versions of this architecture that have incorporated incremental enhancements, resulting in improved performance and faster detection [2, 19].

The YOLO system is comprised of three primary components, namely the backbone module responsible for feature extraction, the neck module responsible for combining these extracted features, and the head module responsible for detection. The backbone module is tasked with extracting significant features from the input image through a series of convolutions. The neck module combines these extracted features to generate complex and abstract representations. Lastly, the head module is responsible for detecting the objects, producing a vector that includes the coordinates of the bounding boxes covering the objects within the image. One notable benefit of the YOLO architecture is its flexibility in terms of the backbone module. It allows for the substitution of the original module (CSPDarkNet-53) with a different one, enabling the training of a model specifically tailored for a particular set of objects. This adaptability enhances efficiency, as the CSPDarkNet-53 architecture is primarily designed for the detection of diverse objects, including those found in the ImageNet dataset [4, 13].

One strategy for improving the ability of a fully connected neural network (FCNN) to identify and analyze patterns in images is to increase the number of layers comprising the network, hence increasing its depth. As a result, there is an increase in the quantity of parameters that must undergo optimization through the gradient descent procedure, so imposing a limitation on the maximum number of layers that may be added to a given model. Due to this, deep neural networks encounter a phenomenon known as the vanishing gradient problem, which is extensively documented. This problem occurs when the gradient vector, which represents the direction and magnitude of the error signal used to update the model’s parameters, diminishes to zero. Consequently, this renders it impractical to continue training the model. As a result, the system becomes unable to continue its learning process. To tackle this matter, the concept of a residual connection was suggested by He et al. [6].

This concept can be mathematically represented as follows:

H (x) = F (x) + x

(1)

where the resulting feature map of the residual layer, denoted as H (x), is a function of the feature map F (x) obtained from the preceding layer. The input to these convolutional layers is represented by x. The effect of this skip connection is the reduction of gradient vanishing, enabling the network to increase its depth without compromising its ability to extract features.

A fully convolutional network architecture is presented in [10] for the segmentation of the Trypanosoma cruzi parasite. The architecture, based on the ResUnet, enhances the segmentation capability of the original model. Improvements are achieved by combining the Unet and ResNet architectures and modeling ResNets as a system that follows a differential equation in the form of:

\dot{x} (t) = f (t, x (t)), x (t_{0}) = t_{0}

(2)

Due to the previously mentioned issues with gradient vanishing, Res2Unet incorporates the Heun method (or improved Euler method) to solve ordinary differential equations. Implementing Heun’s method results in a quadratic reduction of errors and enables the network to employ fewer residual layers. Heun’s method is equivalent to an iteration in the following form:

x_{i + 1} = x_{i} + \frac{h}{2} (f (x_{i}) + f (x_{i} + hf (x_{i})))

(3)

The working principle of Heun’s method is to average the slopes of Euler’s method at t_i and t_i+1. Heun’s method can be mathematically written as

x_{1} = x_{i} + hf (x_{i})

(4)

x_{2} = x_{1} + hf (x_{1})

(5)

x_{i + 1} = \frac{1}{2} (x_{1} + x_{2})

(6)

where x_i is the input to a double residual unit (Res2Unet) and x_i+1 its output. Further, x₁ and x₂ are the outputs of the first and second residual units.

The FCN consists of three primary modules, namely the encoder, bridge, and decoder. The encoder module is responsible for extracting the inherent features of the input image. The bridge module acts as a connector between the encoder and decoder, facilitating the flow of information. Lastly, the decoder module is designed to generate a feature map that represents the segmented image, restoring it to its original size. This is necessary because the size of the feature map decreases by half after each convolution operation in the encoder module.

2.3 Our model

This work introduces a novel backbone for the YOLOv4 architecture, trained specifically for the identification and localization of pollen grains from various plant species found in the region of Yucatan (code may be available at https://github.com/clirlab/polen). The design of our backbone is based on the Res2Unet architecture, improved with the Mish activation function [8] and the utilization of weighted residual connections. The proposed model introduces a convolutional technique that incorporates residual components, as depicted in Fig. 3.

Fig. 3

Proposed residual unit scheme.

Each of the weights used in short and long skips are normalized as follows:

W_{i} = \frac{ReLU (w_{i})}{σ}

(7)

σ = \sum_{i = 1}^{N} ReLU (w_{i})

(8)

where N is the number of layers to be added, in our case N = 2. In addition, a Mish activation is added to each shortcut layer, thus, the shortcuts outputs are described by the following equation:

y = Mish (W_{i} x + W_{i + 1} x)

(9)

Weights W₁ and W₂ are different for the shortcut layers, this is, weights in x₁ are not the same as weights in x₂.

There is evidence that shallower residual networks exhibit superior performance in comparison to their deeper counterparts [17, 20]. Furthermore, the utilization of weights in the residual connections serves to mitigate the issue of gradient vanishing and facilitates the network’s convergence at an accelerated rate in comparison to a conventional network without residual connections [14].

Each residual unit consists of two layers that include merged convolutions with batch normalization and a Mish activation function, as depicted in Fig. 4. Figure 5 visually depicts the adaption of the backbone architecture proposed in the YOLOv4 network. Table 2 provides a more detailed breakdown of the number of filters and kernel size for each convolution in our proposed model. Only 3 × 3 kernels have padding. “sc” means shortcut. Filters column is in the format: Number of filters, Kernel size/Stride.

Fig. 4

Proposed residual unit structure.

Fig. 5

Proposed YOLOv4-based model.

Table 2

Proposed WRes2Net backbone structure

Layer	Filters	Input	Output
Conv1	32, 3/1	Image	416 × 416 × 32
Conv2	32, 3/2	Conv1	208 × 208 × 32
Conv3	32, 3/1	Conv2	208 × 208 × 32
Conv4	64, 3/1	Conv3	208 × 208 × 64
Conv5	64, 1/1	Conv2	208 × 208 × 64
sc1		Conv4-5	208 × 208 × 64
Conv6	64, 3/2	sc1	104 × 104 × 64
Conv7	128, 3/1	Conv6	104 × 104 × 128
Conv8	128, 1/2	sc1	104 × 104 × 128
sc2		Conv7-8	104 × 104 × 128
Conv9	128, 1/2	Conv2	104 × 104 × 128
sc3		Conv9-sc2	104 × 104 × 128
Conv10	128, 3/2	sc3	52 × 52 × 128
Conv11	256, 3/1	Conv10	52 × 52 × 256
Conv12	256, 1/2	sc3	52 × 52 × 256
sc4		Conv11-12	52 × 52 × 256
Conv13	256, 3/2	sc4	26 × 26 × 256
Conv14	512, 3/1	Conv13	26 × 26 × 512
Conv15	512, 3/1	sc4	26 × 26 × 512
sc5		Conv14-15	26 × 26 × 512
Conv16	512, 1/4	shortcut3	26 × 26 × 512
sc6		Conv16-sc5	26 × 26 × 512
Conv17	512, 3/2	sc6	13 × 13 × 512
Conv18	1024, 3/2	Conv17	13 × 13 × 1024
Conv19	1024, 1/2	sc6	13 × 13 × 1024
sc7		Conv18-19	13 × 13 × 1024

Four convolutional neural network architectures were evaluated in experiments aimed at automating the detection of pollen grains. The first YOLOv4 designs were employed, along with YOLOv7 for comparative analysis, while including our own backbone (WRes2Net) in the models. The modifications to the original architectures were implemented using code obtained from the open source repository Darknet¹ [11]. The studies were conducted using a computer equipped with an Nvidia RTX 2070 GPU to enhance the efficiency of the training process.

3 Results

Each of the four models had 10,000 iterations of training. The optimization algorithm employed in the present work involves the utilization of stochastic gradient descent with restarts. The initial learning rate was 0.0026, the momentum value was 0.949 and the decay rate was 0.0005. Figure 6 visualizes the training error behavior of YOLOv4, YOLOv7, YOLOv4+WRes2Net, and YOLOv7+WRes2Net networks. Our model (YOLOv4+WRes2Net) exhibits rapid convergence similar to original YOLOv4.

Fig. 6

Models learning curves: a) Original YOLOv4; b) Original YOLOv7; c) YOLOv4+WRes2Net; d) YOLOv7+WRes2Net.

The model we have developed possesses an estimated 10.2 million trainable parameters, whereas the original YOLOv4 model is equipped with roughly 27.5 million parameters, as indicated in Table 3. Despite this disparity, our model effectively achieves a comparable level of detection.

Table 3

Number of trainable parameters of evaluated models (backbone only)

Backbone	No. Trainable parameters
CSPDarknet-53 (YOLOv4)	27,624,520 (27.5 M)
WRes2Net (Ours)	10,211,726 (10.2 M)
CSPVoVNet (YOLOv7)	13,361,792 (13.3 M)

Figure 7 shows a sample of qualitative results of all plant species for every model. Performance of all trained models is presented in Table 4. The values of the weights (W1 and W2) learned in the shortcut layers were likewise obtained in the models incorporating our backbone (see Table 5).

Fig. 7

Detection results for all plant species (a)–(d) Ground truth, (e)–(h) YOLOv4; (i)–(l) YOLOv7; (m)–(p) YOLOv4+WRes2Net; (q)–(t) YOLOv7+WRes2Net.

Table 4

Performance results from the evaluated models

Model	Precision	Recall	mAP@0.5 (%)	IoU (%)	F1-Score
YOLOv4	0.92	0.99	98.68	81.72	0.96
YOLOv4 + WRes2Net (ours)	0.93	0.99	98.83	83.86	0.96
YOLOv7	0.90	0.85	93.81	75.88	0.87
YOLOv7 + WRes2Net	0.89	0.89	94.91	76.86	0.89

Table 5

Final weight values in the shortcut layers for the YOLOv4 and YOLOv7 models with our backbone

Layer number (shortcut)	W ₁		W ₂
	YOLOv4+WRes2Net	YOLOv7+WRes2Net	YOLOv4+WRes2Net	YOLOv7+WRes2Net
6	0.990074	0.918814	1.014566	1.142164
11	0.983940	0.914263	1.023335	1.212030
14	0.987035	0.870816	1.015991	1.149832
19	0.984751	0.831479	1.014417	1.166121
24	1.000256	1.015152	1.022918	1.208013
27	0.981539	0.799539	1.018605	1.199945
32	0.997042	0.903181	1.005113	1.106653

4 Conclusion and discussion

Pollen analysis is a very useful technique, whether to define the quality of honey, safely label its botanical origin or to determine where the bees have collected nectar. In this study, we present a deep-learning model developed to identify pollen grains in microscopic images. Our model is based on YOLOv4 and Res2UNet, which have been widely recognized for their effectiveness in object detection and segmentation tasks. Results of our model indicate that it achieved a mean average precision (mAP) of 98.83% at a threshold of 0.5 in our database. Having a smaller number of parameters compared to other architectures, our model achieves a performance that is similar to YOLOv4 with its CSPDarknet-53. This improvement in detection speed without losing precision has considerable significance. To the best of our knowledge, this is the first deep model for pollen analysis of plant species of Yucatan.

Footnotes

Notes

References

Battiato

Ortis

Trenta

Ascari

Politi

Pollen13k:

C.Siniscalco.

, A large scale microscope pollen grain image dataset. CoRR, abs/2007.04690, 2020.

Bochkovskiy

Wang

Liao

H.M.

, Yolov4: Optimal speed and accuracy of object detection, CoRR, abs/2004.10934, 2020.

Cázares

A.V. Castillo

Ordóñez

Y.B. Moguel

Cruz

M.A.Cortés

Huerta

E. Espinosa

Velasco

M.E. Arechavaleta

and Avilés

M.A. Mora

, Composición botánica de mieles de la peninsula de yucatán, mediante qPCR y análisis de curvas de disociación, Revista Mexicana de Ciencias Pecuarias, 2016.

Deng

Dong

Socher

L.-J.

Fei-FeiImagenet:

, A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition (2009), 248–255.

Glowacki

Gkantiragas

, Unsupervised representations of pollen in bright-field microscopy. CoRR, abs/1908.01866, 2019.

Zhang

Ren

Sun

, Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.

Martínez

Arguijo

López

A.H.V.

Armenta

R. Ángel Meléndez

, Clasificación de granos de polen: Efectode la distancia inter-pixel en glcm, Research in Computing Science149(8) (2020), 753–762.

Misra

, Mish: A self regularized non-monotonic neural activationfunction. CoRR, abs/1908.08681, 2019.

Murkute

, Robust pollen imagery classification with generative modeling and mixup training. CoRR, abs/2102.13143, 2021.

10.

Ojeda-Pat

A.E.

, Res2Unet: A fully convolutional neural network for Trypanosoma Cruzi parasite segmentation. Master’s thesis, Computational Learning and Imaging Reserach, Facultad de Matemáticas, Universidad Autónoma de Yucatán, 2020.

11.

Redmon

, Darknet: Open Source Neural Networks in C. http://pjreddie.com/darknet/, 2013.

12.

Redmon

Divvala

Girshick

Farhadi

, You only lookonce: Unified, real-time object detection. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016), 779–788.

13.

Russakovsky

Deng

Krause

Satheesh

Huang

Karpathy

Khosla

Bernstein

Berg

A. C.

and Fei-Fei

, ImageNet large scale visual recognition challenge, International Journal of Computer Vision (IJCV)115(3) (2015), 211–252.

14.

Shen

Zeng

, Weighted residuals for very deep networks. CoRR, abs/1605.08831, 2016.

15.

Simonyan

Zisserman

, Very deep convolutional networks forlarge-scale image recognition. CoRR, abs/1409.1556, 2014.

16.

Tan

, EfficientNet: Rethinking model scaling for convolutional neural networks. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 6105–6114. PMLR, 09–15 Jun 2019.

17.

Veit

Wilber

M.J.

Belongie

, Residual networks behave like ensembles of relatively shallow networks. In D. Lee, M. Sugiyama, U.Luxburg, I. Guyon and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016.

18.

Viertel

König

, Pattern recognition methodologies forpollen grain image classification: A survey, Mach. Vision Appl.33(1) (2022).

19.

Wang

C.-Y.

Bochkovskiy

Liao

H.-Y.M.

, Yolov7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors, 2022.

20.

Zagoruyko

Komodakis

, Wide residual networks. CoRR, abs/1605.07146, 2016.