Abstract
Powdery mildew, dead arm and vineyard downy mildew diseases are frequently seen in the vineyards in the Gediz River Basin, West Anatolia of Turkey. These diseases can be detected early using artificial intelligence (AI)–based systems that can contribute to crop yields and also reduce the labour of the farmer and the amount of pesticides used. This article presents a dataset – namely, Hermos – for use in such AI-based systems. Hermos contains four classes of grape leaf images: leaves with powdery mildew, leaves with dead arm, leaves with downy mildew and healthy leaves. We have currently 492 images and 13,913 labels in the dataset. We have published Hermos in the Linked Open Data (LOD) cloud in order to make it easier for consumers to access, process and manipulate the data.
1. Introduction
Diagnosing diseases at an early stage in the vineyards is of great importance. Early diagnosis ensures that the diseased areas are identified and healed quickly, thus preventing the disease from spreading over the vineyard. Early diagnosis of diseased areas and spraying only these areas instead of the entire vineyard help to eliminate the risk of the disease developing resistance to the drugs used and the problems that may occur in the domestic and foreign market due to pesticide residues. However, early diagnosis is a laborious and time-consuming task to be done by humans. Therefore, artificial intelligence (AI) methods are used to reduce the labour of the farmer and the amount of pesticides used. A recent survey [1] summarises the work on this topic.
This article presents a dataset of grape leaves which can be used in such kind of AI-based systems. The dataset can be used to detect powdery mildew, dead arm and downy mildew diseases. These three diseases were chosen because they are very common in the Gediz River Basin (especially in Manisa and its surrounding vineyards). Dataset currently contains the images of 320 leaves with powdery mildew, 240 leaves with dead arm, 100 leaves with downy mildew and 116 healthy leaves. The reason why the total number of images in the dataset is 492 instead of 776 is because the leaves in some images have more than one disease. The following paragraphs give the brief descriptions of these three diseases:
Powdery mildew (Uncinula necator): The disease occurs in all green parts of the vine. A light grey or white powdery spots appear on the grape leaf. In the first period, the disease is barely noticeable on young leaves. Generally, yellowish or shiny spots resembling oil stains appear on the upper surface of the leaves. As the leaf ages, it loses its shine, thickens and becomes brittle, curling inward from the edge.
Dead arm (Phomopsis viticola): The disease is seen in all above-ground parts of the vine except the main stem. Its effects on the vine wood cause first decline and eventually death. It is recognised by dark brown, black pinhead-sized spots scattered over the leaf surface. There is a light-coloured, yellow membrane around these spots. Larger areas appear as the spots coalesce over time.
Downy mildew (Plasmopara viticola): It is seen in regions with a warm and rainy climate during the vegetative development of the vine. In recent years, thanks to the effect of climatic conditions and the drugs applied, the disease does not harm the product as much as before. However, in humid and rainy regions, the disease sometimes causes economic loss. It begins in the form of oil spots on the leaves. A white fungal cover forms under these spots in humid environments. The colour of the spots darkens and turns brown over time, and after a while the stained area dries up.
The raw dataset consists of image files of the leaves and XML (Extensible Markup Language) files showing the labels on these images. In addition to the raw dataset, we have also published the dataset in five-star Linked Open Data (LOD) standards so that consumers can use the data more easily and machines can interpret the data. In the linked data version, besides the images and tags, the environmental conditions in which the image was taken are also stored. According to our literature search, Hermos is the first linked open dataset on the images of the diseased grape leaves.
The following section presents the background and the related work. Section 3 gives information about the dataset and the image annotation process. Section 4 describes the ways to access data via Mendeley Data Repository [2] or LOD cloud [3]. Section 5 presents a CNN (convolutional neural network) application for demonstrating the ease of use provided by the Hermos dataset. Finally, the last section concludes the article and describes future work.
2. Background and related work
Plant diseases are usually manifested by colour and texture changes in the leaves. Thus, it is a common approach to use image processing methods for classification of plant diseases. Table 1 summarises examples of these studies in literature. For example, Xu et al. [4] place an embedded system on a standard camera for the detection of leaf rust disease in wheat. Images taken by humans are classified by the embedded system using image processing techniques and the results are displayed on an Liquid-crystal display (LCD) screen. With this method, a success rate of 96.2% was achieved. Zhang et al. [5] classified seven different diseases of the cucumber plant using images in a dataset with 85.7% success. Sharif et al. [6] classified anthracnose, black spot, cancer, scabies, greening and melanosis fungal diseases in citrus fruits with 97% success. In Ma et al. [7], disease classification is made on cucumber leaves obtained from the Internet. Al-Otaibi et al. [8] made disease classification with artificial neural networks on 30 images of parsley and basil leaves that they took themselves and achieved 80% classification success. Manimegalai and Sivakamasundari [9] achieved 98.46% success with the SVM (support vector machine) technique in a dataset containing images of apple leaves. Chouhan et al. [10] achieved 86.2% success with artificial neural networks on the PlantVillage plant diseases dataset [11]. In Nababan et al. [12], disease classification was performed on the palm tree using the Naive Bayes algorithm with 80% success. In Fuentes et al. [13], a 96% disease classification success was achieved with deep learning on 5000 tomato leaf images collected with a digital camera.
Image processing studies for classification of plant diseases.
In addition to these studies, there are many studies that classify grape leaf diseases using image processing methods (Table 2). Pantazi et al. [14] demonstrate an automated way of crop disease identification on various leaf sample images corresponding to different crop species employing ‘Local Binary Patterns (LBPs)’ for feature extraction and ‘One Class Classification’ for classification. A total success rate of 95% was achieved, meaning that 44 of the 46 tested plant-condition combinations were successfully classified. For achieving this classification, eight vine leaf samples from each condition (healthy, powdery mildew, downy mildew, black rot) were used for the training procedure. In Biswas et al. [15], 900 images of disease infected grape leaves were acquired by the farmers and fieldworkers. This work identifies the severity of the diseases using image processing and machine learning algorithms. The proposed system achieves best classification accuracy of 86% using Random Forest and gray-level co-occurrence matrix (GLCM) features.
Image processing studies for classification of grape leaf diseases.
CNNs: convolutional neural networks.
In Knauer et al. [16], an advanced approach to hyperspectral image classification based on combined spatial and spectral image features, potentially applicable to many available hyperspectral sensor technologies, has been developed and validated to improve the detection of powdery mildew infection levels of Chardonnay grape bunches. The dataset consists of 60 hyperspectral images corresponding to two scans (top view and bottom view) of 30 bunches. From two of these bunches, 128 visually healthy and 136 severely diseased berries were selected and detached for recording of an additional dataset for classifier training and initial validation.
Xie et al. [17] first expand the grape leaf disease images through digital image processing technology, constructing the grape leaf disease dataset (GLDD). Based on GLDD and the Faster Region-based Convolutional Neural Networks (R-CNN) detection algorithm, a deep-learning-based Faster Double-RPN with Inception module and Attention Structure Convolutional Neural Network (DR-IACNN) model with higher feature extraction capability is presented for detecting grape leaf diseases by introducing the Inception-v1 module, Inception-ResNet-v2 module and SE-blocks. The experimental results show that the detection model Faster DR-IACNN achieves a precision of 81.1%. A total of 4449 original images of grape leaf diseases were obtained for this study.
None of the datasets used in the above studies are available to researchers, so they cannot be used as a standard in AI-based systems and cannot provide a benchmark to compare the performance of these systems. In addition, although there are many studies in the literature that detect grape leaf diseases using AI approaches on grape leaf images, it seems that there is not enough effort to create open access datasets from these images. However, it is of great importance for researchers to work on a standard dataset in order to accurately compare the performance of the methods they use. Table 3 summarises the datasets of grape leaf images in the literature. This table compares the datasets using five criteria.
Comparison of the grape leaf disease detection datasets.
The first criterion
Hermos has advantages over other datasets in Table 3 as it is published as open access, labelled and five-star LOD. Datasets that are not open access [17,18] were not evaluated because they could not be reused by other researchers. Although [21] is an open-access dataset, it is not labeled. In this case, the labeling process, which is a labor-intensive task, should be done by the researchers who will use the data set. Moreover, since the labelling is done in different ways by different people, performances of the different applications that use the dataset will not be realised in a standard way. Alessandrini et al. [19] is closest to the Hermos dataset in terms of being an open access and labelled dataset. However, it examines only one disease. In addition, since the dataset is not published in LOD standards, the user must perform additional operations on the labels in order to process the images. Publication of the Hermos dataset in LOD standards will provide an infrastructure for the development of richer tools and applications using the dataset. This infrastructure allows users to write more complex queries using the metadata of the images (basic attributes of the image, the location of its data source and where, when and under what weather conditions it was taken). Finally, although Hermos contains 492 images, the total number of partial images in the dataset increases to 13,913 after labelling diseased and healthy leaf regions using the bounding-box method. The bounding-box heuristic is used in image processing to reconstruct new images from the reprojected pixels of reference images [23].
3. Hermos dataset
3.1. Data description
Images in the dataset were obtained using two different digital cameras, Canon EOS 250D and Sony Alpha A5000. Images were randomly taken from a 30-cm working distance with/without zoom. They were acquired under real conditions with varying light, climatic conditions, backgrounds and angles from the vineyards in Manisa region. In addition, to the Sultana Grape leaf images, other grape leaf species were also photographed, thus diversifying the dataset with leaf images with different lobe numbers. Since early diagnosis is important, the images of the early stages of the diseases in the dataset are numerous. Table 4 presents the specifications of the Hermos dataset.
Data specifications table.
3.2. Image annotation
Image annotation is expressed by associating the tags with the display of the image. It is an important step for supervised machine learning. The success of the machine learning model depends on the accuracy of the tags we feed during the training phase. In this article, image annotation is used to indicate the category of disease(s) on the leaf.
We used ‘Bounding-boxes’ method to locate diseased spots on the leaves. ‘Bounding boxes’ is a common image annotation technique that determines the boundaries of the diseased point by specifying the coordinates of the upper left corner and the lower right corner. LabelImg tool [24] has been used to facilitate manual image annotation process in this work. LabelImg is a graphical image annotation tool to label objects using the bounding-boxes method. It is written in Python. It allows to export annotations as XML files in Pascal VOC (Visual Object Classes) format [25]. A Pascal VOC file is created for each of the annotated images in the dataset. In this work, we defined 13,913 labelled objects using LabelImg tool. The entire labelling process was carried out by domain experts. Figure 1 shows an image from the Hermos dataset.
1 <annotation>
2 <folder> Dead Arm </folder>
3 <filename>01.jpg </filename>
4 <path>./DeadArm/01.jpj</path>
5 <source>
6 <database>DeadArm</database>
7 </source>
8 <size>
9 <width>450</width>
10 <height>319</height>
11 <depth>3</depth>
12 </size>
13 <segmented>0</segmented>
14 <object>
15 <name>Dead Arm</name>
16 <pose>Frontal</pose>
17 <truncated>0</truncated>
18 <difficult>0</difficult>
19 <bndbox>
20 </xmin>90</xmin>
21 </ymin>190</ymin>
22 <xmax>54</xmax>
23 <ymax>70</ymax>
24 </bndbox>
25 </object>
26 </annotation>
A labelled image from the Hermos dataset.
Above is an example of Pascal VOC annotation file. First the filename, path and the size of the original image are defined. Then, it is specified whether the image is segmented. In a segmented image file, parts of the same object class are gathered together. After defining the properties of original image, the objects (labels) on the image are specified. In this example, there is only object labelled as ‘Dead Arm’ on the original image; ‘pose’ field specifies whether the labelled object is on the front or back of the leaf; ‘truncated’ field indicates whether the bounding box specified for the object corresponds to the full extent of the object; ‘difficult’ specifies whether the object is considered difficult to recognise; ‘xmin’, ‘ymin’, ‘xmax’ and ‘ymax’ specify the coordinates of the bounding box.
Figure 2 shows the distribution of the number of bounding boxes per image in the dataset. The average number of bounding boxes per image is 28,28. Computer vision models are built to learn what patterns of pixels correspond to an object of interest. Therefore, to train a model for identifying an object, every appearance of that object in the image dataset should be labelled. The unlabelled objects in some images would introduce false negatives to the model. The average number of bounding boxes per image is one of the metrics for estimating the quality labelling. A very low number indicates that some objects in some images were not labelled. An average of 28,28 bounding boxes per image is an indication of high precision labelling in this study.
Distribution of the number of bounding boxes per image in Hermos dataset.
4. Data accessibility
Data can be accessed in different ways, depending on who is going to consume them: (a) using the Mendeley Data Repository and (b) using LOD cloud. The following subsections describe each of these options.
4.1. Mendeley Data Repository
Mendeley Data is an open, free-to-use research data repository, which enables researchers to make their research data publicly available. By publishing Hermos dataset 1 on Mendeley Data platform, we aimed to increase the reuse, reproducibility, transparency and trust of the data. The dataset contains the image files and XML annotation files. Users can download, view and edit the dataset via a graphical image annotation tool supporting Pascal VOC format.
4.2. Hermos ontology on LOD cloud
We published Hermos dataset on LOD cloud to spread awareness so that people can use and link to the dataset. When publishing and consuming Linked Data, use of ontologies provides less effort in publishing, easier integration with existing data, reasoning support for implicit knowledge and machine readability. An ontology is a formal, explicit specification of a shared conceptualisation [26]. Therefore, we built an ontology schema to publish Hermos dataset (Figure 3) on the LOD cloud. This ontology stores information about the basic attributes of the image, the location of its data source and where, when and under what weather conditions it was taken. The ontology is represented in Web Ontology Language (OWL).
Hermos ontology schema.
After creating the ontology schema, we used the annotation files in the dataset to programmatically populate the ontology. Each image in the dataset is an individual of the ‘FullImage’ class in the ontology. Each annotated ‘bounding box’ is an individual of the ‘PartialImage’ class. The image and its parts are associated via the ‘hasPart’ and ‘isPartOf’ properties.
Finally, we published the Hermos ontology 2 in the LOD cloud under the ‘Life Sciences’ category. There are currently 115,320 axioms, 9 classes, 4 object properties, 62 data properties and 14,405 individuals in the ontology.
5. Case study: a CNN-based image processing application for classification of grape leaf diseases
To demonstrate the ease of use provided by the Hermos dataset, the dataset was used in an AI-based disease detection application. This application uses a CNN approach to classify images in the Hermos dataset. The CNN approach integrates a 4-neuron dense structure into the output layer of the InceptionV3 CNN architecture, where each neuron expresses a disease class. In CNN training, first of all, the diseased and healthy areas on the leaves should be cropped from the original image. Cropped images were obtained without any additional processing by using XML files containing the coordinates of diseased and healthy areas in each image. Thus, classification of leaf diseases was performed on the cropped images, not on the whole image. Each cropped image was resized to 150 × 150. Cropping the original images results in an increase in the number of images used for training and testing. As a result, 94.95% training, 86.94% validation and 83.7% test success were obtained. Figure 4 shows the steps of the disease classification process.
The steps of the disease classification process.
According to the results of the application, the machine-readable definition of diseased areas on an image in the Hermos dataset provides three important advantages:
Most of the existing datasets provide image data, but labelling is left to the user. Since the labelling is labour-intensive and specialistic process, the labelled dataset provides significant time and performance savings to the user.
The use of machine-interpretable labels in accordance with LOD standards allows for automatic generation of cropped images. The user does not need any additional process on the labels to create a cropped image.
Generally, a single leaf image is associated with a single disease in the labelled datasets. However, more than one disease can be found on many leaf images. Therefore, matching leaves containing more than one disease with a single disease can reduce the success of the machine learning application. The Hermos dataset obtains multiple cropped images from an original image, allowing the image to be matched with more than one disease, thus contributing to the performance in machine learning–based applications.
6. Conclusion and future work
This article presents an image dataset – namely, Hermos – to be used in AI-based viticulture systems. Powdery mildew, dead arm and downy mildew diseases were examined in the dataset. Currently, Hermos contains 492 image files and 13,913 tags.
Although there are many image processing applications in the field of agriculture, we have not found standard datasets for the development and comparison of these applications. In this study, a pre-labelled and open dataset was built for early detection of grapevine leaf diseases. Thus, researchers can bypass the labour-intensive process of data labelling. In addition, a common platform has been established where researchers can compare the performance of their applications with others. The dataset has also been published in LOD standards to enable users to easily process the data and create links to it. According to our research, Hermos is the first dataset on grapevine leaf diseases published in the LOD cloud.
Maintenance and support of the dataset is an ongoing work. As a potential future work, we plan to extend the dataset with various diseases of grape leaves.
Footnotes
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study is supported by the Scientific and Technological Research Council of Turkey (TUBITAK) under Project Number 120E098.
