A fusion based land cover classification model using remote sensed images

Abstract

Classification of land cover from remote sensed image is quite challenging task. Since the satellite images preserve spatial and spectral information, thus it is essential to identify the land cover classes and classify them to generate the thematic map. The remote sensed images and thus produced thematic maps are useful for extracting the esteemed information in diagnosing, supervising, and management of earth’s surface. In this paper, a multiclass land cover classification model is proposed that comprise of pre-processing method, a multiclass classifier and performance evaluation strategy. The land cover-based satellite images are applied to this model to generate a land cover map labelled with seven land cover classes. The morphological opening, closing, and a fusion technique are involved in pre-processing stage to extract the spatial information as well as reduce the incurred noise from the input image. Then a supervised classification methodology is introduced to classify the image into 7 number of land cover classes based on the spectral values of each pixel of the image. The overall achievement of the proposed model is compared with some existing multiclass supervised and unsupervised classification techniques such as Naïve Bayes classifier (NBC), Decision tree (DT), K-nearest neighbour (KNN), Convolution Neural Network (CNN).

Keywords

Land cover image land cover classification morphological opening morphological closing image fusion

1. Introduction

Spatial Database is a huge database that collects data from different sources such as satellite images, geological surveys, maps, etc. The size of this database grows rapidly by adding new data day-by-day. The spatial database system requires complex calculations used in spatial queries for retrieving those data, which requires more processing power as well as more memory space. Hence, the classification and prediction techniques are used to extract data models which describe the data classes to predict future trends. Such data analysis is useful with a better understanding of larger data. Land cover [1, 2, 3] specifies the characteristics of the earth surface, which is categorized with water surface, forest, soil surface, human-building structure, etc. The task of multiclass problem is to assign some instances to the particular mutual exclusive classes. Land cover classification is a type of multiclass problem [4, 5] where classifier is used to recognize the different classes to obtain a land cover map for future analysis. Remote sensing land cover image classification is a complicated process that associates many steps, such as determination of a classification system, an assortment of remotely sensed data, collection of a supervised or unsupervised multiclass classification algorithm, generation of the thematic map, and accuracy assessment.

The remotely sensed tool [6, 7, 8, 9] performs an important role in classification of land cover surfaces [10] of any satellite images. There are many classification techniques used in various fields to solve various data analysis, such as Naïve Bayes classification [11, 12, 13], Decision tree [14], Support vector machine [15, 16, 17, 18], K-nearest neighbour [19, 20], Deep Neural network [21, 22], Logistic regression, etc.

Such satellite images are emerged with mixed noise and having loss of data. To classify those data efficiently, some techniques can be used to restore and denoise these image data. The morphological image processing [23] is a nonlinear process of extracting shape characteristic or morphological features of an image with excellent speed and denoising performance [11]. This depends on the shape of a structuring element, which is used as a probe for the image. If the pixel value of the structuring element hits or fits with the pixel value of the image, then the max or min of pixel value will be taken in the resultant image. It mainly adds pixel at the edge or boundary region through the dilation process of morphological operation and subtracts pixels from the boundary by using the erosion process of morphological operation [24].

A substitute approach to intensify the denoising quality of an image is by executing a fusion technique [25], which can be studied from Fig. 1. This technique generally incorporates the most relevant information of different sources to constitute a single image [26, 27], which has more information than any general input image.

Figure 1.

Aim of image fusion.

Fusion is carried out for better image segmentation [28, 29], feature extraction [30], etc. for further processing. Some of the image fusions techniques are served as high pass filtering techniques, PCA based image fusion [26], wavelet transform image fusion etc.

In this paper, morphologically opened and closed images of the same image are fused to gather contextual information of each pixel by employing a pixel level fusion technique. Then, the final fused image will be served as input to the multiclass classifier. The outcome of classification is compared with some existing classification techniques such as Naïve Bayes Classifier (NBC), Decision Tree (DT), K-Nearest Neighbour (KNN) and Convolution Neural Network (CNN).

The organization of this paper with its respective section is described as follows. The introduction is followed by Section 2, where there is a brief description of the literature survey on supervised and unsupervised classification techniques. In Section 3, the proposed FMC is described elaborately with its workflow diagram, and pseudocode. Section 4 is all about the result analysis with five classification parameters. Finally, the last section is ended with a conclusion and references.

2. Previous works on land cover image processing techniques

Table 1
List of some previously used supervised and unsupervised classification techniques on land cover image

SL No.	References	Classification techniques used	Comparison with other technique and accuracy	Application area
1	Zhang et al. [4]	Adaptive random forest classifier	Single random forest classifier (kappa. 9443)	Global automated MODIS lands classification
2	Matikainen et al. [5]	Object based land cover classification	–	Multispectral ALS data
3	Huang et al. [14], Masad et al. [29], Katuwal et al. [5], Huang et al. [30], Simard et al. [31], Bakos et al. [32], Xia et al. [33]	• Some of pre-processing technique are such as divide and conquer rule which performs binary tests on leaf nodes to classify low-resolution patch, Morphological segmentation, random vector functional link (RVFL) network • Decision tree, a new ensemble classifier based on decision tree and random vector functional link (RVFL) network for	–	Some of the application area is • MODIS for land cover classification • High-resolution Image • T2-Mr images • The vegetation cover of global rain forest mapping (GRFM) JERS-1 SAR mosaics’ • Classification of vegetation in plain and mountain areas. • Multi-class classification • Wart treatment
4	Gumus et al. [11]	Random, SVM, ANN, Naive Bayes	–	Used to select the spectral features from ASTER data to identify four species like Sugi, mixed deciduous, hinoki, others
5	Sylva et al. [34]	Multi-layered perceptron	–	Presented future scenario of Taperoa‘ River basin, Brazil for 2035
6	Bagan et al. [2]	SOM (self-organizing map neural network) and MLC (maximum likelihood classification).	–	Agriculturally based remote sensed data
7	Gašparović et al. [18]	A novel unsupervised land cover automated classification method	Maximum likelihood classifier, random forest	Landsat-8 images of 30 m spatial resolution bands
8	Mahmon et al. [1]	Maximum likelihood classifier, Mahalanobis distance classifier, Minimum distance classifier.	Highest accuracy of Maximum likelihood classifier of 82.5%.	Land cover Landsat 8 satellite images

The main objective of using satellite images is to supervise the changes in the earth’s land cover. Land cover is the presentation of the peripheral surface of the earth, which is used in many fields. Many supervised [29, 31, 32, 33, 34, 35] and unsupervised classifiers [10] are trained to classify a land cover image of different remote sensing imagery [4, 5, 35, 36] into different class levels and performance is analysed by different parameter metrics [1, 37].

Figure 2.

Workflow diagram of proposed fusion based multiclass classifier (FMC).

The supervised and unsupervised classification techniques can be combinedly used for better performance. Kumar et al. [37] adopted a hybrid-based classification technique in which maximum likelihood classifier, ISODATA clustering methods, and decision tree approaches are merged for Land Used Land Cover (LULC) multi-temporal dataset classification. Liu et al. [1] used Spatial-Temporal Land Cover Filter (STLCF) to remove the illogical land cover change events and naïve Bayesian equation and decision tree combinedly used for classification. Shaker et al. [36] used two automatic training data selection methods such as Gaussian Mixture Model (GMM) to split preliminarily the land and water region based on the elevation/intensity histogram, and the second method is developed based on the use of Scan Line Intensity-Elevation Ratio (SLIER) demonstrated on multispectral airborne LIDAR data. Li et al. [17] used Bayesian network classifier, decision tree, and convolutional neural network combinedly for UC Merced Land set datasets. Some researchers follow image pre-processing techniques to reduce the noise of raw satellite images. Nair et al. [11] proposed a model named enhanced streaming random tree, in which morphological processing was performed to remove the shadow from the image. Zhang et al. [23] proposed a power disturbance identification scheme based on generalized morphological open-closing and close-opening undecimated wavelet. To extract feature-level information from the land cover image, the fusion technique is adopted by many researchers. Li et al. proposed a novel non-rigid inter-subject multichannel image registration method [25] along with Gabor wavelets transformation and independent component analysis for multichannel image registration. Xia et al. [24] used a cooperative neural fusion algorithm for image restoration and adopted a cooperative neural fusion for the enhancement of the image quality. Chang et al. [26] adopted multisource fusion followed by the fisher criterion-based nearest feature space approach for Lands slide classification.

From the above surveys, we are taking into account the morphological operations and a fusion method to exploit contextual information as well as to reduce the noise before permit the input to the proposed classification method. The performance of classification is evaluated and the classification result is compared with some supervised and unsupervised classification models such as DT, NBC, KNN (with neighbours 3, 5, and 7), and CNN.

3. The work flow diagram of proposed fusion based multiclass classifier (FMC)

The workflow diagram of the proposed fusion based multiclass classifier is demonstrated in Fig. 2. It comprises of three basic stages for land cover image processing. The 1 ${}^{\text{st}}$ stage is a pre-processing stage which is used to reduce noise and data loss of considered image. The 2 ${}^{\text{nd}}$ stage is implemented for image classification using the proposed FMC algorithm. And the final stage is meant for performance analysis of the proposed approach in comparison to a few existing techniques of recent literature. The first stage takes input as any land cover satellite image, on which both morphological operations and fusion operation are carried out to generate the fused image which will be input for the next stage of proposed work. In the second stage, the proposed FMC algorithm is applied to obtain a classified image. In this stage, the intensity values of R, G, B component of each pixel of fused image is extracted and each pixel gets its class level with reference to the training image dataset. As a result, a classified image is obtained. That classified image is re-coloured to generate land cover map for future reference. The outcome of FMC classification results is compared with few other classifiers such as Naïve Bayes classifier, decision tree, KNN by considering 3 neighbours (KNN3), KNN by considering 5 neighbours (KNN5), KNN by considering 7 neighbours (KNN7), and CNN. Five different performance metrics are considered for overall performance analysis of FMC such as precision, recall, f1-score, overall accuracy and kappa coefficient metrics.

3.1 Dataset description and image pre-processing

Google Earth is an approved geolocation program that can be used to explore the world from the android mobile or computer. The spatial resolution of satellite images of Google Earth depends on the source of data. We were searching for images with multiple objects within so that to validate our proposed classification algorithm. So, the study area is not confined. Three image datasets have been chosen from Google earth on the Android 9.0 mobile. The first two images were taken from the location- Data SIO, NOAA, U.S. Navy, NGA, GEBCO, Landsat/Copernicus 22 ${}^{\circ}$ 22 ${}^{\prime}$ 28 ${}^{\prime\prime}$ S 149 ${}^{\circ}$ 35 ${}^{\prime}$ 55 ${}^{\prime\prime}$ E371 km and 20 ${}^{\circ}$ 30 ${}^{\prime}$ 13 ${}^{\prime\prime}$ N 86 ${}^{\circ}$ 04 ${}^{\prime}$ 57 ${}^{\prime\prime}$ E538 km respectively. Both two image datasets in Figs 3 and 4 are having 8 number of bands, a spatial resolution of 30 m, temporal resolution is 16 days. The third image in Fig. 5 is a high-resolution image that is collected from Digital Globe satellites of Maxar technologies, whose location is 19 ${}^{\circ}$ 56 ${}^{\prime}$ 17 ${}^{\prime\prime}$ N 83 ${}^{\circ}$ 06 ${}^{\prime}$ 55 ${}^{\prime\prime}$ E652 m. It has a spatial resolution of 30 cm; the temporal resolution is 1–3 days and 8–11 number of bands. The acquisition date of above mentioned three image datasets is 27.09.2019. All these images are resized into (280 $\times$ 282) dimensions for the pre-processing stage. The pre-processing stage consists of closing and opening of a morphological operation that is mathematically explained as below.

Figure 3.

Image dataset1 (Source: Data SIO, NOAA, U.S. Navy, NGA, GEBCO, Landsat /Copernicus 22 ${}^{\circ}$ 22 ${}^{\prime}$ 28 ${}^{\prime\prime}$ S 149 ${}^{\circ}$ 35 ${}^{\prime}$ 55 ${}^{\prime\prime}$ E 371 km).

Figure 4.

Image dataset2 (Source: Data SIO, NOAA, U.S. Navy, NGA, GEBCO, Landsat/Copernicus 20 ${}^{\circ}$ 30 ${}^{\prime}$ 13 ${}^{\prime\prime}$ N 86 ${}^{\circ}$ 04 ${}^{\prime}$ 57 ${}^{\prime\prime}$ E 538 km).

Figure 5.

Image dataset3 (Source: Maxar technologies 19 ${}^{\circ}$ 56 ${}^{\prime}$ 17 ${}^{\prime\prime}$ N 83 ${}^{\circ}$ 06 ${}^{\prime}$ 55 ${}^{\prime\prime}$ E652 km).

Let $A$ and $B$ are the binary images in $Z^{2}$ , dilation of $A$ by $B$ is denoted as follows

$\displaystyle A\oplus B=\{Z|(\hat{B})z\cap A\neq\varnothing\}$ (1)

Similarly, erosion of $A$ by $B$ is denoted as follows

$\displaystyle A\circleddash B=\{Z|(B)z\subseteq A\}$ (2)

Let $A$ be the image to be processed and $B$ be the structuring element. The opening operation is the erosion followed by dilation that can be expressed below as

$\displaystyle A\circ B=(A\circleddash B)\oplus B$ (3)

Similarly, the closing operation is the dilation followed by erosion, which can be expressed a

$\displaystyle A\cdot B=(A\oplus B)\circleddash B$ (4)

where $\circleddash$ indicates erosion operator and $\oplus$ indicates dilation operator

First, a morphological open operation is applied on the image ( $I$ ) with a disc-shaped structuring element of size 2 ( $\textit{SD}2$ ) to get an opened image (IMO). Then the morphological closing operation is applied on ( $I$ ) with the same structuring element ( $\textit{SD}2$ ) to get a closed image (IMC). According to Eqs (1) and (2), these operations can be written as

$\displaystyle\textit{IMO}=(I\circleddash\textit{SD}2)\oplus\textit{SD}2$ (5) $\displaystyle\textit{IMC}=(I\oplus\textit{SD}2)\circleddash\textit{SD}2$ (6)

where $\circleddash=$ erosion operator, $\oplus=$ dilation operator, $I=$ land cover image, $(\textit{SD}2)=$ a structuring element disc with size 2, $\textit{IMO}=$ morphological opened image and $\textit{IMC}=$ morphological closed image.

Thereafter, both images IMO and IMC are preceded for pixel-level fusion technique. This technique is used to fuse or combine the images into one image IF with an alpha factor $=$ 0.5 so that two images are mixed in equal proportion. For the quality measurement of fused image, two metrics are used such as Peak Signal-to-Noise Ratio (PSNR) and Signal-to-Noise Ratio (SNR) in between fused image and original image. The PSNR is computed from Mean Square Error (MSE), that can be expressed as follows.

$\displaystyle\text{MSE}=\frac{1}{mn}\sum_{i=0}^{m-1}\sum_{j=0}^{n-1}(I_{R}(i,j% )-I_{f}(i,j))^{2}$ (7) $\displaystyle\text{PSNR}=10\log_{10}\max{}^{2}/\text{MSE}$ (8) $\displaystyle\text{SNR}=10\log_{10}\Bigg{[}\sum_{i=0}^{m-1}\sum_{j=0}^{n-1}(I_% {R}(i,j))^{2}\bigg{/}$ $\displaystyle\quad∼{}\sum_{i=0}^{m-1}\sum_{j=0}^{n-1}(I_{R}(i,j)-I_{f}(i,j))^{% 2}\Bigg{]}$ (9)

where $I_{R}(i,j)=$ Original image, $I_{f}(i,j)=$ Fused image, $(m,n)=$ Number of rows and columns of pixels respectively, and MAX $=$ Number of maximum possible intensity levels (minimum intensity level supposed to be 0) in an image.

The applied pixel-level fusion technique is compared with Discrete Wavelet Transform (DWT) fusion technique and Principal Component Analysis (PCA) based fusion technique [38, 39] by these two metrics.

3.2 Proposed fusion based multiclass classifier (FMC) algorithm

The proposed multiclass classifier is based on the concept of supervised classification technique. In this model, first, the model is trained with a training image dataset, after that the model is ready to predict the land cover classes for the test image dataset. Image data are given to the computer in the form of digital values. The smallest unit of the image is known as a pixel. In the RGB colour model, the intensity level of each pixel is the combination of three colour components i.e., red, green and blue. Each component is associated with the intensity values within the range of 0 to 255. For the training dataset, image dataset is chosen from Google Earth and segmented in different classes with predefined colours as shown in Table 2 by annotation tool. In this approach seven number of class levels are considered for land cover classification such as dense forest (Class 1), less dense forest (Class 2), grass (Class 3), pure water (Class 4), impure water (Class 5), bare soil (Class 6), mountain (Class 7). By considering threshold ranges of intensity values of R, G, B, each class level is defined. If the spectral features of one pixel of the tested image dataset are fallen into the spectral features of the training image dataset, then that pixel is classified into a particular class level. In the same way, all pixels are classified with defined land cover classes. After that, each pixel is re-coloured with different ranges of colour according to their associated class levels to generate land cover maps for an elegant view. The colours for pixel classification are mentioned in the 3 ${}^{\text{rd}}$ column of Table 2.

Table 2
List of land cover types with their re-colouring shades

Class labels	Land cover types	Colour for classification
Class 1	Dense forest	Yellow
Class 2	Less dense forest	Black
Class 3	Grass	Pink
Class 4	Pure water	Cyan
Class 5	Impure water	Dark blue
Class 6	Bare soil	Brown
Class 7	Mountain	Green

3.3 Pseudocode for proposed FMC approach

In this section the pseudocode for proposed FMC approach for any land cover image is briefly described.

/* Steps to approach proposed FMC */

Input: Any land cover (LC) image $(I)$

Output: Re-coloured classified image of $(I)$

Step 9: Step 1:
Read the LC image $(I)$
Step 2:
Apply morphological opening and morphological closing operation on $(I)$ to get IMO and IMC using Eqs (5) and (6) respectively.
Step 3:
Apply pixel-level image fusion technique with alpha-factor 0.5 on both IMO and IMC of step 2 to get a fused image IF.
Step 4:
Read the fused image IF from step 3 to determine the size of the image in the form of $m$ and $n$ . /* $m=$ number of rows, $n=$ number of columns */
Step 5:
Initialize a variable named $X$ and set $X=m\times n$ .
Step 6:
Initialize the array with size $(X\times 3)$ to store red, green, and blue values of each pixel.
Step 7:
Read the intensity values of R, G, and B component of each pixel and store those values in the array.
Step 8:
Assign class level according to the ranges of R, G, and B values and add class level to each row of the array. Such that one column is added as class levels to that array of 4 ${}^{\text{th}}$ step.
Step 9:
Each pixel of the original image is re-coloured according to its class level.

Figure 6.
Result of morphological opening (a), closing (b), and corresponding fused image (c) of image dataset1.

Figure 7.
Result of morphological opening (a), closing (b), and corresponding fused image (c) of image dataset2.

3.4 Performance analysis

In this research work, land cover image processing is carried out using fusion based multiclass classifier. Five number of performance metrics such as precision, recall, f1-score, overall accuracy, Cohen’s kappa coefficient is adopted to measure the performance of proposed FMC with existing classifiers such as NBC, DT, KNN3, KNN5, KNN7, and CNN. All these mentioned metrics are evaluated from the confusion matrix or error matrix. It is a matrix of predicted and actual performance result of a classifier. Based on the confusion matrix ( $C_{n\times n}$ ), true positive ( $t p$ ), true negative ( $t_{n}$ ), false positive ( $f_{p}$ ), false negative ( $f_{n}$ ) are calculated from the below equations.

$\displaystyle t_{pi}=c_{ii}$ (10) $\displaystyle f_{pi}=\sum^{n}_{l=1}c_{ii}-t_{pi}$ (11) $\displaystyle f_{ni}=\sum^{n}_{l=1}c_{il}-t_{pi}$ (12) $\displaystyle t_{ni}=\sum^{n}_{l=1}\sum^{n}_{k=1}c_{lk}-t_{pi}-f_{pi}-f_{ni}$ (13)

Considering the values of $t_{p},f_{p},f_{n}$ , and $t_{n}$ , two performance metrics such as precision $(P)$ and recall $(R)$ are evaluated as in Eqs (14) and (15). To measure the balance between these two metrics, f1-score is calculated as in Eq. (16).

$\displaystyle P=\frac{t_{p}}{t_{p}+f_{p}}$ (14) $\displaystyle R=\frac{t_{p}}{t_{p}+f_{n}}$ (15) $\displaystyle\text{f1-score}=\frac{2\times R\times P}{R+P}$ (16)

4. Experimental set up and result analysis

4.1 Image pre-processing

As discussed earlier, three image datasets are randomly collected from Google Earth, whose actual locations are mentioned in Figs 3–5. These images are cropped to the size of (280 $\times$ 282) for the image pre-processing stage. As discussed earlier, morphological operations such as opening and closing are applied to get opened image and closed image respectively. Then both the opened and closed images are fused using image fusion technique. So, there are 3 sets of fused images for the three input image datasets, which are shown in Figs 6–8.

The PSNR and SNR are typically adopted for quality evaluation of the fused image. Table 3 highlights the comparison among the pixel-level fusion technique with two other fusion techniques such as DWT fusion technique and PCA based image fusion technique.

Table 3
Comparison between three fusion techniques by quality measurement metrics PSNR and SNR

Fusion techniques $\downarrow$	Fused Image dataset1	Fused Image dataset2	Fused Image dataset3
Pixel-level fusion with alpha factor 0.5	PSNR 29.39	PSNR 32.43	PSNR 34.15
	SNR 22.77	SNR 24.39	SNR 25.17
DWT	PSNR 22.74	PSNR 27.79	PSNR 22.88
	SNR 16.12	SNR 19.74	SNR 13.89
PCA	PSNR 22.79	PSNR 27.95	PSNR 22.87
	SNR 16.17	SNR 19.91	SNR 13.89

Figure 8.

Result of morphological opening (a), closing (b), and corresponding fused image dataset3.

Figure 9.

Re-coloured classified image dataset1 (a), image dataset2 (b), image dataset3 (c).

Table 4

Precision, recall, f1-score of each class level of proposed FMC and considered classifiers on image dataset1

$\downarrow$ Classifiers	Class level $\rightarrow$	Class 1	Class 2	Class 3	Class 4	Class 5	Class 6	Class 7
FMC	Precision	0.8800	0.6300	–	0.8300	0.1500	0.6600	–
	Recall	0.8800	0.5700	–	0.7900	0.2500	0.8900	–
	F1 score	0.8800	0.6000	–	0.8100	0.1900	0.7600	–
NBC	Precision	0.3900	0.2600	0.0055	0.0000	0.4400	–	–
	Recall	0.7600	0.0045	0.0120	0.0000	0.1400	–	–
	F1 score	0.5100	0.0088	0.0076	0.0000	0.2100	–	–
DT	Precision	0.3900	0.3700	–	0.0750	–	–	0.0710
	Recall	0.4300	0.2300	–	0.1200	–	–	0.0450
	F1 score	0.4100	0.2800	–	0.0920	–	–	0.0550
KNN3	Precision	0.9400	0.9500	0.0000	0.4000	0.0000	0.5900	–
	Recall	0.9100	0.4500	0.0000	0.7800	0.0000	0.7200	–
	F1 score	0.9200	0.6100	0.0000	0.5300	0.0000	0.6500	–
KNN5	Precision	0.9300	0.9100	–	0.4100	–	0.5900	–
	Recall	0.9000	0.4400	–	0.7800	–	0.7300	–
	F1 score	0.9200	0.5900	–	0.5400	–	0.6600	–
KNN7	Precision	0.9300	0.8900	–	0.4200	–	0.5900	–
	Recall	0.8900	0.4400	–	0.7700	–	0.7100	–
	F1 score	0.9100	0.5900	–	0.5400	–	0.6500	–
CNN	Precision	0.3614	0.1788	0.0000	0.0000	0.0000	0.0000	0.0000
	Recall	0.7316	0.2189	0.0000	0.0000	0.0000	0.0000	0.0000
	F1 score	0.4838	0.1968	0.0000	0.0000	0.0000	0.0000	0.0000

Table 5

Precision, recall, f1-score of each class level of proposed FMC and considered classifiers on image dataset2

$\downarrow$ Classifiers	Class level $\rightarrow$	Class 1	Class 2	Class 3	Class 4	Class 5	Class 6	Class 7
FMC	Precision	0.9400	0.8700	0.3100	0.9200	0.1400	0.0000	0.6400
	Recall	0.9400	0.8100	1.0000	0.9200	0.3100	0.0000	0.8600
	F1 score	0.9400	0.8400	0.4800	0.9200	0.1900	0.0000	0.7300
NBC	Precision	0.9600	0.5900	0.0000	0.6000	0.0000	0.0000	0.7700
	Recall	0.8900	0.9700	0.0000	0.8900	0.0000	0.0000	0.2100
	F1 score	0.9200	0.7300	0.0000	0.7200	0.0000	0.0000	0.3300
DT	Precision	1.0000	1.0000	–	0.8300	–	1.0000	0.4300
	Recall	0.9300	0.8100	–	0.9900	–	1.0000	0.8900
	F1 score	0.9600	0.8900	–	0.9000	–	1.0000	0.5800
KNN3	Precision	0.9800	0.8800	–	0.8200	–	–	0.3000
	Recall	0.8600	0.8000	–	0.9800	–	–	0.6500
	F1 score	0.9200	0.8400	–	0.8900	–	–	0.4100
KNN5	Precision	0.9800	0.8400	–	0.8200	–	–	0.3000
	Recall	0.8500	0.7900	–	0.9700	–	–	0.7100
	F1 score	0.9100	0.8200	–	0.8900	–	–	0.4200
KNN7	Precision	0.9800	0.7800	–	0.8200	–	–	0.3000
	Recall	0.8300	0.7800	–	0.9700	–	–	0.6300
	F1 score	0.9000	0.7800	–	0.8900	–	–	0.4100
CNN	Precision	0.5983	0.7001	0.0000	0.0000	0.0000	0.0000	0.0000
	Recall	0.9047	0.7985	0.0000	0.0000	0.0000	0.0000	0.0000
	F1 score	0.7203	0.7460	0.0000	0.0000	0.0000	0.0000	0.0000

4.2 Implementation of proposed FMC

The output of the discussed image pre-processing stage is presented in Fig. 6c, Fig. 7c, and Fig. 8c. These images represent the fused images of image dataset1, image dataset2, and image dataset3. These fused images are served as the inputs for the next stage. For implementing of FMC algorithm, first, the R, G, and B value of each pixel of fused image is extracted and compared these intensity values with R, G, B value of training dataset to obtain the class level of each pixel. According to the class level, each pixel is re-coloured with reference to Table 2. The resultant three re-coloured images of three fused images Fig. 6c, Fig. 7c and Fig. 8c are shown in Fig. 9a–c.

4.3 Result analysis

In this experiment, the precision, recall, and f1-score [29] of each class level are calculated. The validity of prediction by proposed FMC is analysed comparing with NBC, DT, KNN3, KNN5, KNN7, and CNN for each class based on these metrics. The architecture of CNN constitutes 1 convolution layer, 1 max pooling layer, and 3 FC layers for our experiment. Tables 3–5 show the classification result for the input image dataset1, image dataset2, and image dataset3 respectively.

From Table 4, the FMC classify Class 2, Class 4, Class 5, and Class 6 with high precision and recall value. On the other hand, NBC, DT, and KNN3 show high precision and recall value for only one class label such as Class 3, Class 7, and Class 1 respectively.

The f1-score of Class 2, Class 4, Class 6 is the highest in the case of FMC. So, from this Table 4, it is observed that FMC can be considered to be a better performing classifier in comparison to rest classifier for image dataset1. However, evaluating with respect to only one image dataset, analysis can’t be concluded, hence two more images are analysed in the next two tables.

Table 6
Precision, recall, f1-score of each class level of proposed FMC and considered classifiers on image dataset3

$\downarrow$ Classifiers	Class level $\rightarrow$	Class 1	Class 2	Class 3	Class 4	Class 5	Class 6	Class 7
FMC	Precision	0.9800	0.2900	0.7600	0.8600	0.8400	–	–
	Recall	0.9600	0.2000	0.8100	0.8600	0.9100	–	–
	F1 score	0.9700	0.2400	0.7800	0.8600	0.8700	–	–
NBC	Precision	0.7200	0.0001	–	–	–	–	0.2800
	Recall	0.4500	0.0290	–	–	–	–	0.0540
	F1 score	0.5600	0.0001	–	–	–	–	0.0900
DT	Precision	0.9900	1.0000	0.2800	–	0.9300	–	–
	Recall	0.9900	0.0420	0.7600	–	0.8600	–	–
	F1 score	0.9900	0.0810	0.4100	–	0.8900	–	–
KNN3	Precision	0.9600	0.9800	–	–	0.3800	–	–
	Recall	0.8800	0.0240	–	–	0.6800	–	–
	F1 score	0.9200	0.0470	–	–	0.4900	–	–
KNN5	Precision	0.9600	0.9600	–	–	0.3800	–	–
	Recall	0.8800	0.0240	–	–	0.6900	–	–
	F1 score	0.9200	0.0470	–	–	0.4900	–	–
KNN7	Precision	0.9600	0.9400	–	–	0.3800	–	–
	Recall	0.8900	0.0230	–	–	0.6600	–	–
	F1 score	0.9200	0.0450	–	–	0.4800	–	–
CNN	Precision	0.9604	0.6715	0.0000	0.0000	0.0000	0.0000	0.0000
	Recall	0.7180	0.9986	0.0000	0.0000	0.0000	0.0000	0.0000
	F1 score	0.8217	0.8030	0.0000	0.0000	0.0000	0.0000	0.0000

Table 7

Average precision, average recall, average f1-score, overall accuracy, kappa coefficient of image dataset1 with respect to considered classifiers

Metrics $\rightarrow$	Precision	Recall	F1 score	Overall accuracy in %	Kappa coefficient value
$\downarrow$ Classifiers
FMC	0.394	0.393	0.405	79.470	0.708
NBC	0.137	0.102	0.092	35.604	$-$ 0.003
DT	0.113	0.115	0.103	27.283	$-$ 0.039
KNN3	0.360	0.357	0.388	71.563	0.611
KNN5	0.355	0.356	0.366	71.215	0.605
KNN7	0.049	0.036	0.036	70.766	0.599
CNN	0.025	0.135	0.042	41.114	$-$ 0.008

From Table 5, the proposed FMC shows high precision, recall, f1-score for Class 3, Class 4, Class 5, and Class 7. However, DT also shows higher side performance for Class 1, Class 2, and Class 6. Therefore, DT shows superiority over FMC on image dataset2. On the other side FMC identifies maximum number of classes [i.e., Class 3, Class 4, Class 5, and Class 7] as compared to DT [i.e., Class 1 and Class 2]. Hence, for image dataset2, the performance of FMC and DT is comparable. Therefore, another image dataset is taken for deeper analysis purposes, which is listed in Table 6.

Table 8

Average precision, average recall, average f1-score, overall accuracy, kappa coefficient of image dataset2 with respect to considered classifiers

Metrics $\rightarrow$	Precision	Recall	F1 score	Overall accuracy in %	Kappa coefficient value
$\downarrow$ Classifiers
FMC	0.478	0.605	0.512	91.168	0.849
NBC	0.365	0.370	0.338	76.182	0.659
DT	0.533	0.578	0.541	89.820	0.868
KNN3	0.372	0.411	0.382	87.090	0.805
KNN5	0.368	0.415	0.380	86.383	0.793
KNN7	0.360	0.401	0.373	84.961	0.771
CNN	0.185	0.243	0.210	59.184	0.500

Table 9

Average precision, average recall, average f1-score, overall accuracy, kappa coefficient of image dataset3 with respect to considered classifiers

Metrics $\rightarrow$	Precision	Recall	F1 score	Overall accuracy in %	Kappa coefficient value
$\downarrow$ Classifiers
FMC	0.466	0.467	0.465	93.930	0.844
NBC	0.125	0.067	0.296	33.142	0.014
DT	0.400	0.332	0.296	90.883	0.771
KNN3	0.290	0.198	0.182	78.565	0.410
KNN5	0.288	0.199	0.182	78.797	0.414
KNN7	0.285	0.197	0.180	78.446	0.410
CNN	0.233	0.245	0.232	62.701	0.167

Figure 10.

Overall accuracy of all classifiers for image dataset1 (a), image dataset2 (b), image dataset3 (c).

Figure 11.

Shows kappa coefficient value of all classifiers of image dataset1 (a), image dataset2 (b), image dataset3 (c).

From Table 6, the proposed FMC shows the maximum number of class identification with high precision, recall and f1-score value for Class 2, Class 3, Class 4 as compared to DT and NBC.

Further in another direction, the performance of FMC is analysed by the other two metrics such as overall accuracy [15, 40] and kappa coefficient value [1, 13]. The overall accuracy of a classification model is the accuracy of the classifier for all classes rather than a single class, which is calculated from Eq. (17). Similarly, the objective of the kappa coefficient is to measure the agreement between classification and true value. It can be computed from Eq. (4.3).

$\displaystyle\text{Accuracy}=\frac{tp+tn}{tp+fp+fn+tn}$ (17) $\displaystyle\text{Kappa}=\frac{\text{observed accuracy}-\text{expected % accuracy}}{1-\text{expected accuracy}}$

Each classifier is trained with the training dataset of 3 features and seven number of class levels. The training accuracy of each classifier is evaluated as FMC (98%), DT (99%), NBC (95.15%), KNN3 (98.10%), KNN5 (97.18%), KNN7 (96.71%), and CNN (max 91%). The overall accuracy (testing accuracy), kappa coefficient value of each classifier is listed for image dataset1, image dataset2 and image dataset3 in Tables 7–9 respectively. It is quite difficult to analyse precision, recall, f1-score of each class level of each classifier, so the average precision, average recall, and average f1-score are also shown in the same table.

Figure 12.

Shows a comparison between kappa coefficient and overall accuracy of each classifier of image dataset1 (a), image dataset2 (b), image dataset3 (c).

In Table 7, the average precision, average recall, average f1-score is the highest in the case of proposed FMC. The FMC is considered to be more suitable classification model to handle overall classes as compared to other classifiers for image dataset1.

In Table 8, the FMC has the highest performance for three metrics such as recall is 0.605, overall accuracy is 91.168% and the kappa coefficient is 0.849. Further for the rest two metrics, FMC shows comparable result for image dataset2. This indicates the superiority of FMC over other classifiers.

In Table 9, the FMC model has the highest average precision and recall value for image dataset3. This signifies the FMC is a more suitable classification model to handle all classes.

The overall accuracy of all classifiers for the three images is also presented in Fig. 10a–c. The figure shows the overall accuracy of FMC is the highest among all considered classifiers.

The kappa coefficient value varies between 1 to $-$ 1. In Fig. 11a, the classifiers NBC, DT, CNN have negative kappa value, which indicates these classification models are considered to be not suitable for image dataset1. On other hands, the classifiers FMC has the highest positive kappa value 0.708 over KNN3, KNN5, KNN7 in Fig. 11a. Similarly, the classifiers DT and FMC have the highest positive kappa value over image dataset2, image dataset3 in Fig. 11b and c respectively.

A good classification model should have good accuracy and kappa value and both values should be close to each other. The proposed FMC classification model has closed values of overall accuracy and kappa value for image dataset1, image dataset2, and image dataset3 in Fig. 12a–c as compared to other classifiers.

5. Conclusion and future work

This research work presents a fusion based image classification approach for a land cover image. The input images are randomly taken from Google Earth for the classification maps. The employing of morphological (i.e., closing and opening) operations and fusion techniques are highlighted in the image pre-processing stage. Then designed classifier FMC is employed on pre-processed image data. A comparative result is shown taking a few other classifiers such as Naïve Bayes classifier, Decision tree, KNN3, KNN5, KNN7, and CNN. Entire experimentation is carried out taking five performance metrics such as precision, recall, f1-score, overall accuracy, and kappa coefficient metrics. Finally, the overall analysis shows FMC shows its superiority over other classifiers for all considered images.

The purpose of this model is used to classify seven numbers of features of any land cover image and develop a system which monitors the changes on the surface of the earth. However, the designed approach can’t be restricted to land cover image only and can be used to classify any RGB image which is the future work of this research approach.

References

Mahmon

Yaacob

Yusof

. Differences of image classification techniques for land use and land cover classification. in: 2015 IEEE 11th International Colloquium on Signal Processing & Its Applications (CSPA). 2015; pp. 90-94. IEEE.

Bagan

Yang

Takeuchi

Yamagata

. Sensitivity of the subspace method for land cover classification. The Egyptian Journal of Remote Sensing and Space Science. 2018; 21(3): 383-389.

Liu

Cao

Wang

Guan

. Learning from data: A post classification method for annual land cover analysis in urban areas. ISPRS Journal of Photogrammetry and Remote Sensing. 2019; 154: 202-215.

Matikainen

Karila

Hyyppa

Litkey

Puttonen

Ahokas

. Object-based analysis of multispectral airborne laser scanner data for land cover classification and map updating. ISPRS Journal of Photogrammetry and Remote Sensing. 2017; 128: 298-313.

Katuwal

Suganthan

Zhang

. An ensemble of decision trees with random vector functional link networks for multi-class classification. Applied Soft Computing. 2018; 70: 1146-1153.

Magidi

Ahmed

. Assessing urban sprawl using remote sensing and landscape metrics: A case study of City of Tshwane, South Africa (1984-2015). The Egyptian Journal of Remote Sensing and Space Science. 2018;

Lark

Mueller

Johnson

Gibbs

. Measuring land-use and land-cover change using the US department of agriculture’s cropland data layer: Cautions and recommendations. International Journal of Applied Earth Observation and Geoinformation. 2017; 62: 224-235.

Debonne

van Vliet

Verburg

. Future governance options for large-scale land acquisition in Cambodia: Impacts on tree cover and tiger landscapes. Environmental Science & Policy. 2019; 94: 9-19.

Yang

Jin

Danielson

Homer

Gass

Bender

Funk

. A new generation of the United States National Land Cover Database: Requirements, research priorities, design, and implementation strategies. ISPRS Journal of Photogrammetry and Remote Sensing. 2018; 146: 108-123.

10.

Gumus

Kirci

. Selection of spectral features for land cover type classification. Expert Systems with Applications. 2018; 102: 27-35.

11.

Nair

Ram

PGK

Sundararaman

. Shadow detection and removal from images using machine learning and morphological operations. The Journal of Engineering. 2019; 2019(1): 11-18.

12.

Yang

Lei

Yan

. Distributed multi-human location algorithm using naive bayes classifier for a binary pyroelectric infrared sensor tracking system. IEEE Sensors Journal. 2015; 16(1): 216-223.

13.

Nishii

Tanaka

. Accuracy and inaccuracy assessments in land-cover classification. IEEE Transactions on Geoscience and Remote Sensing. 1999; 37(1): 491-498.

14.

Huang

Siu

. Learning hierarchical decision trees for single-image super-resolution. IEEE Transactions on Circuits and Systems for Video Technology. 2015; 27(5): 937-950.

15.

Jamil

Bayram

. Tree species extraction and land use/cover classification from high-resolution digital orthophoto maps. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing. 2017; 11(1): 89-94.

16.

Shu

Cai

. A SVM multi-class image classification method based on DE and KNN in smart city management. IEEE Access. 2019; 7: 132775-132785.

17.

De Beurs

Stein

Bijker

. Incorporating open-source data for bayesian classification of urban land use from VHR stereo images. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing. 2017; 10(11): 4930-4943.

18.

Gašparović

Zrinjski

Gudelj

. Automatic cost-effective method for land cover classification (ALCC). Computers, Environment and Urban Systems. 2019; 76: 1-10.

19.

Wang

Chen

Shi

Wang

. SAR image classification method based on Gabor feature and K-NN. The Journal of Engineering. 2019; 2019(20): 6734-6736.

20.

. Multistep wind power forecast using mean trend detector and mathematical morphology-based local predictor. IEEE Transactions on Sustainable Energy. 2015; 6(4): 1216-1223.

21.

Buda

Maki

Mazurowski

. A systematic study of the class imbalance problem in convolutional neural networks. Neural Networks. 2018; 106: 249-259.

22.

Almutairi

Nikitas

Abdeljaber

Avci

Bocian

. A methodological approach towards evaluating structural damage severity using 1D CNNs. in: Structures. Elsevier. 2021; 34: pp. 4435-4446.

23.

Zhang

. Identification of power disturbances using generalized morphological open-closing and close-opening undecimated wavelet. IEEE Transactions on Industrial Electronics. 2015; 63(4): 2330-2339.

24.

Xia

Kamel

. Novel cooperative neural fusion algorithms for image restoration and image fusion. IEEE Transactions on Image Processing. 2007; 16(2): 367-381.

25.

Verma

. Multichannel image registration by feature-based information fusion. IEEE Transactions on Medical Imaging. 2010; 30(3): 707-720.

26.

Chang

Wang

Han

Chanussot

Huang

. Multisource data fusion and Fisher criterion-based nearest feature space approach to landslide classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing. 2014; 8(2): 576-588.

27.

Oktay

Bai

Guerrero

Rajchl

de Marvao

O’Regan

Rueckert

. Stratified decision forests for accurate anatomical landmark localization in cardiac images. IEEE Transactions on Medical Imaging. 2016; 36(1): 332-342.

28.

Masad

Al-Fahoum

Abu-Qasmieh

. Automated measurements of lumbar lordosis in T2-MR images using decision tree classifier and morphological image processing. Engineering Science and Technology, an International Journal. 2019;

29.

Zhu

Jia

. Multiple 3-D feature fusion framework for hyperspectral image classification. IEEE Transactions on Geoscience and Remote Sensing. 2018; 56(4): 1873-1886.

30.

Zhang

Roy

. Using the 500 m MODIS land cover product to derive a consistent continental scale 30 m Landsat land cover classification. Remote Sensing of Environment. 2017; 197: 15-34.

31.

Zhang

Sun

Zhang

Tong

. Land cover classification of the North China Plain using MODIS_EVI time series. ISPRS Journal of Photogrammetry and Remote Sensing. 2008; 63(4): 476-484.

32.

Huang

Quan

Liu

. Supervised sparse coding with decision forest. IEEE Signal Processing Letters. 2019; 26(2): 327-331.

33.

Simard

Saatchi

De Grandi

. The use of decision tree and multiscale texture for classification of JERS-1 SAR data over tropical forest. IEEE Transactions on Geoscience and Remote Sensing. 2000; 38(5): 2310-2321.

34.

Silva

Xavier

APC

da Silva

Santos

CAG

. Modelling land cover change based on an artificial neural network for a semiarid river basin in north-eastern Brazil. Global Ecology and Conservation. 2020; 21: e00811.

35.

Bakos

Gamba

. Hierarchical hybrid decision tree fusion of multiple hyperspectral data processing chains. IEEE Transactions on Geoscience and Remote Sensing. 2010; 49(1): 388-394.

36.

Shaker

Yan

LaRocque

. Automatic land-water classification using multispectral airborne LiDAR data for near-shore and river environments. ISPRS Journal of Photogrammetry and Remote Sensing. 2019; 152: 94-108.

37.

Kantakumar

Neelamsetti

. Multi-temporal land use classification using hybrid approach. The Egyptian Journal of Remote Sensing and Space Science. 2015; 18(2): 289-295.

38.

Kaur

Kumar

. Fusion of multi-modality medical images: a fuzzy approach. 2018 IEEE 3rd International Conference on Computing, Communication and Security (ICCCS), Kathmandu. 2018; pp. 112-115, doi: 10.1109/CCCS.2018.8586829.

39.

Jinju

Santhi

Ramar

Bama

. Spatial frequency discrete wavelet transforms image fusion technique for remote sensing applications. Engineering Science and Technology, an International Journal. 2019; 22(3): 715-726.

40.

Tyagi

Gupta

Singh

. A hybrid multi-focus image fusion technique using SWT and PCA. 2020 10th International Conference on Cloud Computing, Data Science & Engineering (Confluence), Noida, India. 2020; pp. 491-497, doi: 10.1109/Confluence47617.2020.9057960.

A fusion based land cover classification model using remote sensed images

Abstract

Keywords

1. Introduction

Table 1 List of some previously used supervised and unsupervised classification techniques on land cover image

3.1 Dataset description and image pre-processing

Table 2 List of land cover types with their re-colouring shades

4.1 Image pre-processing

Table 3 Comparison between three fusion techniques by quality measurement metrics PSNR and SNR

4.3 Result analysis

Table 6 Precision, recall, f1-score of each class level of proposed FMC and considered classifiers on image dataset3

References

Table 1
List of some previously used supervised and unsupervised classification techniques on land cover image

Table 2
List of land cover types with their re-colouring shades

Table 3
Comparison between three fusion techniques by quality measurement metrics PSNR and SNR

Table 6
Precision, recall, f1-score of each class level of proposed FMC and considered classifiers on image dataset3