An improved third term backpropagation algorithm – inertia expanded chebyshev orthogonal polynomial

Abstract

The standard backpropagation algorithm has already proven its effectiveness in most of the potential problems, but the major limitation is entrapment of local minima and slow convergence rate. To address these issues, a modified backpropagation algorithm has been proposed by adding a third term called inertia, the physical component used to accelerate the network towards the convergence without getting stuck into local minima. The Chebyshev polynomial form is a convenient method for expanding a function in a linear independent term. Inertia has been expanded using Chebyshev polynomial which is used as a third term in weight updation. The performance of the proposed algorithm outperforms the standard backpropagation algorithm (SBP) and the backpropagation algorithm with momentum (SBPM). The proposed algorithm was tested with the standard benchmark problems such as XOR problem, parity checking problem and dataset from UCI machine learning repository such as iris flower classification, wheat classification, breast cancer detection and wine classification. Experimental results show that the addition of the third parameter called inertia in the backpropagation algorithm gave better performance and faster convergence rate compared to the SBP and SBPM.

Keywords

Back propagation chebyshev polynomial feed forward neural network inertia momentum

1 Introduction

Artificial Neural Network is a logical model used to simulate the learning process of the human brain. The Artificial Neural Network basically work like actual biological neurons in the brain and it is composed of a collection of tiny processing unit called artificial neurons trained to perform the complex calculations. Since the human brain identifies and classifies those things by experience with a lot of learning process, the Artificial Neural Network also trained not to be programmed.

There are many different complex applications solved successfully by an Artificial Neural Network. Some of those are Handwritten character recognition, predicting future trends, health care, diagnosing diseases, medicine, engineering, manufacturing, ocean exploration, predicting an earthquake and so on [1 –5].

There are different types of feed-forward network available. Each network has unique characteristics and designed for some specific purposes. The architecture of feed-forward neural network consists of an input, hidden and an output layer. Each and every node in a layer is connected to every other node in the neighboring layer. The layers are all usually nonlinear processing units but it can use linear activation function to fire the output neuron. The goal of the network is to produce the expected output when a set of input is fed to the network. The output of one layer is given as input to another layer.

The backpropagation neural network algorithm (SBP) is one of the most popular and robust oldest algorithms proposed by Hinton, Rumelhart, and Williams [6]. The BP algorithm actually works to minimize the cost function namely, the mean square error between the actual output and desired output of the network [7]. It is widely used in a lot of practical problems and shows its efficiency especially in the classification problem [8], but there is some drawback such as inaccurate prediction result [9]. According to the research work [10, 11], reported clearly about the poor convergence rate of the backpropagation algorithm because of local minima and flatspot problem. But still, the BP algorithm is preferred and modified to give an enhanced version by a lot of researchers because of its simplicity in both calculation and implementation compared to other algorithms.

The major part of machine learning would be having an optimization algorithm as its basis. The Gradient descent is an optimization algorithm and it works to minimize the cost function by identifying the values of its parameters. The backpropagation algorithm uses the optimization method called gradient descent learning rule to reduce the mean square error between the actual and desired output by having careful selection of free parameters of the network such as learning rate, activation function, initial weight, and bias. The wrong selection of these parameter values lead the network to have slow convergence rate which would stuck into local minima and sometimes network error also.

Apart from initial weights and bias, the important parameters which are often used for weight adjustment is learning rate. Qambar Abbas et al. [12], proposed a novel variable scheme of learning rate. The author proved that the changes in learning rate also changes the training behavior of the backpropagation algorithm [12]. Elsadek and Zahraa [13], proposed an improved error backpropagation algorithm by using cross entropy as a error function and adaptive learning rate. Instead of keeping constant learning rate, the value of learning rate modified based on weight and gradient value of previous iteration gave better accuracy and convergence rate.

Instead of using backpropagation in classification problems, Kosbatwar and S. K. Pathan [14], used for pattern recognition problem and showed better performance in the problem of character recognition. Sornam and Thangavel [15], proposed three-term optical backpropagation algorithm by introducing a nonlinear error function to speed up the convergence rate. Ashwini Sapkala and Kulkarni [16], modified the backpropagation algorithm by added the white gaussian noise in the weighted sum entity and showed that noise injected backpropagation network required less number of epoch for convergence when compared with the SBP. Automatic plate recognition system developed by Joseph Tarigan et al. [17], where they employed the genetic algorithm to effectively identify the number of hidden neurons, learning rate and momentum rate which is used for the backpropagation algorithm and showed that the proposed method 37% faster in the convergence than the SBP. Qun Dai et al. [18], proposed the new two phased and ensemble scheme integrated backpropagation algorithm (TP-ES-BP) for face recognition problem and reveals that the TP-ES-BP algorithm work better to alleviate the problem of local minima in the SBP. Florin Gorunescu and Smaranda Belciug [19], boosted up the SBP with the aid of stimulus sampling technique applied at the output neuron and showed it is more efficient approach for better convergence.

Gibb and Johndar [20], has studied different types of modification in backpropagation algorithm such as Algorithmic modifications, Heuristic change, Regularizer, Techniques on networks and Generative Networks. In algorithmic modification, the Standard BP algorithm(SBP) was modified by introducing momentum term, Quasi Newton method as a second order method, and Resillent backpropation algorithm. In heuristic change method, variation were made in the manner by using initial weight selection technique, learning rate and weight updating method. In the regularizer method, the changes were made in the error function and minimizing the architecture of network. In the category of techniques on network, the modification were made in a way by varying network operations. Example of such work were recurrent neural network and coupled neuron. In final category, generative network, variation were made in SBP which gave the ability to grow. Example of such work was pruning.

In this paper, a modified backpropagation algorithm is proposed with the third term called inertia (MBPI) which is expanded in the form of Chebyshev orthogonal polynomial added in the weight updation rule which is used to accelerate the network for faster convergence. The proposed method shows better performance when compared to standard backpropagation algorithm (SBP) and backpropagation algorithm with momentum (SBPM). The paper is organized as follows: Section 2 explains the proposed algorithm followed by the simulation result in section 3. Section 4 discusses the conclusion.

2 Proposed modified backpropagation algorithm with inertia (MBPI)

2.1 Chebyshev orthogonal polynomials

Chebyshevpolynomial The Chebyshev polynomials of the first kind are a set of orthogonal polynomials defined as the solutions to the Chebyshev differential equation and denoted as recurrence relation T_n (x) [21]. They have applied as an approximation to the least squares, a special case of the Gegenbauer polynomial [22] with α = 0, where α is the index of the polynomial. They are also intimately connected with trigonometric multiple-angle formulae. The Chebyshev polynomials of the first kind are denoted as T_n (x), and are implemented in the Wolfram Language as Chebyshev T[n, x]. They are normalized in such a way that T_n (1) =1. The first few polynomials are illustrated for x ∈ [-1, 1], n=1, 2,..., 5.

The Chebyshev polynomial of the first kind can be defined by the contour integral function, $T_{n} (z) = \frac{1}{4 π i} \oint \frac{(1 - t^{2}) t^{- n - 1}}{(1 - 2 tz + t^{2})} dt$ (1) where the contour encloses the origin which is traversed in a counterclockwise direction.

The Chebyshev polynomials are defined through the identity, $T_{n} (cos θ) = cos (n θ)$ (2) The first few Chebyshev Polynomials are

$\begin{matrix} T_{0} (x) = 1, \\ T_{1} (x) = x, \\ T_{2} (x) = 2 x^{2} - 1, \\ T_{3} (x) = 4 x^{3} - 3 x, \\ T_{4} (x) = 8 x^{4} - 8 x^{2} + 1, \\ T_{5} (x) = 16 x^{5} - 20 x^{3} + 5 x \end{matrix}$ (3)

2.2 Inertia

Based on the first two law of physics stated by Sir Issac Newton,

An object at rest stay remain in rest

An object at motion stay remain in motion

In order to overcome this, apply a force required to move an object. Unlike momentum and mass, it is not the product of mass and velocity. Hence the property of inertia is more useful for the navigation system. The amount of mass possessed by an object can be calculated by measuring the extent the object observe its inertia. This can be done by measuring the amount of force applied to produce a certain acceleration.

Inertia can be classified as Translational Inertia and Rotational Inertia.

The translational inertia [23] is simply the product of the mass (m) and its acceleration (a), $L = ma$ (4) The moment of inertia or rotational inertia [23] can be calculated by using the following formula, $L = 1 / 2 * {mr}^{2}$ (5) where L is inertia, m is mass and r is the distance between the object and the axis of rotation.

Substituting Equation (3) in Equation (5) for mass (m), we get,

$\begin{matrix} L = 1 / 2 * ((16 * m * * 5) - (20 * m * * 3) \\ + (5 * m)) * r^{2} \end{matrix}$ (6)

2.3 Modified SBP with Inertia (MBPI)

The standard backpropagation network consists of one input layer, one output layer and one hidden layer between the input and output layer. These layers have to be fully connected in a feedforward fashion so that the neurons in the input unit fully connected to neurons in the hidden layers, similarly the connections in between the hidden and output layer.

The modified Backpropagation network with inertia augmented the weight connection (U) added with momentum (β) and inertia (γ) in the input to hidden layer whereas in the hidden to output layer weights (V) are added with momentum (β) alone. Figure 1 shows the structure of the proposed modified neural network,

Fig.1

Proposed modified neural network with inertia.

The modified algorithm is an enhanced version of standard backpropagation algorithm containing the third term inertia which is expanded in the form of Chebyshev polynomial that help to increase the convergence speed of the network. The addition of inertia term helps to get out from the chance of stuck into the local minima.

Modified Backpropagation Algorithm with inertia:

Generate random weight matrices U and V between input to hidden layer and hidden to output layer respectively.

From each training pattern set T={X_i, O_i} where X_i is the input vector and O_i is the output vector, feed the input vector to the input layer.

Compute net input for each hidden node using $h_{j} = {Net}_{j} = \sum_{i = 1}^{n} u_{ij} * x_{i}, j = 1, 2, \dots, q$ (7)

Compute output for each hidden nodes $g (h_{j}) = \frac{1}{(1 + \exp (- h_{j})}$ (8) where g(.) is the sigmoid activation function

Compute net input for each output node $Z_{k} = {Net}_{k} = \sum_{j = 1}^{m} v_{jk} * g (h_{j}), k = 1, 2, \dots, p$ (9)

Calculate output for nodes in the output layer by applying activation function $g (Z_{k}) = \frac{1}{(1 + \exp (- Z_{k}))}$ (10)

Calculate gradient of output node $g^{'} (Z_{k}) = g (Z_{k}) * (1 - g (Z_{k}))$ (11)

Compute error at output nodes as follows $δ_{k} = (O_{k} - g (Z_{k})) * g^{'} (Z_{k})$ (12)

Error at hidden neuron can be calculated as $δ_{j} = g^{'} (Z_{k}) * \sum_{k = 1}^{m} δ_{k} * v_{jk}$ (13) Output layer weight updation:

The weight updation with momentum, $V (t + 1) = V (t) + α * δ_{k} * h_{j} + β * V (t - 1)$ (14) where α is the learning rate and β is the momentum factor.

Modified weight updation for Hidden layer:

The weight updation with momentum and inertia: $\begin{matrix} U (t + 1) = U (t) + α * δ_{j} * x_{i} + β * U (t - 1) \\ + (1 / 2 * ((16 * γ^{5}) - (20 * γ^{3}) + (5 * γ)) \\ * (U (t) - U (t - 1))^{2} \end{matrix}$ (15) where α is learning rate, β is momentum factor and γ is inertia’s mass term expanded by Chebyshev orthogonal polynomial using Equation (6), assigned with small positive value and r expressed as the distance between old weight U(t-1) and new weight U(t) using Equation (5).

Repeat Step 2 to 11 until mean square error reduced to 0.01.

3 Simulation results

The proposed system was developed by using Python (version 3.5) in the Spyder environment, 64 bit windows 10 OS with the RAM 4 GB and the processor is Intel core I5, 7^th Generation. The speed of the processor is 2.50GHz.

The performance of the proposed algorithm MBPI has been experimented using selected example benchmark problems such as (i) XOR problem, (ii) Parity Checking problem and four datasets [24] from UCI Machine Learning repository. The weights are randomly generated in the range of (-0.01, 0.01) for binary class problems and (-0.05, 0.05) for multiclass problem. For all the selected problems, the hyperbolic tangent activation function (1 - e^-x)/1 - e^x) was used to fire the hidden neurons and if the problem is binary classification, then sigmoid activation function 1/(1 + e^-x) has been used to fire the output neuron, otherwise softmax function $e (X_{i}) / \sum_{j = 0}^{k} e (X_{j})$ used for the output layer. The convergence of the proposed algorithm was measured by performance metric namely, Mean Square Error (MSE) and the targeted MSE was 0.01. For all the problems, the hyperparameters such as Learning Rate (LR), momentum (β) and inertia (γ) were experimented with different values to find the optimum value. Table 1 represents the simulation result of the proposed MBPI for the experimented dataset. Table 2 shows the comparison performance of the proposed algorithm with standard BPN and standard BPN with Momentum by presenting number of epochs and time for convergence to the targeted mean square error 0.01.

3.1 XOR Problem

The “exclusive or” problem is one of the classic problems in artificial neural network research. It is the problem of predicting the output of XOR gate for given two binary inputs. The function returns a true value as 1 if both inputs are not same and return a false value as 0 if both inputs are same. The proposed algorithm converged by its 160^th epoch with MSE of 0.01. For the same number of epoch, the SBP and SBPM gave MSE of 0.96 and 0.41 respectively. Figure 2 reveals the implementation result of SBP, SBP with Momentum and MBP with Inertia for XOR problem by presenting learning curve of these algorithms. This clearly shows the speedy convergence for the proposed method.

Fig.2

Learning curve plot for XOR.

3.2 Parity checking problem

The parity checking is a method used to confirm that the data transmitted in between the communication node is correct. It is the simplest form of error detecting code. The parity bit is padded to the original binary string to indicate it as odd parity or even parity. If the number of bit 1 in the original string is odd, then it is odd parity otherwise it is termed as even parity. 2⁵ combinations of 5-bit binary string has been generated and given as input to the network. First 16 combination of inputs were used for training and remaining 16 for testing the network. As compared with existing method SBP and SBPM which gave the classification accuracy of 95% and 97% for 500 epochs with a mean square error of 2.50 and 0.63, the proposed MBPI reached its targeted MSE 0.01 with the classification accuracy of 99.9% respectively.

Figure 3 shows the performance of algorithms by representing a learning curve for 500 epochs.

Fig.3

Learning curve plot for parity checking problem.

3.3 IRIS multiclass classification problem

Iris is the best-known dataset [24] from UCI machine learning repository. The iris dataset contains 150 instances to classify three different type of iris flowers such as Iris Setosa, Iris Virginica, and Iris versicolor. This dataset is also called Fisher dataset. The proposed MBPI converged to the target mean square error of 0.01 by its 200^th epoch. The MSE of SBP and SBPM were 0.05 and 0.04 respectively for the same number of epoch. The shape features for classification as follows,

Sepal Length

Sepal Width

Petal Length

Petal Width

Figure 4 shows the learning curve for comparison algorithms.

Fig.4

Learning curve plot for IRIS dataset.

3.4 Wheat classification problem

The second major crop in India is wheat. The classification of wheat is very important and challenging task for breeders and geneticists. The wheat classification dataset [24] from UCI machine learning repository to classify three different major type of wheats such as kama, Rosa and Canadian. The dataset have 210 instances where 70 samples for each type of wheat. The geometrical parameters for predictions are,

area A

perimeter P

compactness C = 4 * pi * A/P²

length of kernel

width of kernel

asymmetry coefficient

length of kernel groove.

Figure 5 illustrates the performance of the MBPI in the wheat dataset by plotting the learning curve for 120 epochs with MSE of 0.01 for MBPI, 0.15 for SBP and 0.39 for SBPM.

Fig.5

Learning curve plot for wheat dataset.

3.5 Breast cancer dataset

The most common dreadful disease for the women in all over the world is breast cancer. In recent days, the number of women suffered by this disease getting increases. Different methodologies proposed for earlier detection of breast cancer. The breast cancer dataset [24] taken from UCI Machine learning repository contains 699 instances and 10 predictors to diagnose the cancer. The dataset have 458 instances for malign and 241 instances for benign. The features for predictions are

Clump thickness

Cell size

Cell shape

Marginal adhesion

Single epithelial

Bare nuclei

Bland chromatin

Normal nucleoli

Mitoses.

Figure 6 shows the learning curve of the algorithms for 150 epochs with a mean square error of 0.03, 0.04 and 0.01 for SBP, SBPM and MBPI respectively.

Fig.6

Learning curve plot for breast cancer.

3.6 Wine classification dataset

The wine dataset [24] from UCI machine learning repository contain 178 samples belonging to three different type of wine grown in specific area of Italy and it is categorized based on following 13 chemical analyses such as

Alcoholic

Malic

Ash

Alcalinity

Magnesium

Phenols

Flavanoids

Non flavanoids

Proanthocyanins

Color

Hue

Dilution

Proline.

Figure 7 represents the learning curve of the algorithms for 40 epoch with MSE of 0.01 for MBPI whereas 0.05 and 0.02 for SBP and SBPM respectively which illustrate the better performance of MBPI.

Fig.7

Learning curve plot for wine classification.

Table 1

Simulation results of selected problems

Problem	Structure	LR	Momentum	Inertia	Acc%
XOR Problem	2-2-1	0.03	0.5	0.005	98%
Parity Problem	5-3-1	0.05	0.90	0.0005	99.9%
IRIS	4-5-3	0.03	0.90	0.0005	98%
Wheat	7-10-3	0.05	0.75	0.0005	99%
Breast Cancer	10-15-1	0.03	0.75	0.0003	99%
Wine	13-15-3	0.05	0.90	0.0003	100%

Table 2

Comparison table for the proposed MBPI with SBP and SBPM

Dataset	Algorithm	Epoch	MSE	Time/Sec
XOR	SBP	900	0.01	32.9
	SBPM	447	0.01	18.32
	MBPI	160	0.01	9.41
Parity	SBP	1200	0.01	10.57
	MBPI	893	0.01	8.97
	MBPI	500	0.01	3.45
IRIS	SBP	600	0.01	78.90
	SBPM	430	0.01	43.63
	MBPI	200	0.01	26.70
Wheat	SBP	600	0.01	130.61
	SBPM	1100	0.01	280.68
	MBPI	120	0.01	0.99
Breast Cancer	SBP	240	0.01	206.29
	SBPM	563	0.01	567.03
	MBPI	150	0.01	123.04
Wine	SBP	90	0.01	48.50
	SBPM	50	0.01	18.65
	MBPI	40	0.01	14.50

3.7 Confusion matrix

Confusion Matrix is one of the performance measure used to describe the classification capability of the classifier. It is applicable when output can be two or more classes. Table 3, depicts the results of the confusion matrix predicted by using SBP, SBPM, and MBPI. The following measures can be determined by using confusion matrix such as Recall, Precision and F1 score. These measures can be calculated based on four basic terminologies:

TP – Both actual and predicted are true.

TN – Both actual and predicted are false.

FP – Actual sample as false but predicted as true.

FN – Actual sample as true but predicted as false.

Recall

Recall is the ratio between the number of true positive and number of positive samples in the dataset. High recall value indicates the classes are correctly recognized and it can be defined by using following equation

$Recall = \frac{TP}{TP + FN}$ (16)

The recall value obtained by using MBPI is around 1 for all the test dataset given in Table 3, shows that the class is at most correctly classified by the proposed classifier and the results have less number of FN.

Precision

Precision obtained by dividing the total number of positive samples correctly classified by the classifier with the number of samples predicted as positive. The precision can be expressed by the relation,

$Precision = \frac{TP}{TP + FP}$ (17)

In Table 3, the precision value is almost 1 for all the experimented dataset in the case of MBPI, reveals that positive samples classified as positive are indeed positive which means there must be a less number of FP.

F1 Measure

The F1 score is the overall measure of the model combine both recall and precision. If the value of F1 score is around one then the model is perfect and the model is failure if it is zero. The F1 score can be computed by $F 1 = 2 * \frac{Precision * Recall}{Precision + Recall}$ (18)

The F1 measure computed from the confusion matrix of the MBPI algorithm given in Table 3 is around one for all the test dataset proved that the proposed classifier is perfect.

Table 3

Average metrics from confusion matrix

Dataset	Algorithm	Recall	Precision	F1 Score
XOR	SBP	0.91	0.90	0.90
	SBPM	0.93	0.93	0.93
	MBPI	0.98	0.98	0.98
Parity	SBP	0.93	0.92	0.92
	SBPM	0.95	0.94	0.95
	MBPI	1.00	0.99	0.99
IRIS	SBP	0.94	0.95	0.95
	SBPM	0.96	0.96	0.96
	MBPI	0.98	0.98	0.98
Wheat	SBP	0.90	0.89	0.88
	SBPM	0.80	0.68	0.56
	MBPI	0.99	0.99	0.99
Breast Cancer	SBP	0.98	0.98	0.98
	SBPM	0.97	0.96	0.97
	MBPI	0.99	0.99	0.99
Wine	SBP	0.97	0.96	0.97
	SBPM	0.99	0.98	0.99
	MBPI	1.00	1.00	1.00

3.8 Cross validation

Cross Validation is one of the technique used to reserve few subset of data which is not given for training the model can be used as test set to cross validate the performance of the model. Different cross validation techniques are available such as Leave one out cross validation, k fold cross validation, stratified k fold cross validation, Adversarial validation and so on. In this work, k fold method used for cross validation. In this technique, the dataset are divided into k folds, and in each fold, k-1 fold of the dataset used for training and k^th fold used for testing. Repeat this process until all k folds served as the test set. The average of the validation errors can be used to predict the performance of the model.

Table 4
Performance of MBPI using K Fold cross validation

Dataset KFold Value Training Acc Testing Acc

XOR 2 0.97 0.98

Parity 2 1.00 1.00

IRIS 5 0.99 0.94

Wheat 10 0.97 0.93

Breast Cancer 10 0.98 0.95

Wine 7 1.00 0.99

Dataset	KFold Value	Training Acc	Testing Acc
XOR	2	0.97	0.98
Parity	2	1.00	1.00
IRIS	5	0.99	0.94
Wheat	10	0.97	0.93
Breast Cancer	10	0.98	0.95
Wine	7	1.00	0.99

3.9 McNemar’s test

In the model machine learning era, a wide variety of learning algorithms proposed for the classification and pattern recognition problems. Hence there is a need for the statistical approach to show the significance of the proposed model. Statistical hypothesis testing is used to confidently present the results of the proposed model. Dietterich [25], concluded that 5x2 cross validation, a statistical test is best when the algorithm need to be executed ten times to analyze the performance of the algorithm and for the algorithm executed only once then McNemar’s Test is best.

McNemar’s Test is one of the non-parametric statistical test and in general used for paired nominal data. It is also called "within subject chi-squared test". In the context of machine learning, McNemar’s test is used to compare the predictive performance of the two classifiers on a single test dataset. Nowadays, this statistical test widely used to test the efficiency of the deep learning models, since it is more suitable for large datasets. The following steps involved to perform the McNemar’s test.

Find the contingency table which contains the transformation of the predicted result of two classifiers. Table 5 represents the structure of the contingency table,

Compute statistics (“chi-squared”) from the contingency table using the formula, $statistics = \frac{(Y / N - N / Y)^{2}}{(Y / N + N / Y)}$ (19)

Fix the Alpha value as 0.05.

H₀, the Null Hypothesis means the two classifiers has the same proportion of error, so there is no significant differences between the two models.

To find the significance of the model, calculate the p-value using Equation (20), $p = 2 \sum_{\binom{i = b}{n}}^{n} i 0 . 5^{i} (1 - 0.5)^{n - i}$ (20) Where n = b+c, b is the number of samples correctly classified by classifier 1 and the same samples misclassified by classifier 2. c represents the number of samples exactly predicted by classifier 2 and the same samples wrongly classified by classifier 1.

The output of the McNemar’s test can be interpreted like as follows,

if p-value >alpha, then there is no significant difference between two classifiers. Both models disagree with the same amount of error then accept H₀.

if p-value<alpha, then there is a significant difference between the two models. Reject H₀ because both classifiers have a different proportion of errors.

Table 5

Structure of the contingency table

	Classifier2 Correct	Classifier2 Incorrect
Classifier1 Correct	Y/Y	Y/Y
Classifier1 Incorrect	N/Y	N/N

Table 6

McNemar’s test results

DATASET	CLASSIFIER 2	MBPI (CLASSIFIER1)
		STATISTICS	P-VALUE
XOR	SBP	1.333	0.125
	SBPM	0.000	0.411
PARITY	SBP	0.000	0.041
	SBPM	0.000	0.039
IRIS	SBP	0.000	0.031
	SBPM	0.000	0.008
WHEAT	SBP	5.000	0.011
	SBPM	7.000	0.043
BREAST	SBP	2.000	0.013
CANCER	SBPM	2.000	0.039
WINE	SBP	0.000	0.031
	SBPM	0.000	0.016

3.10 Discussion

In general, the standard backpropagation algorithm utilizes the two hyperparameters called learning rate and momentum factor to train the artificial neural network. The goal of this algorithm starts at some point in the error function which one is defined over the weights and attempts to reach the global minimum of the function. But in real time cases, the error surface of the nonlinear network is more complex compared with a linear network and contains numerous local minima. By adding the third term called Chebyshev expanded inertia in the weight updation rule changes the path which helps to reaches the global optimum result and gives fast convergence rate without getting stuck in local minima problem.

From the results given in Table 3, the Recall value of the proposed MBPI for the experimented dataset, reveals that most of the positive samples were correctly classified and these values never moved lesser than precision which makes sure that those predicted positive were indeed positive. The F1 score is 1 and around 1 for all the dataset strongly conveyed that the proposed model is the best classifier. The statistical performance metrics of the proposed algorithm derived from the confusion matrix and cross-validation techniques employed strongly reveals that the modified algorithm MBPI gave better performance and faster convergence rate when compared to the existing algorithms SBP and SBPM.

The significance of the proposed model MBPI is proved by conducting the McNemar’s test to compare with two classifiers SBP and SBPM respectively. From Table 6, the p-value is less than 0.05 for all the test dataset which strongly represents the proposed model have significant differences when compared with the other two classifiers except XOR problem, since the sample size is very low in the XOR problem. McNemar’s test mainly depends on the probability of the error of two classifiers, the result was not significant in the case of XOR problem.

4 Conclusion

In this paper, a modified backpropagation algorithm has been proposed by adding a third term called inertia, which is expanded in the form of Chebyshev orthogonal polynomial which makes the term as an independent component and more suitable for function approximation. Because of so many advancements, the Chebyshev polynomial is widely used in many areas of numerical analysis. The value of the cost function and the number of epochs for convergence were significantly reduced when it is compared with SBP and SBPM. The Chebyshev expanded inertia make the network move towards the speedy convergence and it is proved by implementing and tested with the standard benchmark problems of XOR problem, Parity checking problem and UCI datasets namely iris flower classification, wheat classification, breast cancer detection, and wine classification. The statistical measures derived from the confusion matrix, the results of k fold cross validation technique and the results of the McNemar’s test shows that the proposed modified algorithm MBPI gave better convergence rate compared to the existing algorithms SBP and SBPM. As the real time large dataset is imbalanced in nature, the proposed algorithm would be applied in the imbalanced dataset in the future.

References

Attigeri

Neural network based handwritten character recognition system, International Journal of Engineering and Computer Science 7(03) (2018), 23761–23768.

Bhatia

, Pasari

and Mehta

Earthquake forecasting using artificial neural networks, International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences 42(05) (2018), 823–827.

Shahid

, Rappon

and Berta

Applications of artificial neural networks in health care organizational decision-making: A scoping review,e, PLOS One 14(02) (2019), 0212356.

Bolton

and Zanna

Applications of deep learning to ocean data inferenceand subgrid parameterization, Journal of Advances in Modeling Earth Systems 11 (2019), 376–399.

Essid

, Laga

and Samir

Automatic detection and classification of manufacturing defects in metal boxes using deep neural networks, PLoS One 13(11) (2019), e0203192.

Mohd Nawi

, Nawi , Gillani

, Muhammad

and Khan

, A new bat based backpropagation (BAT-BP) algorithm, Advances in Intelligent Systems and Computing 240 (2013), 395–404.

Vora

, Yagnik

and Scholar

A Survey on Backpropagation Algorithms for Feedforward Neural Networks, Citeseer, 2013.

Abiodun

O.I.

, Jantan

, Omolara

A.E.

and Dada

K.V.

State-of-the-art in artificialneural network applications: Asurvey, Elsevier 4 (2018), e.80093.

Addulkadir

S.J.

, Shamsuddin

S.M.

and Sallehuddin

Moisture prediction in maize using three term back propagation neural network, International Journal of Environmental Science and Development 3(2) (2012), 199–204.

10.

Heidelberg

Editors: Nazri Mohd.

Nawi

, Mohammad Zubair

Rehman

, Abdullah

Khan

, A New Back-Propagation Neural Network Optimized with Cuckoo Search Algorithm, Springer, (2013), pp. 413–426.

11.

Whittington

J.C.R.

and Bogacz

Theories of error back-propagation in the brain, Trends in Cognitive Sciences 23(3) (2019), 235–250.

12.

Abbas

, Ahmad

and Imran

Variable learning rate based modification in backpropagation algorithm (Mbpa) of artificial neural network for data classification, Science International 28(3) (2016), 2369–2380.

13.

Ibrahim

E.H.

and Mohamed

Z.E.

, Improving error back propagation algorithm by using cross entropy error function and adaptive learning rate, International Journal of Computer Applications 161(8) (2017), 5–9.

14.

Kosbatwar

S.P.

and Pathan

S.K.

, Pattern association for character recognition by Back-Propagation algorithm using Neural Network approach, International Journal of Computer Science and Engineering Survey 3(1) (2012), 127.

15.

Sornam

and Thangavel

, An improved three-term optical backpropagation algorithm, International Journal of Artificial Intelligence and Soft Computing 2(4) (2011), 321–333.

16.

Sapkal

and Kulkarni

U.V.

, Modified Backpropagation with Added White Gaussian Noise in Weighted Sum for Convergence Improvement, Procedia Computer Science, Elsevier, 143 2018, pp. 309–316.

17.

Tarigan

, Diedan

and Suryana

, Plate recognition using backpropagation neural network and genetic algorithm, Procedia Computer Science, Elsevier 116 (2017), 365–372.

18.

Dai

, Ma

and Xie

Q.Y.

, A two-phased and Ensemble scheme integrated Backpropagation algorithm, Applied Soft Computing, Elsevier 2014(24), 1124–1135.

19.

Gorunescu

and Belciug

, Boosting backpropagation algorithm by stimulus-sampling: Application in computer-aided medical diagnosis, Journal of biomedical informatics, Elsevier 63 (2016), 74–81.

20.

Gibb and Jondar, Back propagation Family Album, 1996.

21.

http://mathworld.wolfram.com/ChebyshevPolynomialoftheFirstKind.html.

22.

Sornam

and Vanitha

, Application of chebyshev neural network for function approximation, International Journal of Computer Sciences and Engineering 6(4) (2018), 201–204.

23.

https://sciencing.com/inertia-object-8135394.html.

24.

https://archive.ics.uci.edu/ml/datasets/.

25.

Dietterich

T.G.

, Approximate statistical tests for comparing supervised classification learning algorithms, Neural Computation 10(7) (1998), 1895–1923.