Analysis of the trade-offs between parametric and non-parametric classifiers for classification of faults in pneumatic actuator. Case: DAMADICS

Abstract

This paper disseminates an extra dimension of substantial analysis demonstrating the trade-offs between the performance of Parametric (P) and Non-Parametric (NP) classification algorithms when applied to classify faults occurring in pneumatic actuators. Owing to the criticality of the actuator failures, classifying faults accurately may lead to robust fault tolerant models. In most cases, when applying machine learning, the choice of existing classifier algorithms for an application is random. This work, addresses the issue and quantitatively supports the selection of appropriate algorithm for non-parametric datasets. For the case study, popular parametric classification algorithms namely: Naïve Bayes (NB), Logistic Regression (LR), Linear Discriminant Analysis (LDA), Perceptron (PER) and non-parametric algorithms namely: Multi-Layer Perceptron (MLP), k Nearest Neighbor (kNN), Support Vector Machine (SVM), Decision Tree (DT) and Random Forest (RF) are implemented over a non-parametric, imbalanced synthetic dataset of a benchmark actuator process. Upon using parametric classifiers, severe adultery in results is witnessed which misleads the interpretation towards the accuracy of the model. Experimentally, about 20% improvement in accuracy is obtained on using non-parametric classifiers over the parametric ones. The robustness of the models is evaluated by inducing label noise varying between 5% to 20%. Triptych analysis is applied to discuss the interpretability of each machine learning model. The trade-offs in choice and performance of algorithms and the evaluating metrics for each estimator are analyzed both quantitatively and qualitatively. For a more cogent reasoning through validation, the results obtained for the synthetic dataset are compared against the industrial dataset of the pneumatic actuator of the sugar refinery, Development and Application of Methods for Actuator Diagnosis in Industrial Control Systems (DAMADICS). The efficiency of non-parametric classifiers for the pneumatic actuator dataset is well proved.

Keywords

Parametric classifiers non-parametric classifiers trade-offs pneumatic actuator DAMADICS accuracy interpretability

1 Nomenclature

Bias

Number of classes in the datasets

Centre terms in rbf

Rod displacement in synthetic dataset, %

Dataset

DAMADICS

Development and Application of Methods for Actuator Diagnosis in Industrial Control Systems

Decision Tree

f₁-f₃, f₁₆,

Different faults in synthetic

f₁₈, f₁₉

and benchmark datasets

FDA

Fischer Discriminant Analysis

Flow rate of medium in industrial dataset, t/h

F_r

Flow rate of medium in synthetic dataset, t/h

Interpretability

Number of nearest neighbours in kNN

kNN

k Nearest Neighbours

Regularization coefficient

LDA

Linear Discriminant Analysis

Logistic Regression

Servomotor

Number of features in datasets

MLP

Multi-Layer Perceptron

Number of nodes in decision trees

Number of samples in dataset

Naïve Bayes

Non-Parametric

Positioner

Parametric

PER

Perceptron

P_i

Inlet valve pressure, in MPa

P_o

Outlet valve pressure, in MPa

P_r

Predictivity

R

Robustness

rbf

Radial Basis Function

Random Forest

S_i

Simplicity

SMOTE

Synthetic Minority Oversampling Technique

Controller Set point

S_t

Stability

svd

Singular value decomposition

SVM

Support Vector Machine

t_fdev

Time taken by the fault to fully develop

T_m

Temperature of medium, in °C

t_start

Start time of the fault

Control Valve

Rod displacement in industrial dataset, %

x_i

Samples in dataset

y_i

Targets variables in dataset

Learning rate of models

β _0, β₁ - β _n

Regression values

Distance based hyperparameter for rbf

Discriminant function

Label noise

Percentage of label noise induced

Parameters

Class means

Standard deviation

ω _i

Updated weights

1 Introduction

In the context of predictive analytics, the objective of classification models is to anticipate exact class labels for the given input data. Real world scenarios such as: sentiment analysis, text and speech analysis, remote sensing, medical signal analysis, fault classification and diagnosis [1] in industrial applications, quality assessment [9] etc., demand classification of data under different situations. Classification models can be either supervised, unsupervised or semi-supervised [18] based on their learning mode. They can be binary, multiclass or multi label classification based on the count of the target classes assigned. Building exemplary classifier models require command over the structure of the dataset like distribution of the data, correlation between the features, outlier information and consistency in the class distribution. It is observed that, algorithms that work with normally distributed dataset, act differently when applied to a dataset following binomial or multinomial distributions. These algorithms are called parametric and non-parametric algorithms, respectively. In this work, some popular parametric algorithms namely: Naïve Bayes (NB), Perceptron (PER), Logistic Regression (LR), Linear Discriminant Analysis (LDA) and some of the popular non-parametric algorithms namely: Multi-Layer Perceptron (MLP), k Nearest Neighbor (kNN), Support Vector Machines (SVM), Decision Tree (DT), Random Forest (RF) models are chosen for analysis [15].

Parametric algorithms are assumption-based models having fixed and finite parameters. These parameters are independent about the sample size [15], such that any prediction ‘y’ on the given data ‘- D’, is independent of the data size, provided that the parameters ‘θ’ are given. This is given in Equation (1). $P (y | θ, Đ) = P (y | θ)$ (1)

The parameter ‘θ’ is an information rich content that aids in executing the task of the algorithm. Statistically, the typical assumptions made by a parametric algorithm include: normality, homogeneity of variance, linearity and feature independence. Naive Bayes classifier is a simple, fast and generative model classifier satisfying the assumptions of normality and independence between features [13]. It handles categorical inputs and multiclass classification efficiently. A slightly evolved form of Naïve Bayes is the Perceptron Classifier with a slack in independence assumption. Perceptron classifier is a straightforward, discriminative parametric algorithm. The basics and evolution of perceptron classifier model in neural networks [14] has taken different heights based on requirement of the applications. The linear threshold function of the Perceptron allows the algorithm to work at ease for linear models. Logistic Regression classifier is a discriminant model classifier [28] having strong linearity assumption between the features. It works with sigmoidal function and fits well for non-linear models. The Logistic Regression models specifies the conditional distribution of the model. Both Perceptron and Logistic Regression classifiers when used for multiclass classification binarizes all the classes and converts them to a pair wise [29] problem to formulate the classification strategy. Linear Discriminant Analysis follows the normality and homogeneity of variance assumptions [2] and provides the joint distribution information about the model. All the above-mentioned algorithms have strong and systematic mathematical formulation.

Non-parametric classification algorithms do not care about the normality of the data. They are quasi assumption free models that handle non-normal data. They have infinite parameters which vary according to the variations in the data [15]. The multi-layer perceptron (MLP) classifier is a type of artificial neural network that consists of multiple layers of interconnected artificial neurons. MLPs are inherently parametric, however, through proper regularisation techniques, it can act as a non-parametric classifier [4]. The k Nearest Neighbors is a memory-based algorithm where the model depends on the dataset only. The ‘lazy predict’ [19] character of kNN performs instance-based learning by allowing only storage during the training phase. It is well suited to identify more complex patterns in the data. Support vector machines do not have any probabilistic models nor do they depend upon the normality assumptions [10]. They are model free and work by applying generalized dot products to determine the correlation between data points. They are efficient for high dimensional data providing less computational complexity. Decision tree algorithms are widely used discriminative tree classifier models [31]. The trees are built in baseline or cost sensitive manner in the aim to decrease the impurities in each split. The decision split points and the decision rules grow in size as the complexity of the data increases. Random forest is an extended version of decision tree method. They are ensemble of trees which are generally weak learners and average out the values in the algorithm [3]. A well-organized year wise survey on random forest methods is given in [24]. Almost all aforementioned classifiers will also perform the regression tasks. From the literature it is inferred that, the non-parametric algorithms are not mathematically strong like parametric algorithms.

The motivation behind this work lies in the fact that the compatibility of applying parametric (P) and non-parametric (NP) algorithms for parametric and nonparametric datasets is a question of choice. In this work, the performances of various parametric and non-parametric classifier models for a non-parametric dataset DAMADICS (Development and Application of Methods for Actuator Diagnosis in Industrial Control Systems) are evaluated. Many works on the fault detection (classification) and diagnosis have been reported in literature including pneumatic actuator fault detection [8] where, classification is treated as a one-class problem.

The performance of the classifiers built is evaluated using performance metrics. The choice of classification metric can become exigent when applying for different datasets like balanced and imbalanced datasets. Using regular metrics to evaluate the classifier under all categories may lead to sub optimal [7] classification models.

The main contributions of this work are 5 folds:

A synthetic dataset is acquired from modeling the pneumatic actuator using DABLib library under normal and faulty conditions.

The imbalance in normal and faulty class distribution of the pneumatic actuator dataset is addressed using oversampling technique.

Both binary and multiclass classification is performed using both parametric and non-parametric classifiers

The trade-offs between the performance of the classification algorithms are analyzed under different criteria like, flexibility, accuracy, interpretability, robustness and complexity.

The analysis results are quantified using suitable performance metrics

The highlight of the work includes the validation of the proposed classifier models on a benchmark industrial dataset (DAMADICS) of the pneumatic actuator in a sugar refinery located in Poland.

Accordingly, this paper consists of four main sections. Section 1 begins with an introduction to the classification and the motivation behind the contribution of this work towards the analysis of trade-offs using definite metrics. The description of the synthetic dataset, machine learning models used and the characteristics of these models are discussed in Section 2. The performance analysis of each classifier and the trade-offs between the models built on the synthetic dataset is well articulated in Section 3 under Results and Discussion. The results obtained from the synthetic dataset is validated against the industrial dataset of the DAMADICS pneumatic actuator in Section 4. Finally, the concluding remarks are presented in Section 5.

An insight into the synthetically generated dataset, algorithms employed and the characteristics analyzed are essential for better understanding for the readers. Hence, Section 2 deals with the elucidation of the dataset, the algorithms chosen and the performance indices considered to evaluate the algorithms.

2 Materials and methods

2.1 Synthetic dataset

The dataset for testing the algorithms is a synthetic dataset derived by modelling the pneumatic actuator [16] system using DAMADICS simulink library [16]. The schematic diagram of the pneumatic actuator is shown in Fig. 1.

Fig. 1

Representation of the DAMADICS actuator scheme [16].

From Fig. 1, it is noted that, the actuator system consists of three parts, namely: Positioner (P), Servomotor (M) and the control valve (V). The measurement variables considered include: controller set point (SP), water flow rate (F_r), inlet pressure of control valve (P_i), outlet pressure of control valve (P_o), temperature of the water (T_m) and the displacement (D) of the actuator rod. The setpoint (SP) and the rod displacement (D) both have a transducer range of 100%. The pressure transducer ranges from (0-4) MPa. The flow transducer range for water flow rate (F_r) measurement ranges from (0-40) t/h and the temperature transducer range is (0-200) °C. However, during data generation, all data are normalized in the range of 0-1.

The data was generated by building the actuator model and invoking different fault conditions namely: clogging, sedimentation and erosion with varying fault intensities and also in normal mode. The parameters used during the simulation of various faults are tabulated in Table 1.

Table 1

Parameters used during the simulation of different faults [16]

Parameters	f₁-Clogging	f₂-Sedimentation	f₃-Erosion
t_start, s	900	900	900
t_fdev, s	900	4600	1500
f_i(min)	0	0	0
f_i(max)	0.5	1	1
Type	Abrupt, medium	Gradual, medium	Gradual, short
Direction	+ve	+ve	+ve

From Table 1, the parameters namely: fault beginning time (t_start), the time taken by the fault to develop completely (t_fdev), the minimum fault intensity f_i(min) and maximum fault intensity, f_i(max) are observed. The variations of the actuator physical variables according to the parameters in Table 1 are studied and the synthetic data is thus generated. The representation of the generated dataset showing the effect of physical variables during normal and faulty modes are shown in Fig. 2.

Fig. 2

Representation of the generated dataset showing the effect of physical variables during (a) normal mode (b) f₁ - clogging fault (c) f₂ - sedimentation fault (d) f₃ - erosion fault.

In Fig. 2a, it can be observed that, the actuator is working without any fault condition induced. Here, the rod displacement (D) tracks the input control variable (SP) between 0.2 and 0.9 i.e., 20% and 90%. Also, the inlet pressure (P_i) is maintained at 0.65, the outlet pressure (P_o) varies around 0.9 and the temperature of water (T_m) is maintained at 0.22. Here, the normalized values are used. When the valve clogging fault f₁, is induced abruptly at the time 900s with a medium intensity of 0.5 as in Fig. 2b, the rod displacement (D) is inhibited leading to disturbed medium flow, F_r.

In Fig. 2c, the sedimentation fault condition f₂ is gradually induced at time 900s and reaches the peak intensity of 1 at 4500s. During this fault, it can be observed that, there is a decrease in the flow of the medium (F_r) due to decreased valve plug travel.

The change in physical variables for the erosion fault condition f₃ is observed in Fig. 2d. The fault is induced at the time 900s and reaches maximum intensity at 1500s itself. Hence, the erosion fault is a short and gradual fault. During the fault period, the flow of the medium is increased due to the increase in the valve plug movement due to erosion.

Thus, the dataset for normal condition and faulty condition was generated using the DAMADICS library. Also, an extensive exploratory data analysis was performed on this synthetic dataset and presented in [22]. The metadata of the generated dataset after performing exploratory data analysis is presented in Table 2.

Table 2

The metadata of the synthetic dataset showing the non-parametric data distribution and imbalance in class distribution

	Description
Attributes	9
Attribute information	Set point (SP), Inlet pressure (P_i), Outlet pressure (P_o), Flow (F_r), Temperature (T_m), rod displacement (D), fault intensity, fault type, fault label.
Sample size	266100
Type	Structured, Multivariate, Non-parametric, Imbalanced
Binary class targets	Class 0 – Normal, Class 1 - Faulty
Multi class targets	Class 0 – Normal
	Class 1 – clogging fault
	Class 2 – sedimentation
	Class 3 – erosion fault
Data distribution	Normal (P_i, P_o, T_m), Bimodal (F_r)
	Multimodal (D)
Class distribution (in %)	Binary	Multiclass
	Class 0 – 69.8	Class 0 – 69.8
	Class 1 – 30.2	Class 1 – 28.8
		Class 2 – 1.0
		Class 3 – 0.4

From Table 2, it can be inferred that, the data distribution throughout the dataset is non-normal and there is imbalance in the distribution of classes. In the case study, to maintain the uniformity among classifier models, class imbalance is treated using a sampling technique namely, SMOTE: Synthetic Minority Over-sampling Technique [20] and no special algorithm or optimization is performed on each algorithm to increase their performance. However, a baseline hyper parameter tuning is practiced to achieve optimal performance from all classifier models.

2.2 Machine learning classifier models

Machine learning based classifier models automate pattern recognition and anomalies in data with minimal time expenditure and maximum accuracy. Recent times, machine learning based models have gained interest for detection and prediction [17] tasks due to their high accuracy rates. The advantages of selecting machine learning models over conventional models are highlighted by the authors [23] where the work statistically supports machine learning algorithms over conventional parametric algorithms for a different case study. In this section, many classifier models are built and the parameters and hyperparameters tuned to obtain optimal performance for each algorithm chosen are presented. Considering the popularity of the existing algorithms compared in this work, the pseudocode of the machine learning algorithms is provided in Appendix and not included in the article body. However, a short description of each algorithm is provided for better understanding and continuity.

2.2.1 Naïve Bayes (NB) classifier

Naïve Bayes Classifier is a vanilla model of theBayes theorem as given in Equation (2). $P (Y | {X)}_{posteriori} = \frac{P (X | {Y)}_{likelihood} P {(Y)}_{priori}}{P {(X)}_{marginal}}$ (2)

The parameters of NB algorithm are priori probability P (Y) _priori and the likelihood probability P(X| Y) _likelihood. It can be noted that, 2 values of priori during binary classification and 4 values of priori during multiclass classification along with the likelihood score is returned by the classifier as given in Table 4. There are no hyperparameters as such for tuning a naïve bayes classifier. Hence, it is simple and swift in performing classification. However, due to its generative property, only the conditional density between X and Y is obtained during classification. To get the overall density function, discriminative methods namely: Perceptron, Logistic Regression and Linear Discriminant Analysis are explored.

2.2.2 Perceptron (PER) classifier

Here, a simple single layer perceptron model as given in Equation (3) is used for classification. $F (x) = sgn (ω^{T} x + b)$ (3) where, x-input vector, ω-weights and b-bias value.

It can be seen that, ω is the only parameter which has to be learnt by the algorithm. Hence, the algorithm returns a weight and bias vector of [c x (m + 1)] as in Table 4. The hyper parameter α learning rate as given in Table 3, is obtained by performing search from a grid of values ranging from 0.001 to 1 and the epochs are fixed for minimum validation error. The accuracy of the Perceptron classifier depends on the linearly separable feature space due to the linear nature of its threshold function. Logistic regression classifier is chosen to overcome this limitation.

Table 3

Parameters and tuned hyper parameters of parametric algorithms

Classifier	Parameters	Hyperparameters
		Binary	Multiclass
NB	Priori, likelihood	–	–
LR	Regression	C = 10,	C = 100,
	coefficients	α= 0.01,	α= 0.1,
		penalty = ‘l1’	penalty = ‘l1’,
PER	Mean and priori	Solver = svd	Solver = svd
LDA	Weights	Epoch = 11,	Epoch = 63,
	and bias	α= 0.01	α= 0.1

Table 4

Parameters learnt by the parametric algorithms

Parameters	Binary	Multiclass
NB	Prior	[0.5, 0.5]	[0.25, 0.25, 0.25, 0.25]
	Likelihood	0.5902967510994135	3.5587471855482984
LR	Regression coefficients	[[4.19821128 1.85513373 27.4598895 -16.08841382 -5.20651594 -2.37463736 19.13364947][4.00132778 0.98665431 14.7119001 -12.8861922 -7.8396340 -2.41235538 22.65521325]]	[0.07971913 6.40257073 -15.18459835 -4.5293485 -5.87807 -1.81186479 -2.9897327][0.11745601 -3.47944028 4.63115913 6.70116352 8.17672088 -9.82810225 10.3704288][-0.54481661 -6.96384333 -13.29347098 -19.02469447 -0.48526115 17.49711197 -18.42812165][0.34764147 4.04071287 23.8469102 16.85287945 -1.81338972 -5.85714493 11.04742555]
	Bias	[-9.53680183 -11.898332]	[12.21903908 -4.28440497 18.442441 -26.37707511]
LDA	Priori	[0.5, 0.5]	[0.25, 0.25, 0.25, 0.25]
	Mean	[0.50008388 0.87490268 0.64954681 0.94904792 0.0658367 0.21514114 0.00357995][0.50190673 0.87470662 0.64997994 0.59892533 0.65083648 0.2165908 0.61089543]	[8.74849915e-01 6.50004719e-01 9.34041434e-01 8.30450321e-02 2.15107391e-01 7.65014535e-05 3.06005814e-04][8.74714575e-01 6.49806078e-01 9.00738822e-01 1.40026435e-01 2.47713613e-01 1.95168708e-02 1.09397079e-01][8.74179644e-01 6.05245707e-01 9.41445335e-01 7.93524247e-02 2.15000105e-01 9.62163281e-02 0.00000000e+00][8.75081029e-01 6.49973563e-01 9.33172181e-01 1.11352048e-01 2.15013491e-01 1.00740134e-01 1.49168834e-01]
PER	Weights	[[4.56328035 0.97414955 29.36160581 -13.13090337 -3.4789897 -2.94299343 16.59906754][3.71189092 0.16332538 16.2382878 -11.1343599 -10.4646888 -7.52752881 21.09865468]]	[-1.09646594 9.33999666 -4.91471417 -1.15596973 -7.47443209 -0.54946362 -7.][0.5542466 -1.43882968 -2.21412346 -2.35709235 12.39337126 -6.70557756 4.][-3.22085854 -17.37636555 -16.75176882 -37.26222258 6.21170808 31.47194922 -33.][-0.75152201 13.81324786 23.99831504 12.99522072 -7.06958978 2.59909727 3.]
	Bias	[-13.0 -7.11]	[-2. 0. 28. -34.]

2.2.3 Logistic Regression (LR) classifier

Logistic regression to predict target y is given in Equation (4). $ln (\frac{y}{1 - y}) = β_{0} + β_{1} x_{1} + \dots + β_{m} x_{m}$ (4) where β₀ represents the bias value and β₁, β₂ … β_m are regression coefficients of the corresponding input values of x for ‘m’ features.

For ‘m’ features, (m + 1) regression coefficients are learnt by the algorithm. Therefore, in this case study, 8 regression coefficients are learnt by the model as given in Table 4. The hyper parameter ‘C’, a positive integer used for regularization is tuned to its optimal value (C = 10 for binary, C = 100 for multiclass) with ‘l1’ regularization along with the learning rate α to achieve improved performance. The LR classifier is very sensitive to outliers and has reduced speed. Hence, another discriminative algorithm, LDA is considered for the case study.

2.2.4 Linear Discriminant Analysis (LDA) Classifier

Generalised discriminant analysis is used to find the linear combination of features that can separate two or more classes by finding a linear projection that maximizes the separation between classes while minimizing within-class scatter. The terms Linear Discriminant Analysis (LDA) and Fischer Discriminant Analysis (FDA) are often used interchangeably to describe this technique. Fisher’s Discriminant Analysis (FDA) [26] is Linear Discriminant Analysis (LDA) when there are only two classes. LDA is the direct extension of FDA to two or more classes. In practice, FDA is not a standalone classifier, since it is a dimensionality reduction technique. The transformed features obtained from FDA is fed into classifiers to make class predictions. Hence, in this work LDA is proposed for classification tasks.

The function F(x) followed by the Linear Discriminant classifier for classification of c classes is given in Equation (5). $F (x) = arg \max_{c} δ_{c} (x)$ (5) where, δ_c (x) is the discriminant function. Here, the mean and priori are fixed parameters of the algorithm that are to be learned during the training phase. The algorithm learns an array of c x m parameters as given in Table 4. The ‘solver’ hyperparameter was chosen as ‘singular value decomposition’ based on random search methods. In LDA, no penalty was imposed on the dataset.

The parameters and hyperparameters tuned for the case study are listed in Table 3.

The parameters projected in Table 4 for each machine learning model justifies the statement that ‘Parametric algorithms are models with unmodifiable and finite parameters.’

Due to the constrained working mode of the parametric algorithm, more flexible and relaxed models (non- parametric algorithms) with no/less assumptions are considered for classification, as discussed in the ensuing sections.

2.2.5 Multi-Layer Perceptron (MLP) classifier

Neural network based MLPs are powerful and flexible classifiers, yet, they are generally considered parametric. However, on introducing non-parametric elements into the MLP model, non-parametric MLPs are obtained. Different approaches to achieve is, by using regularisation techniques such as dropout, hybrid dropout [27] and weight decay [4].

In this work, the weight decay approach is adopted. Similar to perceptron, the weights and bias values are initialised randomly. Here, a penalty term ‘l2’ is added to the loss function during training. This regularization technique can be seen as a form of non-parametric adjustment, as it promotes simpler weight configurations and reduces the model’s reliance on specific parameter values. Here, the hyperparameter namely, the number of hidden layers is fixed to be 2. Also, the optimal epochs are found to be 7 and 28 for binary and multiclass tasks respectively. The activation function of the MLP is a radial basis function and the hyperparameters γ and C are found using the grid search method. Here, γ= 1 for both the cases and C = 10 and 100 for binary and multiclass, respectively. The advantages of using MLP as a non-parametric classifier are its flexibility and ability to capture non-linear relationships effectively. However, it is computationally complex and has many hyperparameters for tuning.

2.2.6 k Nearest Neighbor (kNN) classifier

The kNN algorithm works by finding the ‘k’ nearest neighbors by computing the distance between each data point using local or global distance metrics and declaring the labels based on the majority votes of the neighbors. Here, the distance metric is the parameter that the algorithm learns. As mentioned in section 2, the dataset has a total of 2,66,100 rows hence, the size of the parameter varies subject to any variation on the size of the samples drawn in the testing phase.

The hyper parameter is the ‘k’ factor. Using grid search method, for this case study, k is tuned to the value of 3 for binary classification and 5 for multiclass classification task to obtain optimal model performance. Though kNN is flexible and powerful for classification tasks, the limitation is in speed and memory space consumption. This can be overcome by SVM classifier.

2.2.7 Support Vector Machines (SVM) Classifier

In SVM, an initial hyperplane is declared and the objective is to maximize the minimal distance of the hyperplane. Here, the ‘Radial Basis Function (RBF) kernel’ [6] is chosen since the classes in the dataset are not linearly separable. The distance ‘d’ is calculated for every data point in order to minimize the distance of hyperplane.

Here, the hyper parameter was tuned manually using grid search methods. The value of C is varied from 0.1 to 1000 and γ between 0.0001 to 1. The optimal values were chosen based on the validation error and model accuracy. The speed of SVM becomes slow when hyperplanes for all samples have to be decided when the dataset is large. Also, the multicollinearity between features is dealt easily with decision tree than SVM.

2.2.8 Decision Tree (DT) Classifier

In this classifier, Classification and Regression Tree (CART) model is assigned to build the decision tress. Gini index is used to measure the impurity content of tree. The tree is split until all equal classes are predicted and the Gini index becomes 0.

Decision trees are prone to overfit. The work supporting the overfitting elimination and designing the best fit decision tree model for the DAMADICS dataset is performed systematically by the same authors in their previous work [21] on the synthetic dataset. Here, the resampling technique was applied to determine the best fit model. In this work, the maximum depth of the tree is tuned and the binary decision tree depth is chosen as 5 and the multiclass decision tree depth is chosen as 32. The problem of overfitting can be eliminated by using RF classifier. Also, the RF is used to avoid the problem of forcing that occurs in decision trees.

2.2.9 Random Forest (RF) Classifier

In this algorithm, random samples are drawn and the best split criteria are met. This splitting varies with the change in dataset as in DT classifier. The procedure is repeated on every tree in the forest and the aggregated value of all tress gives the predicted outcome.

In the present case study, the number of trees formed is 10 in binary classification and 73 in multiclass problem. The parameters and hyper parameters used in the non-parametric algorithms are listed in Table 5.

Table 5
Parameters and tuned hyper parameters of non-parametric algorithms

Algorithm Parameter Hyper parameter

Binary Multiclass

MLP Weights &Bias Hidden layers = 2,epochs = 7,α= 0.01,activation = RBF Hidden layers = 2,epochs = 28,α= 0.01,activation = RBF

kNN Euclidean distance k = 3 k = 5

SVM Pair wise distance Kernel = RBF,C = 10,γ= 1 Kernel = RBF,C = 100,γ= 1

DT Split nodes N = 5 N = 32

RF Split nodes Trees = 10 Trees = 73

Algorithm	Parameter	Hyper parameter
MLP	Weights &Bias	Hidden layers = 2,epochs = 7,α= 0.01,activation = RBF	Hidden layers = 2,epochs = 28,α= 0.01,activation = RBF
kNN	Euclidean distance	k = 3	k = 5
SVM	Pair wise distance	Kernel = RBF,C = 10,γ= 1	Kernel = RBF,C = 100,γ= 1
DT	Split nodes	N = 5	N = 32
RF	Split nodes	Trees = 10	Trees = 73

The values of parameters of the non-parametric algorithms cannot be explicitly given as in Table 4 because, the parameters of non-parametric algorithms grow depending on the size of the data and hence, they are out of bound to tabulate.

3 Results and discussions

The performance of parametric and non-parametric algorithms is evaluated using the performance indices namely: Model fitness, accuracy, interpretability, robustness and complexity. The trade-offs in the performance indices for Parametric and Non-Parametric algorithms are shown in Fig. 3.

Fig. 3

Trade off curve between parametric and non-parametric model performance characteristics.

From the flexibility point of view, it can be inferred from Fig. 3 that, flexibility is directly proportional to accuracy, data complexity, overfitting, and inversely proportional to speed and interpretability for both the models due to the influence of data and parameters.

3.1 Flexibility

Statistically, flexibility implies model fitness. In machine learning, bias-variance decomposition analysis is performed to measure flexibility. The bias-variance decomposition for a training dataset with targets y_i associated with inputs x_i having a function $\hat{f} (x)$ to approximate y = f(x) is given as in Equation (6). $\begin{matrix} E [{(y - \hat{f} (x))}^{2}] = {[E [\hat{f} (x)] - f (x)]}^{2} \\ + E [{(E [\hat{f} (x)] - \hat{f} (x))}^{2}] \\ + σ^{2} \end{matrix}$ (6) where,

Bias = ${[E [\hat{f} (x)] - f (x)]}^{2}$ ;

Variance = $E [{(E [\hat{f} (x)] - \hat{f} (x))}^{2}]$ ;

Mean Square Error = $E [{(y - \hat{f} (x))}^{2}]$ and σ denotes irreducible noise. Increase in bias or variance tend to produce underfit and overfit model, respectively. This is because bias is a systematic error in the training model and variance is a growing error in the test model.

The trade-offs between bias and variance should be optimal [25] to design an accurate classifier. Minimum bias and maximum variance make a good prediction model. The results of the bias-variance decomposition method are presented in Table 6. Here, the kNN exhibits a minimum bias for binary and DT for multiclass classification. Variance is high for RF binary and SVM multiclass. Such comparison is rudimentary and may mislead. The bias and variance of the same model is compared to analyze the flexibility of the model.

Table 6

Results obtained through bias-variance decomposition analysis showing flexibility of model

		NB	LR	LDA	PER	MLP	KNN	SVM	DT	RF
Binary class	MSE	51.200	13.929	8.002	69.708	10.712	5.060	12.762	0.019	0.011
	Bias	6.871	2.531	1.932	8.126	4.114	1.270	2.661	0.005	0.009
	Variance	3.990	7.524	4.270	3.677	3.720	3.448	5.682	0.019	0.011
	Average Bias	5.103	1.615	2.178	10.001	7.763	1.085	2.026	0.002	0.009
	Avg.Variance	2.817	7.176	6.569	3.598	4.012	2.663	3.693	0.019	0.006
Multi class	MSE	69.183	77.649	20.827	87.438	81.564	16.817	9.188	2.655	2.242
	Bias	7.921	8.448	4.130	9.007	8.067	3.362	2.085	0.695	0.773
	Variance	6.441	6.281	3.771	6.312	6.091	5.514	4.841	2.172	1.645
	Average Bias	11.078	9.763	7.409	13.151	12.012	3.885	3.099	2.318	1.014
	Avg.Variance	6.300	8.655	5.330	10.613	9.831	4.830	5.571	2.995	2.030

From Table 6 it is observed that, the bias of NB and PER models are greater than their respective variance values. Hence, they act as underfit models for the case study. All other models exhibit some flexibility in their behavior having less bias value. However, the deviation between bias and variance in the DT and RF model is nominal and hence, they can be selected a best fit model for the case study. The Mean Square Error (MSE) is calculated using Equation (6) with σ= 0. The average bias and variance were obtained on 20% of test data for 5 model.

3.2 Accuracy

Accuracy is the simple metric to calculate the correctness of a model. It is a measure of correct predictions over the entire sample set as given in Equation (7). $accuracy (y_{true_i}, y_{pred_i}) = \frac{\sum_{i = 1}^{m} (y_{pred_i} = y_{true_i})}{m}$ (7) where,y_{true_i} is the i^th true value in the m number samples, corresponding to the i^th predict valuey_{pred_i}.

Misclassification error can be calculated from accuracy metric as given in Equation (8). $\begin{matrix} misclassification_error \\ = 1 - accuracy (y_{true_i}, y_{pred_i}) \end{matrix}$ (8) where, y_{true_i} is the i^th true value in the m number samples, corresponding to the i^th predict value y_{pred_i}.

For imbalanced dataset, the balanced accuracy metric as given in Equation (9) is to be considered since it is a weighted average score. $\begin{matrix} Balanced_accuracy (y_{true_i}, y_{pred_i}) \\ = \frac{\sum_{i = 1}^{m} (y_{pred_i} = y_{true_i}) w_{i}}{\sum w_{i}} \end{matrix}$ (9) where w_i is the updated weight of the samples. The accuracy and error metrics are presented in Table 7.

Table 7

Accuracy and error metrics of P and NP models

Threshold based		P models					NP models
Metrics		NB	LR	LDA	PER	MLP	KNN	SVM	DT	RF
Binary	Accuracy	0.981	0.944	0.986	0.954	0.998	0.999	0.999	0.999	1
	Balanced accuracy	0.801	0.969	0.990	0.507	0.991	0.999	0.999	0.999	1
	MCE	0.019	0.056	0.014	0.046	0.002	0.001	0.001	0.001	0
Multi	Accuracy	0.059	0.009	0.400	0.697	0.709	0.998	0.996	0.991	0.831
	Balanced accuracy	0.289	0.25	0.362	0.25	0.532	0.998	0.991	0.911	0.495
	MCE	0.941	0.991	0.6	0.303	0.291	0.002	0.004	0.009	0.169

From Table 7, it can be observed that, the accuracy of the parametric algorithms for binary classification is satisfactory. However, the accuracy of the multiclass parametric classifiers is poor. High misclassification errors are seen in parametric models. The performance of non-parametric algorithms for both binary and multiclass classification is satisfactory. In this case study, the balanced score is not very significant since, the imbalance in dataset is treated using the minority oversampling method as mentioned in Section 2.

The Area Under – Receiver Operating Characteristics (AU-ROC) value epitomizes the model efficiency. Also, it is used to confirm the prediction ability of the model. The AU-ROC curve is plotted between the false positive rate and true positive rate as in Fig. 4. The ROC score can range from 0 to 1.

Fig. 4

ROC curve showing the performance of parametric and non-parametric classifiers for binary classification problem.

In Fig. 4, it can be seen that the non-parametric algorithms provide a score of 1. This makes them skillful classifiers than the parametric algorithms. The ROC score of parametric algorithms is lesser than 1, however not poor. Hence, in this case study, both parametric and non-parametric algorithms tend to show decent fitness for binary problem. The performance of parametric classifiers for multiclass classification problem is shown in Fig. 5.

Fig. 5

ROC curves of parametric classifiers showing class-wise performance for multiclass classification problem (a) Naive Bayes classifier (b) Logistic regression classifier (c) Linear discriminant analysis classifier (d) Perceptron classifier.

From Fig. 5, it can be seen that the ROC score of parametric algorithms is in the range of 0.5 and below. The Parametric algorithms provide poor prediction ability for all classes ranging from class 0 to class 4. Figure 6 depicts the performance of the non-parametric algorithm for multiclass problem.

Fig. 6

ROC curves of non-parametric classifiers showing class-wise performance for multiclass classification problem (a)Multi-Layer Perceptron classifier (b) k Nearest neighbor classifier (c) Support vector classifier (d) Decision tree classifier (e) Random Forest classifier.

From Fig. 6, it can be seen that the ROC curve provides a value nearest to 1 for all classes. On observing, it is noted that kNN and decision tree classifiers are more skilled than support vector machines and random forest classifiers since, the number of features is less than the number of training data and also, due to the automatic feature interaction nature of the decision trees. Nevertheless, other algorithms have also showed reasonable performances. Thus, classification is well established by non-parametric algorithms than the parametric algorithms.

3.3 Interpretability

According to [12], interpretability is defined as “the ability to explain the model in an understandable form”. Here, Triptych analysis [32] is employed, where the absolute score of interpretability (I) is given by Equation (10). $I = \sum (P_{r}, S_{t}, S_{i})$ (10) where P_r denotes predictivity, which is a measure of accuracy independent of the target classes. S_t denotes the stability score and S_i is the simplicity score.

All the metrics range from 0-1. The model with increased P_r, S_t and S_i score is declared to be an interpretable model.

On applying the Triptych analysis, the absolute value of interpretability is obtained as a summation of predictivity, stability and simplicity scores of each algorithm and it is listed in Table 8. Here, the predictivity value is the accuracy score of the model and the F1 score is the stability score.

Table 8

Results from the Triptych analysis showing absolute values of interpretability scores of models

		NB	LR	LDA	Per	MLP	KNN	SVM	DT	RF
Binary	P_r	0.986	0.992	0.996	0.965	1	1	1	1	1
	S_t	0.867	0.791	0.928	0.495	0.999	0.998	0.999	0.997	1
	S_i	0.8	0.9	0.8	1	0.6	0.5	0.4	0.3	0.3
	I	2.653	2.683	2.724	2.46	2.599	2.498	2.399	2.297	2.3
Multi class	P_r	0.051	0.009	0.441	0.699	0.704	0.998	0.996	0.991	0.847
	S_t	0.063	0.004	0.228	0.205	0.621	0.979	0.894	0.894	0.571
	S_i	0.8	0.9	0.8	1	0.6	0.5	0.4	0.3	0.3
	I	0.914	0.913	1.469	1.904	1.925	2.477	2.29	2.185	1.718

In Table 8, as discussed earlier, the predictivity which is given by the accuracy metric, is nominal for both algorithms when performing binary classification and reduced for parametric algorithms when performing multiclass classification. Also, the stability of non-parametric models is consistently satisfactory for both the cases. The simplicity score is naively chosen from a scale of 0-1 depending on the understanding ability of the humans about the algorithm structure. However, the absolute value of ‘I’ is calculated using the Equation (10) and the interpretability scores are plotted in Fig. 7.

Fig. 7

Interpretability score Vs models.

In Fig. 7, there is a trade off in the interpretability score between the parametric and non-parametric algorithms. It can be seen that, parametric algorithm showing good interpretability score for binary classification, failed otherwise owing to the increase in data complexity.

3.3.1 Robustness

Robustness explains the performance of an algorithm for a given task in the presence of noise with minimum or no deviation from accuracy of the model with zero noise. In machine learning models, noise can be induced in the generated data or the label [5]. Let ‘ɛ’ be the label noise induced in the dataset D, having X inputs and Y targets such that (X, Y, ɛ)< (X, Y), then noise robustness $R$ is measured using accuracy as in Equation (11) $R \to {(A (X, Y, ɛ) |}_{ɛ = 0} \approx {A (X, Y, ɛ) |}_{ɛ = η %})$ (11) where, η is the percentage of label noise induced.

The robustness analysis of the machine learning classification model is carried out by inducing 5-20% label noise. About η% of the negative labels were replaced with positive labels and the classification is performed. It is found through the accuracy metric that the built models were susceptible to changes in the stability of the models which is measured quantitatively using accuracy metric. The robustness of machine learning models against 5-20% of induced label noise is listed in Table 9. Here, the intermodel and intramodel stability analysis of parametric and non-parametric models is analyzed using accuracy as a metric.

Table 9

The accuracy of machine learning models against induced label noise of 5-20% showing robustness of models

		η= 0%	η= 5%	η= 10%	η= 15%	η= 20%
NB	Binary	0.981	0.704	0.572	0.510	0.433
	Multiclass	0.059	0.059	0.017	0.001	0.001
LR	Binary	0.944	0.816	0.691	0.625	0.525
	Multiclass	0.009	0.003	0.001	0.001	0.001
LDA	Binary	0.986	0.907	0.836	0.737	0.610
	Multiclass	0.400	0.391	0.310	0.197	0.021
Per	Binary	0.954	0.632	0.557	0.411	0.313
	Multiclass	0.697	0.583	0.347	0.284	0.173
MLP	Binary	0.998	0.769	0.663	0.532	0.401
	Multiclass	0.709	0.511	0.385	0.274	0.092
KNN	Binary	0.999	0.921	0.879	0.846	0.801
	Multiclass	0.998	0.931	0.856	0.799	0.704
SVM	Binary	0.999	0.897	0.840	0.806	0.794
	Multiclass	0.996	0.882	0.683	0.617	0.500
DT	Binary	0.999	0.983	0.976	0.974	0.973
	Multiclass	0.991	0.969	0.950	0.937	0.912
RF	Binary	1	0.999	0.999	0.998	0.998
	Multiclass	0.831	0.817	0.784	0.753	0.736

From the results, it can be inferred that, random forest algorithm showed noticeable robustness when compared to other models. It must be understood that the noise elimination of the machine learning models can be learnt/reduced by tuning the hyperparameters of the model. However, for the purpose of explaining the trade-offs between the algorithms, the optimal values of the hyperparameters achieved during model building process is maintained throughout the robustness analysis.

3.3.2 Complexity

Complexity on the whole denotes the data, time and speed complexity of the model. The analysis applied is called the asymptotic analysis. Time complexity is basically the rate at which the model’s run time increases with growth in input data size. Space complexity is the rate at which the memory consumed by the model increases with growth in input data size. Here, for the asymptotic analysis, the Big O notation [30] is used to define the efficiency of the models.

The time and space complexity values of parametric and non-parametric algorithms are tabulated in Table 10, with reference to their general Big O notation for the corresponding model.

Table 10
Time and space complexity values of each model calculated w.r.t Big O notation

Train time complexity (T_train) Train space complexity (S_train) Run time complexity (T_run) Run space complexity (S_run)

Binary Multiclass Binary Multiclass Binary Multiclass Binary Multiclass

NB 4,789,800 9,579,600 18 35 18 35 18 35

LR 2,394,900 2,394,900 2,394,900 2,394,900 9 9 9 9

LDA 21,554,100 21,554,100 21,554,100 21,554,100 2,394,900 2,394,900 2,394,900 2,394,900

Per 2,66,100 2,66,100 2,66,100 2,66,100 18 35 18 35

MLP 29,80,320 23,842,560 4,25,760 8,51,520 7,45,080 5,960,640 1,06,440 2,12,880

kNN 7,184,700 11,974,500 2,394,900 2,394,900 7,184,700 11,974,500 2,394,900 2,394,900

SVM 9,579,600 9,579,600 2,394,900 2,394,900 9 21 9 21

DT 7,218,022 56,300,573 5 39 5.425 5.425 5 39

RF 129,924,399 948,448,117 81 2932 54.259 396.028 81 2932

	Train time complexity (T_train)	Train space complexity (S_train)	Run time complexity (T_run)	Run space complexity (S_run)
NB	4,789,800	9,579,600	18	35	18	35	18	35
LR	2,394,900	2,394,900	2,394,900	2,394,900	9	9	9	9
LDA	21,554,100	21,554,100	21,554,100	21,554,100	2,394,900	2,394,900	2,394,900	2,394,900
Per	2,66,100	2,66,100	2,66,100	2,66,100	18	35	18	35
MLP	29,80,320	23,842,560	4,25,760	8,51,520	7,45,080	5,960,640	1,06,440	2,12,880
kNN	7,184,700	11,974,500	2,394,900	2,394,900	7,184,700	11,974,500	2,394,900	2,394,900
SVM	9,579,600	9,579,600	2,394,900	2,394,900	9	21	9	21
DT	7,218,022	56,300,573	5	39	5.425	5.425	5	39
RF	129,924,399	948,448,117	81	2932	54.259	396.028	81	2932

From Table 10, it can be seen that, the primitive models of parametric type algorithms namely: perceptron and logistic regression classifiers contribute to lesser time and space complexity. However, this reduction costs high in model accuracy. For the data complexity, the evolution of complexity in train time and train space for the kNN classifier is plotted in Fig. 8.

Fig. 8

Training time and training space complexity of kNN for binary classification.

In Fig. 8, the increase in time for execution and the memory consumed by the algorithm for varying sizes of the training dataset is observed. It can be noted that, for small input size, i.e., when only 20% of the training dataset is given, (53,220 instances = n), k = 3 for binary, and number of features m = 9, the training time complexity of kNN is calculated as O(knm) and space complexity is calculated as O(nm). A linear increase in time and memory usage is observed for kNN. The parameters and hyperparameters during the binary and multiclass classification remain the same throughout.

For the purpose of performance comparison, the models were run in different platform. The execution times of each algorithm in different platforms like CPU and GPU are shown in Fig. 9.

Fig. 9

Performance comparison of execution time in CPU and GPU platforms.

The CPU performance was evaluated when the models were run on Python-Jupyter notebook with windows 64bit OS + Processor Intel core i5 ∼ 2.4 GHz. For the GPU, the execution time of the models in Google Colab was used. It is found that, similar to the time and space complexity discussed above, parametric algorithms performed in less execution time. However, the performance of models in GPU were found to be faster than that in CPU for the dataset.

4 Validation

DAMADICS, a real time industrial dataset [11] of the pneumatic actuator in Polish sugar refinery, utilised for water level control of steam boiler in the evaporation stage (stage III) of the plant is used for the validation against the synthetic model. The dataset has about 21,59,974 data points with 7 attributes. The working range of the variables remain the same as given in Section 2. However, the faults recorded includes: f₁₆ - positioner supply pressure fault, f₁₈ - bypass valve fault and f₁₉ - flowrate sensor fault [16]. The faults recorded in the real time actuator dataset are shown in Fig. 10. The normal mode of the actuator is shown in Fig. 10a. In Fig. 10b, the positioner pressure drop fault (f₁₆) occurs in the duration: 57475s (pressure drops) – 57530s (pressure ok) and also from 57675s (pressure drops) – 57800s (pressure ok). During this transition, it is observed that, the rod displacement (X) drops to zero and the flow (F) is disturbed. The bypass valve partially or fully open fault (f₁₈) occurs at 57340s (valve opens) – 57890 (valve closes) as shown in Fig. 10c. When the bypass valve (V₃ in Fig. 1) is partially opened, it is observed that, the medium flow (F) drops to zero level. In Fig. 10d, at 58150s (sensor fault ON) – 58325s (sensor fault OFF), the flow rate sensor fails to read the flow data and shows a zero reading though the other variables constituting the process works well. The specifications of the faults in the industrial dataset are listed in Table 11.

Table 11
List of Parameters of faults in the industrial data

Notation & Fault mode fs_(min,max) Type Direction

f₁₆ - positioner supply pressure <0,1> Gradual +

f₁₈ - bypass valve opening <0,1> Abrupt +

f₁₉ - flowrate sensor < -1,1> Gradual +/-

Notation & Fault mode	fs_(min,max)	Type	Direction
f₁₆ - positioner supply pressure	<0,1>	Gradual	+
f₁₈ - bypass valve opening	<0,1>	Abrupt	+
f₁₉ - flowrate sensor	< -1,1>	Gradual	+/-

Fig. 10

Representation of the industrial DAMADICS dataset showing the effect of physical variables during (a) normal mode (b) f₁₆ - positioner supply pressure fault (c) f₁₈ - bypass valve fault (d) f₁₉ - flowrate sensor fault.

The parameters and hyperparameters of the parametric and non-parametric machine learning models are listed in Table 12.

Table 12

Parameters and hyperparameters of the Machine Learning models used

Algorithm	Binary	Multiclass
NB	–	–
LR	C = 10, p = ‘l1’	C = 100, p = ‘l1’
	α= 0.001	α= 0.01
LDA	Solver = ‘svd’	Solver = ‘svd’
Per	Epoch = 25,	Epoch = 89,
	α= 0.001	α= 0.01
MLP	Hidden layers = 2,epochs = 7,α= 0.01,activation = RBF	Hidden layers = 2,epochs = 28,α= 0.01,activation = RBF
kNN	k = 3	k = 5
SVM	C = 10, γ= 1	C = 100, γ= 1
DT	N = 5	N = 32
RF	Trees = 10	Trees = 73

Initially, the industrial dataset is pre-processed. The values are normalized and outliers are removed using sigma method. On the non-parametric dataset, both binary classification (normal, faulty) and multiclass classification are performed using parametric and non-parametric algorithms. The performance of the algorithms on the dataset is studied and tabulated for different metrics as shown in Table 13.

Table 13

Evaluation metrics showing the trade-offs between the performance of parametric and non-parametric classifiers in the benchmark industrial dataset

	NB	LR	LDA	PER	MLP	kNN	SVM	DT	RF
MSE	33.196,	105.542,	28.895,	26.539,	17.186,	0.060, *	0.121,	0.108,	0.202,
	65.676	249.520	108.505	56.529	49.984	0.324	0.294	0.095*	0.114
Bias	3.989,	9.541,	5.144,	4.071,	2.874,	0.018, *	0.092,	0.096,	0.091,
	7.370	15.035	10.239	6.559	3.799	0.165	0.184	0.077*	0.084
Variance	17.284,	14.422,	2.435,	9.966	3.652,	0.060	0.113,	0.099,	0.194,
	11.360	23.469	3.668	13.509	10.534	0.297	0.261	0.090*	0.107
Average bias	19.070	12.577	7.118	17.115	5.082,	0.099,	0.098,	0.003, *	0.008,
	31.399	26.010	11.620	23.333	11.822	0.131	0.172	0.071	0.099
Average variance	11.047,	10.072,	4.320,	11.611,	9.364,	0.171,	0.156,	0.142, *	0.973, *
	8.664	19.415	9.539	8.060	11.703	0.187	0.104	0.126	0.198
Accuracy	0.746,	0.800,	0.912,	0.730,	0.830,	0.817,	0.861,	0.970,	0.943,
	0.570	0.611	0.724	0.543	0.592	0.806	0.783	0.927*	0.899
MCE	0.254,	0.200,	0.088,	0.270,	0.170,	0.183,	0.139,	0.030,	0.057,
	0.430	0.389	0.276	0.457	0.408	0.194	0.217	0.073*	0.101
Balanced accuracy	0.732,	0.780,	0.907,	0.714,	0.801,	0.827,	0.880,	0.986,	0.931,
	0.583	0.586	0.706	0.521	0.491	0.799	0.809	0.922*	0.800
Predictivity	0.753,	0.801,	0.871,	0.705,	0.815,	0.817,	0.868,	0.970,	0.936,
	0.571	0.600	0.704	0.534	0.581	0.809	0.796	0.931*	0.901
Stability	0.593,	0.681,	0.789,	0.517,	0.814,	0.810,	0.808,	0.962,	0.913,
	0.171	0.202	0.307	0.102	0.592	0.763	0.776	0.848*	0.896*
Simplicity	0.800	0.900 *	0.800	1.00 *	0.6	0.500	0.400	0.300	0.300
Interpretability	2.146,	2.382,	2.460, *	2.222,	2.197,	2.127,	2.076,	2.232, *	2.149,
	1.542	1.702	1.811	1.636	1.773	2.072	1.972	2.079	2.097*
Robustness	0.703,	0.769,	0.849,	0.690,	0.801,	0.793,	0.860,	0.966,	0.934,
η= 5%	0.499	0.552	0.627	0.411	0.328	0.748	0.782	0.927	0.896*
η= 10%	0.637,	0.692,	0.813,	0.560,	0.662,	0.788,	0.843,	0.962,	0.938,
	0.305	0.404	0.610	0.363	0.265	0.731	0.770	0.921	0.891*
η= 15%	0.581,	0.603,	0.794,	0.430,	0.355,	0.742,	0.834,	0.959,	0.926,
	0.267	0.353	0.580	0.288	0.194	0.711	0.761	0.917	0.887*
η= 20%	0.502,	0.511,	0.770,	0.380,	0.245,	0.718,	0.800,	0.947,	0.917,
	0.262	0.310	0.541	0.234	0.163	0.690	0.734	0.909	0.894*
Speed	30,239,600,	15,119,825	105,838,775	14,	24,191,706,	45,359,475	60,479,300	95,775,756	1.340860e9,
Tt	60,479,300			21	193,533,648				8.90714e9
Ts	14/21	15,119,825	105,838,775	21,59,975	3,54,954,	15,119,825	15,119,825	7,	14,
					69,11,916			54	93
Rt	14,	7	15,119,825	14,	60,47,930,	45,359,475	14,	6.33444	6.33444
	21			21	48,383,440		28
Rs	14,	7	15,119,825	2,159,975	8,63,990,	15,119,825	14,	7,	14,
	21				17,27,980		28	54	93
CPU	11276	3731*	7041	5663	29708	18578	87766	74444	96576
GPU	5475	1542*	3988	2407	40462	11658	60035	56811	61252

*Significant values.

In view of the exhaustive information provided in section 3 on analysis, the summary of the trade-off analysis for the industrial dataset is directly produced from interpreting Table 13.

On comparing the bias, kNN and DT has low bias for binary and multiclass classification respectively. However, for both cases, DT had performed with minimum bias. For the variance, kNN showed greater variance. However, a nominal bias/variance trade-off is seen in DT and is the best fit model. Here, RF and SVM show overfitting (i.e., SVM> >RF) and high MSE values. Also, average bias and average variance is taken for 10 models.

In the case of accuracy, LDA has high accuracy score during binary classification. However, the parametric algorithm did not work well for multiclass classification. Consistent performance is shown by non-parametric algorithms except MLP. The non-parametric MLP failed to achieve optimal performance for the industrial dataset. DT and RF are remarkable for both classification tasks.

Undoubtedly, due to increased accuracy, the predictivity and stability of non-parametric algorithms are better however, not their simplicity score. The absolute value of interpretability is high for DT in binary and RF in multiclass classification. Surprisingly, the performance of LDA is same as that of the non-parametric algorithm in this analysis.

RF proved to be more robust with a minimum accuracy deviation of about 2.6% from 0 to 20% noise levels during binary classification and 0.5% only in multiclass classification. The space and time complexity highly favored the parametric algorithms than non-parametric as expected, in particular for LR.

The ranking of each algorithm based on the best performance indices is illustrated as a colormap in Fig. 11.

Fig. 11

Colormap showing the ranks of algorithms for different metrics while performing classification on (a) binary (b) multiclass.

From Fig. 11, for binary case, the non-parametric algorithm, DT is ranked 1 among all other models due to its good accuracy, minimum MCE values and high robustness. Also, kNN showed minimum MSE. In parametric algorithms, LDA classifier stood first for interpretability and LR classifier based on complexity. Similarly, it can be seen that, the last rank is obtained by the parametric models. Though P algorithms satisfy characteristics like flexibility, simplicity, space and time complexity, the trade-off between non-parametric algorithms on other important characteristics is an important measure for model selection. Here, notable trade-offs are witnessed among non-parametric models itself. Thus, the results obtained from the classifiers of the industrial dataset corroborate the better performance of non-parametric models. It also evinces that, the ensemble/tree models outperform kNN or SVM models for both binary and multiclass classification tasks.

5 Conclusion

This paper presented a systematic framework to analyze the trade-offs between the parametric and non-parametric algorithms when subjected to an imbalanced pneumatic actuator dataset which follows a non-normal data distribution. Initially, the structure of the DAMADICS actuator dataset was thoroughly analyzed and found to have non-normal data distribution and imbalance distribution in both binary and multi class targets. Classifier models were built using different parametric and non-parametric algorithms for binary (normal, faulty) classification and multiclass (i.e., normal, clogging fault, sedimentation fault and erosion fault in synthetic dataset; normal, positioner supply pressure drop fault, bypass valve fault and flowrate sensor fault in industrial benchmark dataset) classification. Among the performance of parametric and non-parametric models, the advantages of interpretability and speed of the parametric model remain defenseless against the non-parametric algorithm’s accuracy, robustness and flexibility. Within non-parametric classifiers, k nearest neighbor and decision tree algorithms performed well and was consistent in their outputs for both binary and multiclass problem for synthetic dataset whereas, random forest classifier was found to work ideally due to overfitting of the model. However, for the industrial dataset, random forest classifier was found to outperform and satisfy most of the required statistical characteristics. Since, this work concludes the conformity of non-parametric classifiers for DAMADICS dataset, extending the best fit model to the identification and prediction of faults of the pneumatic actuator with rare category analysis is an important direction for future research.

Footnotes

Acknowledgment

This work is supported by the Centre for Research, Anna University, Chennai, India by providing financial assistance through the award of Anna Centenary Research Fellowship (ACRF).

Appendix

The pseudocodes for the parametric and non-parametric algorithms discussed in Section 2 is given here.

References

Escobet

, Nebot

À.

and Cellier

F.E.

, Fault diagnosis system based on fuzzy logic: Application to a valve actuator benchmark, Journal of Intelligent & Fuzzy Systems 22(4) (2011), 155–171.

Tharwat

, Gaber

, Ibrahim

and Hassanien

A.E.

, Linear discriminant analysis: A detailed tutorial, AI Communications 30(2) (2017), 169–190.

Verikas

, Gelzinis

and Bacauskiene

, Mining data with random forests: A survey and results of new tests, Pattern Recognition 44(2) (2011), 330–349.

Kadra

, Lindauer

, Hutter

and Grabocka

, Regularization is all you need: Simple neural nets can excel on tabular data. arXiv preprint arXiv:9, (2021).

Frénay

and Verleysen

, Classification in the presence of label noise: a survey, IEEE Transactions on Neural Networks and Learning Systems 25(5) (2013), 845–869.

Scholkopf

, Sung

K.K.

, Burges

C.J.

, Girosi

, Niyogi

, Poggio

and Vapnik

, Comparing support vector machines with Gaussian kernels to radial basis function classifiers, IEEE Transactions on Signal Processing 45(11) (1997), 2758–2765.

Ferri

, Hernández-Orallo

and Modroiu

, An experimental comparison of performance measures for classification, Pattern Recognition Letters 30(1) (2009), 27–38.

Bezerra

C.G.

, Costa

B.S.

, and

L.A.

and Angelov

P.P.

, An evolving approach to unsupervised and real-time fault detection in industrial processes, Expert Systems with Applications 63 (2021), 134–144.

Senthilkumar

, Reshmy

A.K.

and Noornisha

, Multi-task learning framework for predicting water quality using non-linear machine learning technique, Journal of Intelligent & Fuzzy Systems 42(6) (2022), 5667–5679.

10.

Pisner

D.A.

and Schnyer

D.M.

, Support vector machine. In Machine learning, Academic Press (2020), 101–121.

11.

DAMADICS, Damadics information web site. http://diag.mchtr.pw.edu.pl/damadics/. Accessed: 04 February, 2020.

12.

Doshi-Velez

and Kim

, Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.08608, (2017).

13.

Yang

F.J.

, An implementation of naive bayes classifier. In International conference on computational science and computational intelligence (CSCI), IEEE (2018), 301–306.

14.

Zhang

G.P.

, Neural networks for classification: a survey, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) 30(4) (2000), 451–462.

15.

Murphy

K.P.

, Machine learning: a probabilistic perspective, MIT Press; 2012.

16.

Bartyś

, Patton

, Syfert

, de las Heras

, and Quevedo

, Introduction to the DAMADICS actuator FDI benchmark study, Control Engineering Practice 14(6) (2006), 577–596.

17.

Bertolini

, Mezzogori

, Neroni

and Zammori

, Machine Learning for industrial applications: A comprehensive literature review, Expert Systems with Applications 175 (2021), 114820.

18.

Han

, Li

, Wang

, Zhang

and Cheng

, Review of ensemble classification over data streams based on supervised and semi-supervised, Journal of Intelligent & Fuzzy Systems 43(3) (2022), 3859–3878.

19.

Zhang

M.L.

and Zhou

Z.H.

, ML-KNN: A lazy learning approach to multi-label learning, Pattern recognition 40(7) (2007), 2038–2048.

20.

Chawla

N.V.

, Bowyer

K.W.

, Hall

L.O.

, and

L.O.

and Kegelmeyer

W.P.

, SMOTE: synthetic minority over-sampling technique, Journal of Artificial Intelligence Research 16 (2002), 321–357.

21.

Mahalingam

, Kalpana

and Thyagarajan

, Overfit Analysis on Decision Tree Classifier for Fault Classification in DAMADICS. In IEEE Madras Section Conference (MASCON), (2021), 1–4.

22.

Mahalingam

, Kalpana

, Sendhilkumar

and Thyagarajan

, Prefatory data analysis approach to synthetically generated pneumatic actuator data set, Proceedings of the Institution of Mechanical Engineers, Part I: Journal of Systems and Control Engineering 236(10) (2022), 1807–1818.

23.

Mahalingam

, Dharmalingam

and Thangavelu

, Univariate Sensor Data Prediction Using Conventional and Machine Learning Based Time Series Techniques. In Advances in Automation, Signal Processing, Instrumentation, and Control, Springer, Singapore, (2021), 651–660.

24.

Resende

P.A.A.

and Drummond

A.C.

, A survey of random forest-based methods for intrusion detection systems, ACM Computing Surveys (CSUR) 51(3) (2018), 1–36.

25.

Espinheira

P.L.

, Silva

L.C.

and Cribari-Neto

, Bias and variance residuals for machine learning nonlinear simplex regressions, Expert Systems with Applications 185 (2021), 115656.

26.

Fisher

R.A.

, The use of multiple measurements in taxonomic problems, Annals of Eugenics 7(2) (1936), 179–188.

27.

Mondal

, Pal

and Dey

, A Hybrid Regularized Multilayer Perceptron for Input Noise Immunity. IEEE Transactions on Artificial Intelligence, (2023).

28.

Dreiseitl

and Ohno-Machado

, Logistic regression and artificial neural network classification models: a methodology review, Journal of Biomedical Informatics 35(5–6) (2002), 352–359.

29.

Raudys

and Raudys

, Pairwise costs in multiclass perceptrons, IEEE Transactions on Pattern Analysis and Machine Intelligence 32(7) (2010), 1324–1328.

30.

Rubinstein-Salzedo

, Big o notation and algorithm efficiency, In Cryptography, Springer, Cham (2018), 75–83.

31.

Safavian

S.R.

and Landgrebe

, A survey of decision tree classifier. Methodology, IEEE Transactions on Systems, Man, and Cybernetics 21(3) (1991), 660–674.

32.

Margot

and Luta

, A new method to compare the interpretability of rule-based algorithms, AI 2(4) (2021), 621–635.

Analysis of the trade-offs between parametric and non-parametric classifiers for classification of faults in pneumatic actuator. Case: DAMADICS

Abstract

Keywords

1 Nomenclature

1 Introduction

2.1 Synthetic dataset

2.2.1 Naïve Bayes (NB) classifier

2.2.6 k Nearest Neighbor (kNN) classifier

2.2.7 Support Vector Machines (SVM) Classifier

2.2.8 Decision Tree (DT) Classifier

2.2.9 Random Forest (RF) Classifier

Table 11 List of Parameters of faults in the industrial data Notation & Fault mode fs (min,max) Type Direction f16 - positioner supply pressure <0,1> Gradual + f18 - bypass valve opening <0,1> Abrupt + f19 - flowrate sensor < -1,1> Gradual +/-

Footnotes

Acknowledgment

Appendix

References

Table 11
List of Parameters of faults in the industrial data

Notation & Fault mode fs_(min,max) Type Direction

f₁₆ - positioner supply pressure <0,1> Gradual +

f₁₈ - bypass valve opening <0,1> Abrupt +

f₁₉ - flowrate sensor < -1,1> Gradual +/-