Combining classifiers with decision templates for automatic fault diagnosis of electrical submersible pumps

Abstract

A machine learning approach to perform automatic detection and diagnosis of faults of electrical submersible pump systems is presented. Several thousand vibration patterns were acquired from vertically distributed accelerometers along the string of motors, pumps and protectors. Intermediate features are extracted from the raw vibration signals originating from the set of accelerometers. Each pattern was labelled by a human expert to provide ground truth with respect to the different operation classes (normal, sensor fault, rubbing, unbalance or misalignment). A software framework is used to compare several classifier architectures (K-Nearest-Neighbor, Random Forest, Support Vector Machine, Naïve Bayes and Decision Trees) in a bias aware performance evaluation. In order to boost the classification performance, an ensemble of different versions of a classifier architecture is constructed using the Decision Templates fusion function. The robustness of the system with respect to the emergence of new faults (i.e., untreated faults so far) is corroborated by a systematic analysis methodology.

Keywords

Fault diagnosis electrical submersible pump classification performance criteria decision templates

1. Introduction

Off-shore oil exploration is an expensive activity that involves the use of sophisticated and expensive equipment. The core subsystem that aids raising the crude petroleum and gas to the surface is known as electrical submersible pump (ESP) as seen in Fig. 1. Such subsystems operate under deep water and, therefore, are of difficult access, hindering real-time supervision. Additionally, production interruptions are very expensive, which reinforces the need for reliable operation of the equipment once deployed. In order to avoid costs with removal and replacement of a defective equipment that is under operation, they are carefully examined in a special testing environment [1, 2] prior to acquisition. Such tests aim to highlight potential faults of the equipment.

Figure 1.

Illustration of a submersible pump string. This figure shows the six parts of the string: two pumps (in light gray), two protectors (in gray) and two motors (in dark gray). At least one pair of accelerometer sensors is attached in the x and y directions (x-y plane is orthogonal to the main string axis) of each part.

Most of the potential faults are due to mechanical problems and appear as a drift of the vibration patterns as compared to the healthy state. Vibration patterns can be measured with accelerometer sensors and then transformed into a more descriptive set of features to aid distinguishing between different operation patterns. The pump subsystems are tested in an artificial environment under various operation conditions while having vibration patterns collected by accelerometer sensors. Two accelerometers sensors are orthogonally placed (X-axis and Y-axis) in several equally distributed points along the six principal components of the equipment. Such a sampling distribution results in a total of 6 $\times$ 3 $\times$ 2 $=$ 36 simultaneously acquired vibration signals in the time domain. Once collected, the signals are taken to the frequency domain with conventional Fourier Transform, where the evaluation of the signals can be performed (see Fig. 2 for an example of the signals). Once the data is analyzed, the equipment can be diagnosed as healthy or faulty.

Figure 2.

Example of data acquired by an accelerometer. The upper plot shows the raw vibration signal in the time domain whereas the lower plot shows the corresponding transformation to frequency domain via Fourier transform.

This work focuses on three types of motor pump faults: shaft misalignment, pump blade unbalance, and mechanical rubbing. Additionally, it considers faulty accelerometer sensors, generating abnormal vibration behaviour as a faulty pattern (the abnormal behaviour is not necessarily related to the pump). Considering the previous faults and the healthy operating condition, a total of five mutually exclusive categories are used to describe a motor pump: misalignment, unbalance, rubbing, sensor, normal. Although a few pumps show signals with evidence of two or more categories, they are rare and were disregarded in this work due to the lack of examples.

Usually, a human expert visually inspects a large amount of vibration spectra of frequencies looking for an error pattern, but this work is time consuming due to the many different curves that have to be analyzed. Moreover, this task requires a lot of tacit knowledge acquired by the human expert along many years of experience. Visually inspecting the vibration spectra and deciding whether the equipment should be rejected is not an easy task. In addition, this knowledge is not transferred or learned by other engineers without years of study and practice. Therefore, the possible loss of a professional with this experience and knowledge significantly affects the decision-making ability of the company and produces a considerable financial impact on its production processes. The expert system presented in this work performs such tasks automatically and systematically, and enables the work to be executed by non specialized personnel. More important, it can keep this type of corporate knowledge.

A frequent concern presented by the company is the need of using a technology that can be adjusted to identify new types of defects. In particular, the inclusion of patterns of new defects in the training dataset should not impair the discriminative power of the previously existing defects. Since the existing knowledge about the diagnosis of electrical submersible pump faults is still incomplete, it is expected that in the future new types of defects will be identified and incorporated into the training dataset when a significant number of examples are accumulated. Indeed, a fact observed throughout this research project is that the human expert eventually pointed to a pattern suspected of being a fault, but without enough conviction to reject the equipment as faulty or characterize the new fault. As a result, the technology must be capable of adapting to the discovery of new defects. Potentially, the technology developed in this work should have this behavior, but it was important to formally show this feature.

Since the popularization of computers, machine learning has been explored as a mean of aiding or replacing daily human tasks, or even specialized human tasks. Classification is one of the key tasks studied in machine learning. Classification tasks are handled in two steps: training and test. In the first, the classifier is presented with examples of labeled data in order to learn a classification model. In the second, the trained model is presented with an unlabeled example in order to predict one of the possible labels. There are many types of classifiers that can be used, with each one being more suitable for specific tasks. Given the literature of fault diagnosis presented below, this work focuses on K-Nearest Neighbor (KNN), Support Vector Machines (SVM), J48 Decision Tree (DTree), Naïve Bayes (NB) and Random Forests (RF).

The KNN [3] is a mandatory comparative classifier, since it has no hyperparameters, except the number $K$ of neighbors and gives a reliable upper bound performance estimation based on the Cover and Hart inequality. It states that in the case of a sufficiently large number of training samples, the performance cannot be worse than twice the theoretical Bayes limit. It is therefore included into the comparative study. SVM [4] is another important and popular classifier that often has achieved the highest accuracy scores in comparative studies. The works [5, 6] are examples of studies using SVM as classifier. Decision trees [7] and Naïve Bayes [8] are two other classifiers that were included in this work. The first is a classifier that constructs a tree with internal nodes representing tests on attributes and with leaf nodes representing classes. The second are simple probabilistic classifiers based on Bayes theorem considering the Naïve assumption of feature independence. A representative of the decision tree classifiers is the J48 implementation of the C4.5 algorithm [9]. In [8], the J48 was compared to the NB classifier. RF is a method introduced by Ho [10] and consolidated by Breiman [11]. It combines many decision tree predictors constructed based on a random subset of input variables and a random subset of samples of the training data. There exist other classifiers not investigated in this work, as for example the Probabilistic Neural Networks that were successfully used for classification in [12, 13]. A neural dynamic approach for classification was presented in [14] and applied to earthquake warning [15]. A hardware support for error backpropagation in a Multilayer Perceptron in the context of deep learning can be found in [16].

Classification is the key tool in model-free fault diagnosis when knowledge is exclusively based on labeled machine condition patterns. Fault diagnosis has been addressed in many computer applications and a set of examples are presented in the following. Surveys of machine fault diagnosis using support vector machine are presented in [17, 18]. A fuzzy approach is applied to fault diagnosis of power systems [19]. Dimensionality reduction and multilayer perceptrons are used to detect surface roughness for face milling operations in [20]. The work in [21] applied qualitative reasoning to monitor and diagnose electrical power quality. Classifier ensembles are used for the detection and isolation in sensor networks in [22]. The Naïve Bayes classifier is compared with a Bayesian net and the J48 algorithm in the context of fault diagnosis of roller bearing using sound signals [23].

The focus of this study is on fault diagnosis based on vibration pattern analysis. Wavelets and local discriminant bases selection algorithm are applied to vibration signals in the fault diagnosis of a single-cylinder spark ignition engine [24]. RFs are applied to vibration signals based fault diagnosis of an induction motor in [25] and to spur gears in [26]. The authors in [27] use locally linear embedding and SVMs for submersible plunger pump fault diagnosis. A systematic use of the Case Western Reserve University (CWRU) data set as a benchmark is proposed in [28]. The data set describes the specific problem of bearing defects and was also explored with a multiscale permutation entropy approach to select wavelet features in [29]. In the same context, the KNN is used as the classifier, together with statistical feature models [30]. Two other works propose to diagnose bearings using wavelet features [31, 32]. An ensemble of SVM classifiers is used to diagnose faults based on vibration signals of oil rig horizontal centrifugal motor pumps [33].

More specifically in the context of this work, a comparative study of KNN, SVM, Decision Trees and RF is performed for ESP fault diagnosis based on vibration signals [34]. Results showed that the evaluated classifiers have equivalent performance and also indicated that the standardization procedure can improve their performance. In a following work [35], the group investigated the use of Extreme Learning Machine (ELM) considering two different versions (based on random and kernel hidden units). Although those works addressed very well the fault diagnosis problem, they only explored the use of single classifiers to detect the faults, leaving room for investigating combination of classifiers. Additionally, they performed a limited evaluation of the system performance without considering the possibility of appearance of new faults along the time.

Following along the line of the research presented in [34], this work presents two new main contributions to the problem of automatic fault diagnosis of electrical submersible pumps.

The first contribution is the use of classifier ensembles with the inclusion of a powerful technique, called Decision Templates (DT) [36, 37], to combine results of more than one classifier constructed via bagging and improve the final performance. A comparative study is performed using real data acquired in tests accomplished before acquisition of electrical submersible pumps. The dataset comprises thousands of entries of accelerometer sensor data labelled by a human expert as one of the considered scenarios (normal, faulty sensor, faulty pump with rubbing, misalignment or unbalance). Differently from [34], results showed higher differences between the classifiers when considering the macro-averaged F-measure performance criterion and showed that the Decision Templates can improve the Random Forest classifier performance (best evaluated performance).

The second contribution is a systematic analysis of the faults by removal and substitution of each of the fault classes in order to evaluate the impact of the emergence of new defect patterns on the performance of the classifiers. New fault patterns might be discovered by the experts and would have to be included in the classifier as a new class. Such new faults might degrade the current performance of the classifier and therefore deserved investigation. It should be highlighted that no similar systematic analysis on the emergence of new defects has been reported in the fault diagnosis literature. Therefore, the proposed methodology may be applied in different contexts of fault diagnosis in which the same type of problem is present. The results of the study showed that the chosen classifier, Random Forest with Decision Templates, is robust with respect to the appearance of new fault patterns.

The remainder of this paper is organized as follows: Section 2 describes the proposed solution for submersible motor pump fault diagnosis, the feature extraction procedure, the classifier ensemble model and the Decision Templates technique. Section 3 describes the experimental methodology used to realize the comparative study. Section 4 presents the experimental results. Section 5 concludes the paper and presents perspectives for future work.

2. Motor pump fault diagnosis system

Expensive electrical submersible pump systems must be tested prior to deployment. This is necessary to avoid unnecessary costs since on-site, sub-sea correction is unfeasible. This work proposes a machine learning approach to address an important oil industry engineering problem. The complete work flow of the diagnosis system is composed of the following stages: i) acquiring raw vibration signals from accelerome-ters sensors attached to multiple positions of the pump system and converting them to the frequency domain, ii) extracting custom made features from the spectrum, iii) training a classifier offline and iv) using the trained classifier to diagnose faults. This workflow is used to determine the most appropriate classifier model combined with DT bagging ensembles for the fault diagnosis task. The feature model, the classifiers ensembles and the DT are described in more detail in the following subsections.

2.1 Feature extraction

The raw vibration data constitute a huge amount of information, and conventional Fourier transformation of the time to the frequency domain provides a spectrum that still has too much information. Therefore, it is inappropriate to be fed directly into a classifier. Although there are several works in the literature that explore the feature extraction by dimensionality reduction [20, 24], in this work hand-crafted features proposed in [34] were used to emulate the tacit knowledge of the domain expert when diagnosing a motor pump. They were designed to jointly create information about the peaks and shape of the spectrum in the range of significant frequencies in order to identify the considered faults. High peaks at the first and second harmonics are related to unbalance and misalignment faults, whereas an exponential decrease at low frequencies are relevant to detect rubbing and sensor fault.

Eight statistical features have been extracted from specific bands of the vibration spectrum of each sensor to be used as input of the classifiers. Each of these features is described below.

•
$x_{\textit{rf}}$ : Rotation frequency (first harmonic) of the ESP;
•
$x_{\textit{rfm}}$ : Magnitude of the rotation frequency (first harmonic);
•
$x_{\textit{rfm2}}$ : Magnitude of the double of the rotation frequency (second harmonic);
•
$x_{\textit{rfrms}}$ : Root mean square of the magnitudes in the frequency band around the rotation frequency, [ $x_{\textit{rf}}$ $-$ 1 Hz, $x_{\textit{rf}}$ $+$ 1 Hz];
•
$x_{\textit{rfmm}}$ : Median of the magnitudes in the frequency band around the rotation frequency, [ $x_{\textit{rf}}$ $-$ 1 Hz, $x_{\textit{rf}}$ $+$ 1 Hz];
•
$x_{\textit{m3to5}}$ : Median of the magnitudes of the low frequency band [3 Hz, 5 Hz];
•
$x_{\textit{ilr}}$ : Intercept ( $a$ ) of the linear regression of the logarithm of the frequency magnitudes (Mag) onto the frequency (Fq) band [5 Hz, 19 Hz], Eq. (1);
•
$x_{\textit{slr}}$ : Slope ( $b$ ) of the linear regression of the logarithm of the frequency magnitudes (Mag) onto the frequency (Fq) band [5 Hz, 19 Hz], Eq. (1);

$\displaystyle\log(\textit{Mag})=a-b\times\textit{Fq}$ (1)

This feature set was designed to optimize the identification of the considered operation conditions of the process taking into consideration the analysis performed by the human expert. This first feature set $\{x_{\textit{rf}},x_{\textit{rfm}},x_{\textit{rfm2}},x_{\textit{rfrms}},x_{% \textit{rfmm}}\}$ is responsible for the detection of peaks around the first and second harmonics, whereas the set $\{x_{\textit{i3to5}},x_{\textit{ilr}},x_{\textit{slr}}\}$ focuses on the exponential decay of the frequency magnitudes in the low frequency half-band.
2.2 Classifier ensembles

Each of the five type of classifiers (Naïve Bayes, KNN, SVM, J48 Decision Tree, Random Forest) motivated in Section 1 are combined in ensembles to raise the performance scores relative to a single classifier. As both SVM and the decision tree used in this study are binary classifiers, for the multi-class case, a set of one-against-one classifiers were trained for all possible class pairs, and were combined in a error-correcting output code multi-class model [38]. It should be stressed that classifier ensembles are homogeneous in this work. Thus, a classifier ensemble is not composed of different types of classifiers, but of different classifiers of the same type. So, there is an ensemble composed by different SVM classifiers, but there is no ensemble mixing, for instance, SVM and KNN classifiers.

Given a number $L$ of classifiers, the $L$ different versions $D_{i}$ , $i=1\ldots,L$ of the same type of classifier are generated by bagging (aka bootstrap aggregating) [39]. A training set $\mathcal{T}$ of $n$ samples is repeatedly submitted $L$ times to bootstrapping [40], i.e. random sampling with replacement to create $L$ different training sets $\mathcal{T}_{i}$ , $i=1\ldots,L$ , each containing $n$ training samples, possibly repeated or absent, compared to the original training set $\mathcal{T}$ . Then, the $i$ -th training set $\mathcal{T}_{i}$ is used to induce classifier $D_{i}$ .

In order to classify an unseen instance, the output of each member of the ensemble is collected and then an ensemble information fusion function determines the final classification result. The choice of this function has fundamental importance for the operation of the ensemble and directly influences its performance. In this work, three ensemble information fusion functions were considered: majority voting, weighted voting and decision templates. In majority voting each classifier votes in one class (typically the best evaluated). The most frequent voted class is chosen. In weighted voting, the mean support of all classifiers is calculated specifically for each class. The final class is the one with the highest mean support. Decision Templates, focus of this work, are described in the next subsection.

2.3 Decision Templates

Decision Templates [36, 37] is a method for deciding the final classification for a single-label classifier ensemble. Given a number $L$ of classifiers $D_{i}$ , $i=1\ldots,L$ for a pattern ${\boldsymbol{x}}$ belonging to exactly one of the classes $\omega_{j}$ of a total of $c$ classes $\{\omega_{j}|j=1,\ldots,c\}$ , the continuous output $d_{i,j}\in\mathbb{R}$ of each of these classifiers for each of the $c$ possible classes is organized in a Decision Profile (DP). One line of the DP store the ‘scores’ $d_{i,j}$ for each of the $c$ classes that the classifier $D_{i}$ assigns.

$\displaystyle\text{DP}({\boldsymbol{x}})\!=\!\left(\!\begin{array}[]{ccccc}d_{% 1,1}({\boldsymbol{x}})&\ldots&d_{1,j}({\boldsymbol{x}})&\ldots&d_{1,c}({% \boldsymbol{x}})\\ \ldots&&&&\\ d_{i,1}({\boldsymbol{x}})&\ldots&d_{i,j}({\boldsymbol{x}})&\ldots&d_{i,c}({% \boldsymbol{x}})\\ \ldots&&&&\\ d_{L,1}({\boldsymbol{x}})&\ldots&d_{L,j}({\boldsymbol{x}})&\ldots&d_{L,c}({% \boldsymbol{x}})\\ \end{array}\!\right).$ (2)

Ideally these scores are probabilities based on the true class conditional distributions. Estimated probabilities, likelihoods, fuzzy set membership degrees or distances are also plausible as scores. Hence the entry $d_{i,j}$ in the DP is the score of the $i$ -th classifier from a total of $L$ different classifiers for the $j$ -th class of a total of $c$ classes.

Consider the same training set, composed of $n$ samples, that is used to train the classifiers $D_{i}$ used to determine the DP Eq. (2) for a sample ${\boldsymbol{x}}$ , now separated for each class with $n_{j}$ samples, $\sum_{j=1}^{c}n_{j}=n$ as $\{{\boldsymbol{x}}_{j,k}|k=1,\ldots,n_{j}\}$ , $j=1,\ldots,c$ . Then the DT of class $\omega_{j}$ is calculated as

$\displaystyle\text{DT}_{j}=\frac{1}{n_{j}}\sum_{k=1}^{n_{j}}\text{DP}({% \boldsymbol{x}}_{j,k}).$ (3)

It is a matrix of the same dimension and structure as a DP, and can be considered as the expected DP of class $\omega_{j}$ . In the operational stage of a classification system, the DP of a new pattern ${\boldsymbol{x}}$ is calculated as $\text{DP}({\boldsymbol{x}})$ and related to the DTs of all classes to obtain a class specific continuous ‘support’ $\mu_{\ell}\in\mathbb{R}$ for class $\omega_{\ell}$ as

$\displaystyle\mu_{\ell}({\boldsymbol{x}})=1-||\text{DP}({\boldsymbol{x}})-% \text{DT}_{\ell}||_{2}^{2}=1-\sum_{i=1}^{L}\sum_{j=1}^{c}\left[d_{i,j}({% \boldsymbol{x}})-\text{DT}_{\ell}({i,j})\right]^{2},$ (4)

where $||.||_{2}$ is the entry-wise matrix norm, or equivalently the Euclidean distance between the linearized DP and DT (all lines have been sequentially assembled into a $L\times c$ vector). After the support $\mu_{\ell}$ has been obtained for each class, a further evaluation takes the final decision. Usually, in a single-label crisp classification task, the highest score defines the selected class. For example, in a binary classification task the first class is chosen if $\mu_{1}({\boldsymbol{x}})\geqslant\mu_{2}({\boldsymbol{x}})$ .

3. Experimental methodology

This section describes the methodology designed to evaluate the performance of the considered classifiers. It starts explaining how data from the motor pumps were collected and how each instance was labelled. With the dataset in hands, the next subsection describes the evaluation metrics, cross-validation procedure and the statistical framework used to compare the performance of the methods. The final subsection describes the methodology used for systematically analyzing the possible emergence of new faults.

3.1 Benchmark data set

The dataset is composed of vibrational signals collected from nine different motor pumps operating at different frequencies (40, 45, 50, 55 and 60 Hz) and flow rates, following strict tests procedures described in the standard API RP 11S8 for ESP vibration analysis. All procedures are accomplished in test wells using electrically isolated waterproof (100 m of depth) accelerometer sensors with a 0.1 V/g sensitivity and operating in the frequency range of 0.6 to 10000 Hz (maximum tolerance $\pm$ 10%). The accelerometers shall sample at a rate of 4096 Hz (to avoid spectral anti-aliasing filter distortion in the end of the spectrum scale) during the minimum time length of 160 seconds.

For most of the data used in this study, a total of 36 accelerometer sensors were attached to the components of each equipment (each motor pump is composed of pumps, protectors and motors), resulting in the acquisition of a total of 4570 vibration signals. The vibration spectra of each signal were analyzed by a domain expert in order to define the labels. The expert assigned each spectrum to one out of the five considered categories resulting in the following percentage of occurrence: normal (3706 samples representing 78.02%), sensor (294 samples representing 6.19%), unbalance (485 samples representing 10.21%), misalignment (50 samples representing 1.05%), or rubbing (35 samples representing 0.74%). As expected, the dataset is imbalanced with the normal condition being the majority category.

3.2 Performance evaluation

In order to ensure a fair and unbiased evaluation of each classifier, a common method was created and used to test each classifier type. In the context of machine learning it is an accepted fact that simple training-test splits of the data are not enough to reliably obtain a realistic statement about the performance of a classifier. In fault diagnosis applications a sophisticated methodology to estimate performance scores is rarely encountered. The best hyperparameters of a certain classifier are often obtained by submitting the same data over and over again, until a good set is found. For instance, the constraint parameter $C$ of a C-Support Vector Machine and the spread parameter $\gamma$ of an associated RBF kernel are often determined by trial-and-error, using the total data set. This introduces an unfair bias into the classifier. The methodology used in this work avoids this bias by isolating the search for the best parameters into a tuning procedure that has no access to the test set, employing considerably more elaborated validation techniques [41].

The performance evaluation framework is divided into three nested layers. The outer layer executes $R$ rounds. In each of these rounds, the total amount of $n$ patterns is randomly split into $K$ stratified folds. The intermediate layer trains the classifier with $(K-1)$ folds and tests the remaining fold in a usual $K$ -fold cross validation (CV) procedure (outer CV loop). However, before the training of the classifier is performed, the best hyperparameters for the classifier are estimated in a tuning step. The tuning is achieved by an inner layer (inner CV loop). The inner CV loop further subdivides the training data of the outer CV loop into $\tilde{K}$ stratified folds. Now, the search space of the hyperparameters is exhaustively examined. For each hyperparameters candidate, the performance of the classifier is estimated by using the $\tilde{K}$ folds in a $\tilde{K}$ -fold cross validation. The performance mean of the $\tilde{K}$ folds is used to decide the best hyperparameters set that are finally used in the training stage of the outer CV loop. In this work, the number $\tilde{K}$ of the inner loop is set to $\tilde{K}=K-1$ . This strategy permits to reuse the fold division of the outer loop.

As an example for the search of the best hyperparameter in the inner CV loop for tuning, take the K-Nearest Neighbor classifier.1 Supposing that the Euclidean distance metric is not modified, the only hyperparameter is the number $K$ of neighbors. The search would then seek the best $K$ , for instance using $K=$ 1, 3, 5, $\ldots$ as candidates.

This strategy implies that $R(\textit{rounds})\times K(\textit{folds})$ different optimized hyperparameters could be detected, such that is not possible to indicate a globally valid best set of hyperparameters. For instance in the K-Nearest Neighbor classifier, different best number of neighbors may occur during the tuning stage, or different kernels and kernel parameters for the SVM. In the tuning stage, the $\tilde{K}$ folds within round $r\in R$ is the only knowledge that is allowed to be used to determine the optimized hyperparameters $\mathcal{P}^{*}$ . The basic search strategy is a grid search that exhaustively explores a Cartesian product of all hyperparameters that can be varied. For instance, in the Random Forest classifier there are two hyperparameters to be varied, namely the number of trees and the number of features used in the tree. Table 1 gives an overview of the type of hyperparameters for each classifier and the range used in the grid search. For a particular classifier, the sweep range of the grid search is the Cartesian product of all values of all hyperparameters. For instance, the SVM tests all the following $(\log_{2}(C),\gamma)$ combinations during tuning: {(5, 2.0), (7, 2.0), (13, 2.0), (15, 2.0), (5, 8.0), (7, 8.0), (13, 8.0), (15, 8.0)}. The possible combinations for the classifiers had to be limited to allow reasonable execution time, mainly depending on the complexity of each classifier architecture. The complexity of the evaluation framework with $R$ rounds, $K$ and $\tilde{K}$ folds of the test and tuning stage, and $L$ versions of the classifier in the ensemble is $\mathcal{O}(R\cdot K\cdot\tilde{K}\cdot L)$ . However, this complexity must be multiplied with the inner complexity of the particular classifier which is quite distinct.

Table 1
Hyperparameters used for each classifier architecture

Method	Hyperparameters	Range
NB	Kernel density estimator	{yes, no}
	Supervised discretization	{yes, no}
KNN	Number of neighbors	{1, 3, 5, $\ldots$ ,15}
SVM	C-SVM C, $\log_{2}(.)$	{5, 7, 13, 15}
	RBF Kernel $\gamma$	{2.0, 8.0}
DTree	Use binary splits only	{yes, no}
	Reduced error pruning	{yes, no}
	Laplace smoothing	{yes, no}
	Pruning confidence threshold	{0.2, 0.25, 0.3}
	Reduced error pruning folds	{2, 3, 4}
	Minimum instances per leaf	{2, 3, 4}
RF	Number of trees	{100, 1000}
	Number of features	{1, 2, 3, 4, 5}

The performance evaluation framework is depicted in Algorithm 1. The following symbols are used:

$\mathcal{C}$	classifier
$\mathcal{D}$	total data set of $n$ patterns
$\mathcal{T}$	training set
$\mathcal{V}$	test set
$\mathcal{P}$	hyperparameter candidate set
$\mathcal{P}^{\ast}$	best hyperparameter set
$R$	number of rounds
$K$	number of folds of the outer CV loop for overall performance estimation
$f_{r,k}$	$k$ -th fold of $r$ -th round
PM	$R\times K$ performance matrix
$\tilde{K}$	number of folds of the inner CV loop for tuning
$c r i t$	performance criterion (macro-averaged F-measure)

Range Performance evaluation framework

FnFunctionend PM $\leftarrow$ ModelPerformance( $\mathcal{D}$ ,R,K) $n$ labelled patterns $\mathcal{D}$ from $c$ classes, $R$ number of rounds, number of folds $K$ (outer loop), $\tilde{K}$ (inner loop) For each round $r\leftarrow 1,\ldots,R$ make $K$ stratified folds. One fold is fixed as test, remaining folds are train and tune. Calculate performance criterion for each combination of round and fold and put into $R\times K$ performance matrix PM

( // all rounds) $r=1$ R // generate $K$ stratified folds $f_{r,k},k=1,\ldots,K$ ( // all folds) $k=1$ K // training set of $k$ -th fold of $r$ -th round $\mathcal{T}\leftarrow\left\{f_{r,1}\cup\ldots\cup f_{r,K}\right\}\setminus f_{% r,k}$

// test set of $k$ -th fold of $r$ -th round $\mathcal{V}\leftarrow f_{r,k}$

$\mathcal{P}^{*}\leftarrow$ Tuning( $\mathcal{T}$ , $\tilde{K}$ )

$\mathcal{C}\leftarrow$ trainclassifier( $\mathcal{T},\mathcal{P}^{*}$ ) $crit\leftarrow$ testclassifier( $\mathcal{C},\mathcal{V}$ ) PM(r,k) $\leftarrow crit$ $\mathcal{P}^{*}\leftarrow$ Tuning( $\mathcal{\tilde{D}}$ , $\tilde{K}$ ) Data set $\mathcal{\tilde{D}}$ , number of tuning folds $\tilde{K}$ Optimal hyperparameter set $\mathcal{P}^{*}$ of classifier model

initialize best criterion $maxcrit\leftarrow 0$ // generate $\tilde{K}$ stratified folds $\tilde{f}_{k},k=1,\ldots,\tilde{K}$

( // grid search for best hyperparameter set $\mathcal{P}^{*}$ )search completed generate hyperparameter candidate set $\mathcal{P}$

( // all folds) $k=1$ $\tilde{K}$

// training set of $k$ -th fold $\mathcal{T}\leftarrow\left\{\tilde{f}_{1}\cup\ldots\cup\tilde{f}_{\tilde{K}}% \right\}\setminus\tilde{f_{k}}$

// test set of $k$ -th fold $\mathcal{V}\leftarrow\tilde{f}_{k}$

$\mathcal{C}\leftarrow$ trainclassifier( $\mathcal{T},\mathcal{P}$ ) $\textit{crit}_{k}\leftarrow$ testclassifier( $\mathcal{C},\mathcal{V}$ )

$\overline{\textit{crit}}\leftarrow\textit{mean}(\textit{crit}_{k})$ $\overline{\textit{crit}}>\textit{maxcrit}$ $\textit{maxcrit}\leftarrow\overline{\textit{crit}}$ ; $\mathcal{P}^{\ast}\leftarrow\mathcal{P}$

3.2.1 Performance criterion

The main performance criterion used in this work is the macro-averaged F-measure [42], i.e. the relations between the data positive labels and those given by a classifier. This choice is motivated by the simultaneous consideration, in one single value, of precision and recall, derived from the confusion matrix. Macro-averaging is chosen since it treats all classes equally while micro-averaging favors classes with more examples. In a fault diagnosis task, the normal class is usually over-represented, since much more patterns come from a normal machine operating condition than faulty patterns. Consider a multi-class classification problem with $c$ classes $\{\omega_{j}|j=1,\ldots,c\}$ . For each class $\omega_{j}$ , in a one-against-all scenario, the individual true positives, false positives and false negatives are $\text{tp}_{j}$ , $\text{fp}_{j}$ and $\text{fn}_{j}$ , respectively. From these parameters, the macro-averaged precision over $c$ classes can be defined as

$\displaystyle\text{Precision}_{M}=\frac{1}{c}\sum_{j=1}^{c}\frac{\text{tp}_{j}% }{\text{tp}_{j}+\text{fp}_{j}},$ (5)

and the macro-averaged recall over $c$ classes can be defined as

$\displaystyle\text{Recall}_{M}=\frac{1}{c}\sum_{j=1}^{c}\frac{\text{tp}_{j}}{% \text{tp}_{j}+\text{fn}_{j}}.$ (6)

Finally the macro-averaged F-measure is

$\displaystyle F_{M}=\frac{(1+\beta^{2})\text{Precision}_{M}\text{Recall}_{M}}{% \beta^{2}\text{Precision}_{M}+\text{Recall}_{M}}.$ (7)

The parameter is $\beta=1$ in all experiments, attributing the same importance to both precision and recall.

Data standardization is applied, i.e. substituting the original feature value $x_{i}^{j}$ of the $i^{th}$ feature of the $j^{th}$ pattern by its standard score $(x_{i}^{j}-\mu_{i})/\sigma_{i}$ , where $\mu_{i}$ and $\sigma_{i}$ are the mean and standard deviation of the $i^{th}$ feature. This preprocessing step was motivated by previous work [34] that suggested improved performance compared to non-standardized features. Note that the data standardization is performed such that the test data is isolated (i.e. the standardization parameters are calculated with the training set and used for the test), and, for brevity, this process is not shown in Algorithm 1. In summary, F-measure scores are generated for all classifier methods, with standardization, with and without bagging ensembles using the three information fusion functions (majority voting, weighted voting and decision templates), i.e. 5 $\times$ 4 $\times R\times K$ scores, and are statistically evaluated considering the $R\times K$ results for each method.

In addition to the macro-averaged F-measure ( $F_{M}$ ), the results of the experiments with the best combination method are further explored with a confusion matrix and individual performance criteria (accuracy, precision, recall and F-measure) to improve the interpretability of the classifier performance with respect to individual faults. The single representative confusion matrix is calculated as a sum of the confusion matrices of all rounds and folds. The overall accuracy and the precision, recall and F-measure for each class are calculated based on the respective number of individual true positives, false positives and false negatives of this single confusion matrix.

3.2.2 Statistical analysis

In this section, the statistical procedure to compare the performance of the evaluated classifiers is described. There is a total of twenty different classifiers, five different architectures (RF, SVM, KNN, J48, NB), each without and with the application of bagging ensembles using the three information fusion functions. The classifiers were compared pair-wisely in order to select the most suitable for the proposed classification problem. The dataset split over the rounds and folds was ensured for every classifier.

One of the assumptions of the well-known paired $t$ -test is the independence among samples. This is not the case for this experimental framework, since the intersection of the training sets is not the empty set. For that reason, the corrected $t$ -test proposed by [43] was used to assess whether the performances of each pair of classifiers are significantly different on average. In [43], it was proposed a simple estimation of the correlation derived from the overlapping training sets to correct the usual test statistic of the paired $t$ -test shown in Eq. (8). As in the usual $t$ -statistic, $\overline{d}$ and $s^{2}$ are the mean and variance of the differences between F-measure scores of two different methods, respectively. The ratio $n^{\text{TE}}/n^{\text{TR}}$ measures correlation of the differences, where $n^{\text{TR}}$ is the size of the training set and $n^{\text{TE}}$ is the size of the test set.

$\displaystyle t=\frac{\overline{d}}{\sqrt{s^{2}\left(\frac{1}{R\times K}+\frac% {n^{\text{TE}}}{n^{\text{TR}}}\right)}}.$ (8)

Large values of $t$ indicate significant difference in performance between the particular pair of classifiers. The $p$ -value of the test is given by 2 $\times P(T>|t|)$ , where the random variable $T$ follows the standard Student’s $t$ -distribution with $R\times K-1$ degrees of freedom.

In order to control the family-wise error, i.e. the cumulative Type I error, Holm’s step-down procedure [44] was employed. It starts by increasingly ordering the $p$ -values associated with each pairwise comparison, $p_{(1)},\ldots,p_{(q)}$ , where $q=\frac{m(m-1)}{2}$ is the total number of comparisons ( $q=$ 45, i.e. number of combinations between $m=$ 5 $\times$ 2 $=$ 10 different algorithms, c.f. also the upper triangular part of Fig. 4). Then, for a given significance level $\alpha$ , it finds the minimum index $v$ that satisfies

$\displaystyle p_{(v)}>1/(q+1-v).$ (9)

Finally, it rejects the hypotheses of similarity in performance of tests with the following $p$ -values $p_{(1)},\ldots$ , $p_{(v)}$ . In other words, the rejected hypotheses associated with a pair of classifiers express the fact that this pair is significantly different. These pairs are tagged in boldface in the upper triangular part of Fig. 4. All other pairs with $p>p_{(v)}$ are accepted as performing similarly.

3.3 Robustness analysis

The objective of this methodology is the analysis of the diagnosis system with respect to the emergence of new faults. For instance, it may happen that at the operational stage of the system, a novel fault class arises that has never been seen before, and it should be classified. Alternatively, a new fault already present in the data might be discovered and has to be classified. The system should be able to be retrained with examples of such cases while exhibiting a certain amount of robustness for these situations, i.e. it should be able to handle the new class without degrading the performance of the old classes. In an ideal scenario, the system performance would not be affected by inclusion of a new fault class.

In order to test this robustness, an existing fault is either omitted or is declared as a normal situation. In this way, the omitted existing fault can simulate a new fault that was never seen before and the existing fault declared as normal can simulate an already seen pattern that was not yet considered as a fault. For each of the four considered fault classes, three different scenarios are assembled, c.f. Table 2. In the case of “Removal”, the patterns associated with an existing fault are erased from both the training and the test set, and the performance evaluation of Algorithm 1is rerun to obtain a new set of $R\times K=$ 10 $\times$ 10 $=$ 100 macro-averaged F-measure values.

Depending on the hardness to diagnose the particular fault, the F-measure could drop or raise, with respect to the case when the fault is not removed. In order to permit a comparison with the latter, the “Original” case is defined. From the original training and test evaluation which produces a 5 $\times$ 5 confusion matrix, the row and column belonging to the fault are removed, recalculating the F-measure from the reduced 4 $\times$ 4 confusion matrix. The F-measures from the original 5 $\times$ 5 matrix are not directly comparable to the $4\times$ 4 “Removal” matrix, since the macro-averaged F-measure is calculated based on the individual F-measures of each class. Therefore, since a different number of classes would lead to a biased comparison, the row and column in the confusion matrix of the analyzed class is removed.

Table 2
Experimental methodology for robustness analysis

	Methodology
	Original	Removal	Substitution
Affected item
Training set	Unchanged	Fault patterns removed	Fault patterns merged with normal class
Test set	Unchanged	Fault patterns removed	Fault patterns merged with normal class
5 $\times$ 5 Confusion matrix	Line and column of fault removed, becomes 4 $\times$ 4	Becomes 4 $\times$ 4	Becomes 4 $\times$ 4

The “Substitution” case is when the labelled patterns of a certain fault are re-labelled as normal patterns, thus subtracting the fault patterns from the training and test sets and adding them to the normal case. Again, the consequences on the confusion matrix is the reduction of the dimension to four, since the fault does not appear as a separate class anymore.

4. Experimental results and discussion

This section presents the results obtained with the experiments described in the previous section. The performance evaluation framework was developed in the WEKA workbench (version 3.4.3) [45]. The classifiers (Naïve Bayes, KNN, SVM, Decision Tree, Random Forest) were tested using their respective WEKA implementation (NaiveBayes, IBk, LibSVM – wrapper class for the libsvm library, J48 and RandomForest).

The experiments are divided into two parts. The first part is a comparative study of all classifier architectures, evaluating each type of classifier with and without the use of the ensemble information fusion functions (DT, Majority voting, Weighted voting). The objective is to obtain a hint that the use of ensembles is beneficial concerning the performance scores. The classifier that achieved the highest score is retained for the second part of the experiments. This part analyzes the effects of removing a certain fault from the classes (i.e. removing from the examples), or considering a fault as a normal class (i.e. changing their label to the normal class). The objective of this experiment is to evaluate the performance of the classification framework for more realistic situations. During the operation of the fault classification, it is plausible that new situations arise. For instance, broken rolling bearings manifest themselves in changed vibration pattern. It is expected from the diagnosis system that it is still able to discern the learned faults with a certain amount of robustness. That means that the classifier performance should not degrade significantly when it is trained considering new fault classes.

4.1 Comparison of standalone classifiers and bagging ensembles

The aim of this experimental stage is to determine which classifier performs best for the given database of the machine conditions. The experiments were performed with the total amount of $n=$ 4570 examples for $R=$ 10 rounds, in each round using a $K=$ 10 fold cross validation performance estimation. The number of different versions used in the bagging process was set to $L=$ 10 classifiers. The boxplot of the chosen performance criterion for each of the classifiers is a very expressive non-parametric graphical presentation tool adopted in this study. Figure 3 illustrates the results of the F-measures for each method with and without bagging ensembles for each of the three ensemble information fusion functions. The corresponding numerical values are exposed in Table 3.

Figure 3.

Boxplot of resulting macro F-measures for each classifier model. The superscript indicates the ensemble information fusion function (DT $=$ Decision Templates, WEI $=$ Weighted voting, MAJ $=$ Majority voting).

Table 3

Summary of the macro F-measures performance criteria boxplot statistics for each classifier method. The superscript indicates the ensemble information fusion function (DT $=$ Decision Templates, WEI $=$ Weighted voting, MAJ $=$ Majority voting). The highest scores for each of the boxplot statistic parameters and each of the classifier models are highlighted in boldface

Method	Minimum	1st quartile	Median	3rd quartile	Maximum
RF	0.6866	0.7505	0.8033	0.8663	0.9148
RF ${}^{\text{DT}}$	0.6821	0.7803	0.8324	0.8631	0.9317
RF ${}^{\text{WEI}}$	0.6821	0.7433	0.7861	0.8231	0.9123
RF ${}^{\text{MAJ}}$	0.6632	0.7482	0.7861	0.8189	0.9089
SVM	0.5872	0.6976	0.7425	0.7876	0.8930
SVM ${}^{\text{DT}}$	0.5816	0.6753	0.7181	0.7641	0.8702
SVM ${}^{\text{WEI}}$	0.5705	0.6690	0.7131	0.7506	0.8646
SVM ${}^{\text{MAJ}}$	0.58	0.6732	0.7197	0.7544	0.8511
KNN	0.6503	0.7557	0.8016	0.8409	0.9360
KNN ${}^{\text{DT}}$	0.5337	0.6715	0.6927	0.7173	0.7998
KNN ${}^{\text{WEI}}$	0.5325	0.6524	0.6833	0.71	0.7519
KNN ${}^{\text{MAJ}}$	0.5311	0.6402	0.6744	0.7053	0.7549
J48	0.6275	0.7303	0.7573	0.7903	0.9008
J48 ${}^{\text{DT}}$	0.6806	0.7529	0.8023	0.8414	0.9471
J48 ${}^{\text{WEI}}$	0.65	0.7147	0.7666	0.8110	0.9409
J48 ${}^{\text{MAJ}}$	0.65	0.7147	0.7666	0.8110	0.9409
NB	0.5673	0.6372	0.6573	0.6877	0.7446
NB ${}^{\text{DT}}$	0.5635	0.6358	0.6611	0.6902	0.7253
NB ${}^{\text{WEI}}$	0.5267	0.5756	0.6033	0.6296	0.6868
NB ${}^{\text{MAJ}}$	0.5255	0.5754	0.6025	0.6302	0.7131

Analyzing the results, one may see that the RF classifier using DT as the ensemble information fusion function achieved the best performance among all classifiers since it has the highest median performance and high values of the other statistical metrics. Moreover, DTs outperformed the other ensemble information fusion functions for all classifiers, except for SVM where the majority voting was slightly better. Therefore, the classifiers with and without ensembles are pairwisely compared considering DT as the ensemble information fusion function.

Figure 4 presents the results of the corrected $t$ -test for all combinations of pairs of classifier methods with and without DT bagging ensembles. The principal diagonal indicates the method. The upper triangular part shows the $p$ -values of the corrected $t$ -test for each pair of methods. The highlighted $p$ -values correspond to hypotheses that can be rejected at a significance level of 5% ( $\alpha=$ 5%), based on the threshold calculated from Eq. (9). The lowest $p$ -value is $1.11e-15$ at position (1,10) of Fig. 4 for the pairwise comparison of Random Forest with Decision Templates (RF ${}^{\text{DT})}$ against Naïve Bayes with Decision Templates (NB ${}^{\text{DT}}$ ). The highest $p$ -value in the ordered sequence that satisfies the constraint of Eq. (9) is $p_{(v=18)}=$ 0.001207 $<\alpha/(q+1-(v))=$ 0.05/(45 $+$ 1 $-$ 18) $=$ 0.0017857 for the pair SVM-NB at position (3,9) of Fig. 4. Consequently 18 similarity hypotheses are rejected. The lower triangular part of Fig, 4 shows the histograms of the differences between the F-measures for each pair of methods. The histograms allow to see whether the difference between the pair of methods is mostly positive, negative, or around zero.

Figure 4.

Matrix plot of results of the pairwise comparison between methods. The upper triangle presents corresponding $p$ -values of the correlated $t$ -test for each pair of methods. The lower triangle shows the histograms of the differences of the macro F-measures for each pair of method. The zero of the horizontal axis is in the center of each histogram box.

The results suggest that the employment of Decision Templates has a considerable impact on the performance for a particular classifier. In the case of KNN and SVM, the F-measure value decreases after DTs are used. However, an improvement can be observed, when DT is used, considerably for RF and J48 and slightly for NB. Performance variance is probably due to the stability of the learning algorithm. Stable algorithms do not change considerably their predictions when the training data is slightly modified. According to [39], bagging ensembles do not have good performance with stable algorithms. KNN and SVM are stable while RF, J48 and NB are less stable.

Based on the extensive quantity of cross-validation experiments with $R\times K=$ 100 tests, the performance scores of Fig. 3 and Table 3 are taken as a basis for deciding which classifier model will be used for the following experiments. Even though none of the methods were able to statistically outperform the others on average, the RF ${}^{\text{DT}}$ was chosen because it is significantly different from the models SVM ${}^{\text{DT}}$ , KNN ${}^{\text{DT}}$ , NB and NB ${}^{\text{DT}}$ , and exhibits the highest mean and median F-measures among all architectures.

The macro-averaged F-measure might give an incomplete impression of the classifier performance. Thus, for improving the interpretability of the performance of the winning classifier RF ${}^{\text{DT}}$ , a confusion matrix is shown in Table 4. Since there is one confusion matrix for each evaluation round and fold (i.e., $R=10\times K=10$ ), the information is synthesized in one single representative matrix (Table 4) by summing up the $100$ confusion matrices. In result, the sum of the rows in this matrix is $10\times$ the number of examples in each class since each round comprises 10 folds with $1/10$ of the examples to be used as the test set. In addition, the precision, recall and F-measure values for each class are exposed in Table 5. Once again, in order to consider the $R$ rounds and $K$ folds of the evaluation framework, the overall accuracy value of 0.98812 and the additional metrics were calculated using the confusion matrix of Table 4.

The results in Tables 4 and 5 show that the patterns of some classes are more informative than the others. Indeed, rubbing and misalignment are more often wrongly labeled, whereas normal, accelerometer faults and unbalance are more precisely identified. The inferior performance of the classifier for identifying rubbing and misalignment is probably due to the low numbers of labeled examples for those classes. Therefore, an increase in performance is expected when more data of those faults becomes available.

Table 4

Confusion matrix of the RF ${}^{\text{DT}}$ classifier. The matrix was calculated as a sum of each of the 100 matrices, i.e., considering the ten rounds times ten folds

Norm.	Acc. Fault	Rubb.	Misa.	Unba.	True label
36253	198	221	39	349	Norm.
111	2812	7	10	0	Acc. Fault
161	0	169	10	10	Rubb.
38	0	0	380	82	Misa.
63	0	9	49	4729	Unba.

Table 5

Precision, Recall, and F-measure metrics for each class. These metrics were calculated using the information in Table 4

Metric	Normal	Accelerometer Fault	Rubbing	Misalignment	Unbalance
Precision	0.98982	0.93422	0.41626	0.77869	0.91470
Recall	0.97822	0.95646	0.48286	0.76000	0.97505
F-measure	0.98399	0.94521	0.44709	0.76923	0.94391

4.2 System robustness

Figure 5.

Boxplot of macro F-measures for each of the three scenarios, “Original”, “Removal” and “Substitution”, described in Table 2, separated for each of the four considered fault classes that were removed or substituted.

This subsection describes the behavior of the RF ${}^{\text{DT}}$ classifier considering the emergence of new faults. The boxplots for the three scenarios, grouped for each fault class, are shown in Fig. 5. The corresponding numerical values are exposed in Table 6. For the accelerometer fault, the median performance decreases for removal and substitution, compared to the original fault diagnosis task. However for rubbing and misalignment, the removal and substitution increases the median performances. For unbalance, the removal and substitution improve and deteriorate the median performance respectively. None of the differences were by a large margin and the findings may be different if other statistics are considered to draw the conclusions. This suggests that there is no statistically significant change in the diagnosis capability of the system when a new fault surges. In order to corroborate this hypothesis, the same statistical test of Section 4.1 to determine the best classifier model is used. If for all faults no significant difference could be confirmed when a fault is removed or declared as normal, then the robustness of the system seems indicated. The results of Holm’s step-down procedure are shown in Table 7.

Table 6

Summary of the macro F-measures performance criteria boxplot statistics for each fault class, comparing the three scenarios of Table 2. 1 $=$ Accelerometer Fault, 2 $=$ Rubbing, 3 $=$ Misalignment, 4 $=$ Unbalance

Statistics	ORIG1	REM1	SUB1	ORIG2	REM2	SUB2	ORIG3	REM3	SUB3	ORIG4	REM4	SUB4
Minimum	0.6198	0.6489	0.6202	0.8116	0.8181	0.8222	0.7080	0.7148	0.7097	0.6153	0.6667	0.6542
1st Quartile	0.7391	0.7535	0.7505	0.8943	0.8951	0.8928	0.7879	0.8024	0.7869	0.7350	0.7732	0.7430
Median	0.8035	0.7964	0.7909	0.9149	0.9238	0.9173	0.8360	0.8421	0.8403	0.8021	0.8240	0.7829
3rd Quartile	0.8418	0.8432	0.8452	0.9407	0.9491	0.9461	0.8790	0.8803	0.8719	0.8423	0.8610	0.8438
Maximum	0.9362	0.9483	0.9591	0.9832	0.9803	0.9869	0.9408	0.9369	0.9558	0.9246	0.9395	0.9474

The input parameters of Holm’s step-down procedure are $m=$ 3 cases (Original, Removal and Substitution), hence $q=\frac{m(m-1)}{2}=$ 3 pairwise comparisons. Accordingly for a significance level of $\alpha=$ 0.05 the $q=$ 3 dynamic thresholds Eq. (9) are calculated as $\left\{\frac{\alpha}{q+1-1},\frac{\alpha}{q+1-2},\alpha\right\}=\left\{\frac{1% }{60},\frac{1}{40},\frac{1}{20}\right\}$ . The $p$ -values of Table 7 are considerably above this dynamic threshold, after having been ordered in an ascending order. Therefore, for all four faults the hypothesis rejection index is $(v)=$ 0 which means that no significant difference between the pairwise performance comparison of two scenarios could be detected.

This result supports the hypothesis that the system is robust relative to the inclusion of new faults since the inclusion of a new defect did not significantly change the performance of the other classes. Therefore, it indicates that the system is able to handle new faults in case it has to be retrained to cope with additional fault patterns. This fact is important from the practical point of view because new fault pattern might be perceived by an expert with time. In this case, the system would have to be able to handle the new fault without degrading the performance with the old patterns. The “Removal” results indicate that the system performance will not degrade in case the oil company discovers a new fault pattern that was not yet present in the training examples. As an example consider the case in which the research department of the oil company discovers new fault patterns through simulation and decides to train the system to look for those patterns. The “Substitution” results indicate that the system performance will not degrade in case the oil company expert realizes that a pattern, apparently normal, is actually a new fault pattern that should be considered by the system.

When looking at each fault individually in Table 5, it is possible to see that rubbing is one of the most difficult fault to identify. This is confirmed by the result presented in Fig. 5 that shows a performance increase when the rubbing is removed from the problem (i.e., comparing only the “Original” experiments among the different faults). Following the same logic, the second most difficult fault is misalignment and this can also be confirmed by the results presented in Table 5, and Fig. 5. The unbalance fault shows a peculiar behavior in Fig. 5. When compared to the original experiment, it slightly decreases performance with the “Substitution” and increases with “Removal”. This could indicate that unbalance is somewhere in between the normal and fault classes in the feature space, which brings the other classes closer to normal during the “Substitution” and further apart during the “Removal”. This fact would cause respectively a decrease in performance (due to the harder problem the classifier had to solve) and an increase in performance (due to the simpler problem the classifier had to solve).

Table 7

Results for Holm’s step-down procedure

Pair	ORIG vs REM	ORIG vs SUB	REM vs SUB
Accelerometer fault
$p$ -values	0.709	0.819	0.979
Rubbing
$p$ -values	0.418	0.719	0.694
Misalignment
$p$ -values	0.549	0.903	0.642
Unbalance
$p$ -values	0.135	0.982	0.086

5. Conclusion

This paper investigated the use of bagging ensembles with Decision Templates together with different classifier architectures in order to define the most appropriate model for the proposed diagnosis system of submersible pumps. Bias aware extensive cross validation was first used to determine that the Random Forest together with Decision Templates is the system with the highest scores. The chosen RF ${}^{\text{DT}}$ classifier architecture was then investigated for special situations of the occurrence of faults in order to perform a stress test of the robustness of the diagnosis system. First a certain fault was removed from the data base. Then, the patterns of the fault were incorporated into the normal class. In both scenarios, the diagnostic performance of the system did not change significantly, as suggested by the results of Holm’s procedure to check the hypothesis that classifiers have similar performance scores.

From the engineering point of view, the results suggest that the system can aid non experts to perform, with less supervision, the work of highly trained experts. This results in an economy for the oil company of the hiring process, since a smaller number of expensive highly trained experts would be necessary. In addition, it would reduce the evasion of knowledge caused by the absence of the highly trained expert. Today, this kind of expert is in high demand because they have to supervise the whole pump testing procedure that is performed at the pump manufacturer, including all expenses (travel and personnel). One single test costs in turn of 500,000 dollars, therefore, it must be performed with precision and the analysis must be conclusive. The results showed that the system can perform systematically well. A proper fault detection avoids high intervention cost losses (ranging from 20 to 70 million dollars) to pull out ESP system from sub-sea satellite wells.

In future work, the causes of the performance variation of the classifiers should be investigated more and additional fault classes should be considered. Furthermore, data from the same motor pump was collected multiple times, however any dependencies between sensors were ignored in this work. An analysis considering the correlations between various vibration signals might be envisioned as future work when data from more motor pumps become available. Moreover, it is worth mentioning that the proposed system suffers the general drawbacks of every model-free fault diagnosis system. Enough representative data samples have to be provided in order to obtain a good generalization of the classifier. This is especially critical, if some machine conditions are underrepresented because they are very rare situations. Lastly, deep learning techniques have shown an incredible performance in various areas of research. Thus, some unidimensional convolution approaches, such as [46, 47], should be investigated to try to replace the handcrafted features designed for this problem.

Footnotes

The letter ‘K’ is generally both used for the number of neighbors of the KNN classifier and for the number of folds of a K-fold cross validation. Please be aware of the context.

Acknowledgments

This work was supported by CENPES-Petrobras under Grant Termo de Cooperação 0050.00070332.11.9 Petrobras-UFES.

References

Toliyat

Nandi

Choi

Meshgin-Kelk

. Electric Machines: Modeling, Condition Monitoring, and Fault Diagnosis, CRC Press, 2016.

Pasman

. Risk Analysis and Control for Industrial Processes – Gas, Oil and Chemicals: A System Perspective for Assessing and Avoiding Low-Probability, High-Consequence Events, Elsevier Science, 2015.

Cover

Hart

. Nearest neighbor pattern classification, Information Theory, IEEE Transactions on. 1967 Jan; 13(1): 21-27.

Cortes

Vapnik

. Support-vector network, Machine Learning. 1995; 20.

Dai

Cao

. A Wavelet Support Vector Machine-Based Neural Network Metamodel for Structural Reliability Assessment, Computer-Aided Civil and Infrastructure Engineering.2017; 32(4): 344-357.

Khedher

Illán

Górriz

Ramírez

Brahim

Meyer-Baese

. Independent Component Analysis-Support Vector Machine-Based Computer-Aided Diagnosis System for Alzheimerâ€™s with Visual Support, International journal of neural systems. 2017; 27(3): 1650050.

Bustillo

Grzenda

Macukow

. Interpreting tree-based prediction models and their data in machining processes, Integrated Computer-Aided Engineering. 2016; 23: 349-367.

Patil

Sherekar

. Performance analysis of Naive Bayes and J48 classification algorithm for data classification, International Journal of Computer Science and Applications. 2013; 6(2): 256-261.

Quinlan

. C4.5: Programs for Machine Learning, Morgan Kaufmann, San Francisco, 1993.

10.

. Random decision forests, In: Proceedings of the Third International Conference on Document Analysis and Recognition, 1995; 1: 278-282.

11.

Breiman

. Random Forests, Machine Learning. 2001; 45(1): 5-32.

12.

Ahmadlou

Adeli

. Enhanced probabilistic neural network with local decision circles: A robust classifier, Integrated Computer-Aided Engineering. 2010; 17: 197-210.

13.

Zeinali

Story

. Competitive probabilistic neural network, Integrated Computer-Aided Engineering. 2017; 24: 105-118.

14.

Rafiei

Adeli

. A new neural dynamic classification algorithm, IEEE transactions on neural networks and learning systems. 2017; 28(12): 3074-3083.

15.

Rafiei

Adeli

. NEEWS: a novel earthquake early warning model using neural dynamic classification and neural dynamic optimization, Soil Dynamics and Earthquake Engineering. 2017; 100: 417-427.

16.

Ortega-Zamorano

Jerez

Gómez

Franco

. Layer multiplexing FPGA implementation for deep back-propagation learning, Integrated Computer-Aided Engineering. 2017; 24: 171-185.

17.

Widodo

Yang

. Support Vector Machine in Machine Condition Monitoring and Fault Diagnosis, Mechanical Systems and Signal Processing. 2007; 21.

18.

Yin

Hou

. Recent advances on SVM based fault diagnosis and process monitoring in complicated industrial processes, Neurocomputing. 2016; 174: 643-650.

19.

Peng

Wang

Shi

Pérez-Jiménez

Riscos-Núñez

. Fault diagnosis of power systems using fuzzy tissue-like P systems, Integrated Computer-Aided Engineering. 2017; 24: 401-411.

20.

Grzenda

Bustillo

Quintana

Ciurana

. Improvement of surface roughness models for face milling operations through dimensionality reduction, Integrated Computer-Aided Engineering. 2012; 19: 179-197.

21.

Santofimia

del Toro

Roncero-Sánchez

Moya

Martinez

Lopez

. A qualitative agent-based approach to power quality monitoring and diagnosis, Integrated Computer-Aided Engineering. 2010; 17: 305-319.

22.

Roveri

Trovò

. An Ensemble Approach for Cognitive Fault Detection and Isolation in Sensor Networkss, International journal of neural systems. 2017; 27(3): 1650047.

23.

Sharma

Sugumaran

Kumar

Amarnath

. A comparative study of naïve Bayes classifier and Bayes net classifier for fault diagnosis of roller bearing using sound signal, International Journal of Decision Support Systems. 2015; 1(1): 115-129.

24.

Tafreshi

Sassani

Ahmadi

Dumont

. Local discriminant bases in machine fault diagnosis using vibration signals, Integrated Computer-Aided Engineering. 2005; 12: 147-158.

25.

Yang

Han

. Random forests classifier for machine fault diagnosis, Journal of mechanical science and technology. 2008; 22(9): 1716-1725.

26.

Cerrada

Zurita

Cabrera

SÃ¡nchez

ArtÃ©s

. Fault diagnosis in spur gears based on genetic algorithm and random forest, Mechanical Systems and Signal Processing. 2016; 70–71: 87-103.

27.

Liu

Zeng

Zhang

. LLE for submersible plunger pump fault diagnosis via joint wavelet and SVD approach, Neurocomputing. 2016; 185: 202-211.

28.

Smith

Randall

. Rolling element bearing diagnostics using the Case Western Reserve University data: A benchmark study, Mechanical Systems and Signal Processing. 2015; 64: 100-131.

29.

Vakharia

Gupta

Kankar

. A multiscale permutation entropy based approach to select wavelet for fault diagnosis of ball bearings, Journal of Vibration and Control. 2015; 21(16): 3123-3131.

30.

Tian

Morillo

Azarian

Pecht

. Motor bearing fault detection using spectral kurtosis-based feature extraction coupled with K-nearest neighbor distance analysis, IEEE Transactions on Industrial Electronics. 2016; 63(3): 1793-1803.

31.

Ziaja

Antoniadou

Barszcz

Staszewski

Worden

. Fault detection in rolling element bearings using wavelet-based variance analysis and novelty detection, Journal of Vibration and Control. 2016; 22(2): 396-411.

32.

Chandra

Sekhar

. Fault detection in rotor bearing systems using time frequency techniques, Mechanical Systems and Signal Processing. 2016; 72: 105-133.

33.

Wandekokem

Mendel

Fabris

Valentim

Batista

Varejão

, et al. Diagnosing multiple faults in oil rig motor pumps using support vector machine classifier ensembles, Integrated Computer-Aided Engineering. 2011; 18: 61-74.

34.

Oliveira-Santos

Rauber

Varejão

Martinuzzo

Oliveira

Ribeiro

, et al. Submersible Motor Pump Fault Diagnosis System: A Comparative Study of Classification Methods, In: Tools with Artificial Intelligence (ICTAI), 2016 IEEE 28th International Conference on. IEEE; 2016; p. 415-422.

35.

Rauber

Oliveira-Santos

de Assis Boldt

Rodrigues

Varejao

Ribeiro

. Kernel and random extreme learning machine applied to submersible motor pump fault diagnosis, In: 2017 International Joint Conference on Neural Networks (IJCNN); 2017. p. 3347-3354.

36.

Kuncheva

Bezdek

Duin

RPW

. Decision templates for multiple classifier fusion: an experimental comparison, Pattern Recognition. 2001; 34(2): 299-314.

37.

Kuncheva

. Combining pattern classifiers: methods and algorithms, John Wiley & Sons; 2004.

38.

Dietterich

. Error-Correcting Output Coding. In: Machine Learning Proceedings 1995; Proceedings of the Twelfth International Conference on Machine Learning, Tahoe City, California, July 9–12 1995. Morgan Kaufmann; 2016. p. 313.

39.

Breiman

. Bagging Predictors, Machine Learning. 1996; 24(2): 123-140.

40.

Efron

Tibshirani

. An introduction to the bootstrap. CRC presss, 1994.

41.

Japkowicz

Shah

. Evaluating Learning Algorithms: A Classification Perspective. Cambridge University Press, 2011.

42.

Sokolova

Lapalme

. A systematic analysis of performance measures for classification tasks, Information Processing & Management. 2009; 45(4): 427-437.

43.

Nadeau

Bengio

. Inference for the Generalization Error, Machine Learning. 2003; 52(3): 239-281.

44.

Holm

. A Simple Sequentially Rejective Multiple Test Procedure, Scandinavian Journal of Statistics. 1979; 6(2): 65-70.

45.

Witten

Frank

Hall

Pal

. Data Mining: Practical machine learning tools and techniques. Morgan Kaufmann, 2016.

46.

Acharya

Hagiwara

Tan

Adeli

. Deep convolutional neural network for the automated detection and diagnosis of seizure using EEG signals. Computers in Biology and Medicine, 2017.

47.

Rafiei

Khushefati

Demirboga

Adeli

. Supervised Deep Restricted Boltzmann Machine for Estimation of Concrete, ACI Materials Journal. 2017; 114(2): 237-244.

Combining classifiers with decision templates for automatic fault diagnosis of electrical submersible pumps

Abstract

Keywords

1. Introduction

2.1 Feature extraction

2.3 Decision Templates

3.1 Benchmark data set

3.2 Performance evaluation

Table 1 Hyperparameters used for each classifier architecture

Table 2 Experimental methodology for robustness analysis

4.1 Comparison of standalone classifiers and bagging ensembles

Footnotes

Acknowledgments

References

Table 1
Hyperparameters used for each classifier architecture

Table 2
Experimental methodology for robustness analysis