Sequential prediction for imbalanced data stream via weighted OS-ELM and dynamic GAN

Abstract

This paper focuses on the problem of sequential prediction for imbalanced data streams. A novel hybrid algorithm called Weighted OS-ELM and Dynamic Generative Adversarial Nets (GAN-WOSELM) is presented for handling this issue. In the data reconstruction stage, GAN is utilised for generating lifelike minority class samples to equilibrate the data distribution. Then a principal component score threshold helps judge unusual data. In the model update stage, the new constructive ELM is applied to forecast time-varying data chunk. After concerning fitting accuracy and data change, the analytical relationship between the new weight and the shifting imbalance ratio is determined. Therefore, the GAN-WOSELM can update weight quantificationally, and it avoids interactive parameter optimization. According to the suitable weight for the arriving chunk, the proposed method is able to perceive the changeable data distribution and do the adaptation by itself, thus building a reliable model with a low fitting deviation. Numerical experiments are conducted on four different kinds of UCI datasets. The results demonstrate that the proposed algorithm not only has better generalisation performance but also provides higher numerical stability.

Keywords

Sequential prediction imbalanced data weighted OS-ELM dynamic GAN

1. Introduction

Time series with imbalanced labels is universally presented in real world applications, such as network intrusion detection [1], traffic flow analysis [2], handwritten character recognition [3] and spam filtering [4]. Different from batch learning processes, sequential prediction involves learning from chunk-by-chunk data without reprocessing observed examples [5]. The unbalanced sequence leads to a varying imbalanced ratio (IR) and unknown sample size. Particularly in shifty environments, forecasting model cannot always be rebuilt when the new data chunk arrives. If the update strategy cannot accommodate fast-changing test data, the original predictive model may become inaccurate.

As we all known, there are two main ways to solve the imbalanced learning problem [6]. One technique is applied at the data level, the other at the algorithm level. For the first type, the unequal cost of different class sample errors is taken into consideration. Usually, the cost of erroneous classification from minority samples is greater than the majority one [7]. Multiple resampling technologies are put forward to generate similar examples so as to equilibrate the class frequency distribution [8, 9]. If the new datasets achieve approximate equilibrium, the normal sequential learning algorithm will have a role in prediction. For the second method, multiple individual methods are integrated to alleviate the whole bias [10, 11]. A strong learner emerges. In view of the sequential prediction, the ensemble learning methods are created to deal with class imbalance dynamic data streams [12, 13]. A representative classification method for sequential prediction of imbalanced dataset is the weighted online sequential extreme learning machine (WOS-ELM) [14]. It is a general learning method that alleviates the class imbalance problem in both one-by-one and chunk-by-chunk. These two types of methods strive to solve the imbalanced problem in the variable environment. New dataset or changeable weight will help to improve the generalization performance.

In this paper, we present a novel hybrid method with Weighted OS-ELM and Dynamic Generative Adversarial Nets (GAN-WOSELM). Similar to SOMTE-OSELM [15], GAN-WOSELM is suitable for sequential prediction of imbalanced time series. In contrast to the SOMTE-OSELM, the proposed method can optimise the OS-ELM structure by adjusting for the changes in imbalance ratio. The model passes helpful information from the current chunk to the arriving one, so the proposed method avoids the rebuilding in the online learning process. In addition to the above contribution, the proposed method will produce new data integrally according to game theory. The new sample is beneficial to balance category of varying data streams. As a supplementary note, a positive label represents the minority class and a negative label describes the majority class in the rest of this paper.

The remainder of this paper is organised as follows. In Section 2, the related work preliminaries involving hybrid classification method and online prediction are described. In Section 3, a new weighted OS-ELM with a Dynamic GAN is introduced. Experimental results and a discussion are presented in Section 4. Finally, Section 5 concludes the paper and suggests the future work.

2. Related work

In this section, we first review hybrid classification method, and then discuss the online learning frame for data stream. These preliminaries give theoretical support.

2.1 Hybrid classification methods

Hybrid classification methods better inherit the merits of the two types of methods [16, 17]. The meta-cognitive algorithms, whose parameters acquire global optimisation using a biological intelligence approach, have widely used to select suitable learning strategies for both stationary and non-stationary environments [18, 19, 20, 21]. That means these learning models need to tune parameters repeatedly. In order to figure out the issue, a computationally efficient ensemble algorithm is achieved for sequential imbalance learning [22]. And a regularised proximal learning framework has been used to solve the composite optimisation problem [23]. These methods tend to better cope with the concept drift problem. However, many hybrid methods rely on multiple parameters. It is troublesome to obtain the appropriate weight before the next chunk appears. When the data distribution of the time series changes continually, the model can hardly execute well or even become useless. Particularly data with high-dimensional features, correlation analysis is crucial and the dimension reduction method cannot be neglected [24]. Although all the above hybrid methods can obtain an adaptive model, the neighbouring data cannot guarantee the validity of the generated data.

2.2 Online prediction

The online learning frame updates the original model according to the predicted deviations [25]. This frame has to address high-dimensionality sequence that involves only a fixed number of samples [26]. Then to break through the storage limitation, the Projectron algorithm guarantee the forecasting boundary [27]. The intrinsic characteristics between feature and label are a major concern, and the representative technique for fast nonlinear mapping is the extreme learning machine (ELM) [28]. As a fast and competitive machine learning technique, the extreme learning machine (ELM) calculates output weight analytically [29]. Due to the advantages of complete theoretical system and easy hardware implementation, it has become feasible for real-time prediction. After the new data chunk arrives, forecasting model usually updates the structure. For example, the convex incremental ELM (CI-ELM) avoids overfitting causing by enlarging network scale [30]. The error minimized ELM (EM-ELM) cut off redundancy connecting branch towards the predictive target [31]. D-ELM better adapts the network size and the target function is extracted effectively [32]. It not only has reliable accuracy, but also befits varying task types. While these methods depend on the original imbalance ratio, when the distribution characteristics of a dataset change frequently, model change is delayed. So an automatic well-matched model is essential for guaranteeing the generalisation error of global minima.

3. The proposed method

In this section, we introduce the proposed GAN-WOSELM in detail. GAN-WOSELM establishes a relationship between the learning model and the shifting imbalance ratio. It helps the novel ELM fit arriving data. The proposed hybrid algorithm includes two phases: data reconstruction and model update. They all greatly affect the generalisation performance of sequential class imbalance learning problems.

3.1 Data reconstruction phase

First of all, data generation is necessary to obtain the balanced sample. Generative adversarial Nets (GANs) are multiple neural networks which constitute a new framework for estimating generative models [33]. A continuous sequence is created by similar semantic features and dynamic GAN. Figure 1 illustrates the framework of the data generation.

Figure 1.

Framework of data generation.

Considering the current sample set $\textit{Set}_{k}=\{(x_{i},y_{i})\}_{i=1}^{N_{k}}$ , the whole feature of minority samples in this fragment is expressed as ${\bf{X}}_{k}^{+}$ . Same size noise is denoted by ${\bf{Z}}$ . To maintain the semantic integrality of the new samples, the generated number $\textit{num}_{k}$ is adopted as a measure for the stop threshold, and is defined as follows:

$\displaystyle\textit{num}_{k}=\left\lfloor{\frac{{N_{k}^{-}}}{{N_{k}^{+}}}}% \right\rfloor,$ (1)

where $N_{k}^{+}$ and $N_{k}^{-}$ represent the sizes of the $k^{th}$ positive and negative classes. $\left\lfloor\cdot\right\rfloor$ denotes rounding down. What is more, in order to reduce overfitting, the unusual new data must be wiped off. For the original training data chunk, we calculate the maximum spatial distance according to the principal component analysis. After standardisation, the correlation coefficient matrix of the current chunk is calculated as follows:

$\displaystyle{\bf{R}}_{k}=\bf{X}_{k}^{\rm T}{\bf{X}}_{k}/\left({N_{k}-1}\right).$ (2)

We arrange eigenvalues from large to small as $\lambda_{1},\lambda_{2},\ldots\lambda_{N_{k}}$ . The principal component scores can be calculated as

$\displaystyle p_{j}=\sum\limits_{i=1}^{N_{k}}\tilde{x}_{i}\bm{\eta}_{j}^{\rm T% },j=1,2,\ldots N_{k}.$ (3)

where $\tilde{x}_{i}$ is the $i^{th}$ sample of the standard feature matrix. The two biggest eigenvalues $\lambda_{1}$ , $\lambda_{2}$ are selected to decide the spatial distribution. $\left|{p_{1}}\right|_{\max}$ and $\left|{p_{2}}\right|_{\max}$ express the maximum absolute value of the two biggest principal component scores. Then the proposed method oversamples the positive class using GAN. The optimal object is to minimise the expectation of cross entropy [34], which is formulated as:

$\displaystyle\mathop{\min}\limits_{G}\mathop{\max}\limits_{D}V(D,G)=E_{{\bf{X}% }_{k}^{+}\sim p_{\textit{data}}({\bf{X}}_{k}^{+})}\left[{\log D({\bf{X}}_{k}^{% +})}\right]+E_{Z\sim p_{\bf{Z}}({\bf{Z}})}\left[{\log(1-D(G({\bf{Z}})))}\right],$ (4)

where $p_{\textit{data}}({\bf{X}}_{k}^{+})$ is the distribution of the generator, and $p_{\bf{Z}}({\bf{Z}})$ is the distribution of noise. In order to solve the optimum function, the training process of the discriminator and generator is iterative. The discriminator $D$ is updated by an ascending gradient as follows:

$\displaystyle\nabla_{\theta_{d}}\frac{1}{{{N}_{k}^{+}}}\sum\limits_{i=1}^{{N}_% {k}^{+}}{\left[{\log D(x_{i})+\log(1-D(G(z_{i})))}\right]}.$ (5)

The generator $G$ is updated by a descending gradient, as follows:

$\displaystyle\nabla_{\theta_{g}}\frac{1}{{N_{k}^{+}}}\sum\limits_{i=1}^{N_{k}^% {+}}{\left[{\log(1-D(G(z_{i})))}\right]},$ (6)

where $\theta_{d}$ is the parameter of discriminator $D$ and $\theta_{g}$ is the parameter of generator $G$ . Symbols $x_{i}$ and $z_{i}$ represent the samples in ${\bf{X}}_{k}^{+}$ and ${\bf{Z}}$ , respectively. After the joint learning process, if $G$ recovers the data chunk distribution and $D$ closes to 0.5, the transient generated feature chunk ${\bf{X^{\prime}}}_{k}^{+}$ is confirmed. Considering the two largest principal component scores as $\left|{p^{\prime}_{1}}\right|_{\max}$ and $\left|{p^{\prime}_{2}}\right|_{\max}$ , if $\left|{p^{\prime}_{1}}\right|_{\max}<\left|{p_{1}}\right|_{\max}$ and $\left|{p^{\prime}_{2}}\right|_{\max}<\left|{p_{2}}\right|_{\max}$ , the new feature chunk is assigned ${\bf{X^{\prime}}}_{k}^{+}={\bf{X}}_{k,\textit{temp}}^{+}$ . The generated processes continue until the number of iterations $\textit{num}_{\textit{ite}}$ is equal to $\textit{num}_{k}$ .

After the cyclic adversarial process, we obtain the new training sample set, and the new imbalance ratio is represented as

$\displaystyle IR^{\prime}_{k}=\frac{{\left\lfloor{\frac{{N_{k}^{-}}}{{N_{k}^{+% }}}}\right\rfloor\cdot N_{k}^{+}}}{{N_{k}^{-}}}.$ (7)

Merging the new minority sample set and majority sample set, the number of data increases from $N_{k}$ to $N^{\prime}_{k}$ .

3.2 Model update phase

Expanded WOS-ELM, the proposed method adjusts the weights according to the changeable imbalance ratio and drift sample size. The nonlinear mapping function is chosen as a sigmoid function. It maps the variable to $\left[{0,1}\right]$ and reflects the standardised dataset accurately. At the $\left({k+1}\right)^{th}$ chunk, the change in the imbalance ratio is calculated as follows:

$\displaystyle\Delta IR_{k+1}=IR^{\prime}_{k+1}-IR_{k}=\frac{{\left\lfloor{% \frac{{N_{k+1}^{-}}}{{N_{k+1}^{+}}}}\right\rfloor\cdot N_{k+1}^{+}}}{{N_{k+1}^% {-}}}-\frac{{N_{k}^{-}}}{{N_{k}^{+}}}.$ (8)

The change of imbalance ratio is composed of two main parts. One part comes from the operation of generating minority class samples. The other part is from the varying data distribution between the adjacent chunks. Larger chunks are associated with a smaller change in the imbalance ratio, and vice versa. According to the ELM structure, the weights between the input layer and the hidden layer are randomly selected, and the hidden learning parameters are calculated according to the Moore-Penrose generalised inverse [35]. The basic mapping relation can be expressed as

$\displaystyle y_{i}=\sum\limits_{p=1}^{L}{\beta_{p}\phi(a_{p}\cdot x_{i}+b_{p}% )},i=1,\ldots,N,$ (9)

where $\phi(x)$ is the activation function, $L$ is the number of hidden nodes. $a_{p}$ and $b_{p}$ are the input weight vector and offset, respectively. $\beta_{p}$ is the learning parameter connecting the $p$ th hidden node to the output node. In a more concise expression, Eq. (1) can be written in a matrix form as follows:

$\displaystyle{\bf{Y=H\beta}},$ (10)

where

$\displaystyle{\bf{H}}=\left[{\begin{array}[]{*{20}c}{\phi(a_{1}\cdot x_{1}+b_{% 1})}&\ldots&{\phi(a_{L}\cdot x_{1}+b_{L})}\\ \vdots&\ldots&\vdots\\ {\phi(a_{1}\cdot x_{N}+b_{1})}&\ldots&{\phi(a_{L}\cdot x_{N}+b_{L})}\\ \end{array}}\right]_{N\times L},$

${\bf{\beta}}=[\beta_{1},\beta_{2},\ldots,\beta_{L}]^{\text{T}}$ and ${\bf{Y}}=\left[{y_{1},y_{2},\ldots,y_{N}}\right]^{\text{T}}$ . It is obvious that the solution process for the ELM need no iteration, which is different from traditional gradient descent method.

At the original learning phase, added weight factor, the output weight can be calculated as

$\displaystyle{\bf{\beta}}_{0}={\bf{K}}_{0}^{-1}{\bf{H}}_{0}^{\text{T}}{\bf{W}}% _{0}{\bf{Y}}_{0},$ (11)

At the original learning phase, ${\bf{K}}_{0}={\bf{H}}_{0}^{\text{T}}{\bf{W}}_{0}{\bf{H}}_{0}$ and the ELM weights are initialised as follows:

$\displaystyle w_{0}=\left\{{\begin{array}[]{ll}w_{0}^{+}=\frac{{N_{0}^{+}}}{{N% _{0}^{-}}},&if\,y_{0}=+1\\ w_{0}^{-}=1,&if\,y_{0}=-1\\ \end{array}}\right.,$ (12)

where $w_{0}^{+}$ is the weight of the minority class samples and $w_{0}^{-}$ is the weight of the majority class samples. All weights in the current chunk are constituted in the form of diagonal matrices as:

$\displaystyle W_{0}=\left({\begin{array}[]{*{20}c}{w_{0}^{+}}&&&\\ &{w_{0}^{+}}&&\\ &&\ddots&\\ &&&{w_{0}^{-}}\\ \end{array}}\right).$ (13)

With increasing size of the dataset, the element number of the weight set goes up. Then the weight at the $\left({k+1}\right)^{th}$ chunk can be expressed as follows:

$\displaystyle{\bf{W}}_{k+1}=\left({1-\Delta IR_{k+1}}\right){\bf{W}}_{k}+\frac% {{\Delta IR_{k+1}}}{{N_{k+1}}},$ (14)

where $W_{k}$ is the weight set at the $k^{th}$ chunk, and $N_{k+1}$ is the size of the next sliding window. When the new set of samples $\{(x_{q},y_{q})\}_{q=N_{k}+1}^{N_{k+1}}$ arrives, the target equation is described as follows:

$\displaystyle\overline{\textit{error}}=\left\|{\left[{\begin{array}[]{*{20}c}{% {\bf{H}}_{k}}\\ {{\bf{H}}_{k+1}}\\ \end{array}}\right]{\bm{\beta}}_{k+1}-\left[{\begin{array}[]{*{20}c}{{\bf{Y}}_% {k}}\\ {{\bf{Y}}_{k+1}}\\ \end{array}}\right]}\right\|_{\min}.$ (15)

Besides, the Moore-Penrose generalised inverse and matrix transformation helps simplify the target equation. Hence, the output weights can be calculated as:

$\displaystyle{\bm{\beta}}_{k+1}={\bf{K}}_{k+1}^{-1}\left[{\begin{array}[]{*{20% }c}{{\bf{H}}_{k}}\\ {{\bf{H}}_{k+1}}\\ \end{array}}\right]^{\text{T}}\left[{\begin{array}[]{*{20}c}{{\bf{W}}_{k}}&{% \bf{0}}\\ {\bf{0}}&{{\bf{W}}_{k+1}}\\ \end{array}}\right]\left[{\begin{array}[]{*{20}c}{{\bf{Y}}_{k}}\\ {{\bf{Y}}_{k+1}}\\ \end{array}}\right],$ (16)

where

$\displaystyle{\bf{K}}_{k+1}=\left[{\begin{array}[]{*{20}c}{{\bf{H}}_{k}}\\ {{\bf{H}}_{k+1}}\\ \end{array}}\right]^{\text{T}}\left[{\begin{array}[]{*{20}c}{{\bf{W}}_{k}}&{% \bf{0}}\\ {\bf{0}}&{{\bf{W}}_{k+1}}\\ \end{array}}\right]\left[{\begin{array}[]{*{20}c}{{\bf{H}}_{k}}\\ {{\bf{H}}_{k+1}}\\ \end{array}}\right].$

After the simplified processes, ${\bf{K}}_{k+1}$ can be deduced as the following equation:

$\displaystyle{\bf{K}}_{k+1}={\bf{H}}_{k}^{\text{T}}{\bf{W}}_{k}{\bf{H}}_{k}+{% \bf{H}}_{k+1}^{\text{T}}\left[{\left({1-\Delta IR_{k+1}}\right){\bf{W}}_{k}+% \frac{{\Delta IR_{k+1}}}{{N_{k+1}}}}\right]{\bf{H}}_{k+1}={\bf{K}}_{k}+{\bf{H}% }_{k+1}^{\text{T}}{\bf{W}}_{k}{\bf{H}}_{k+1}+{\bf{H}}_{k+1}^{\text{T}}\left[{% \Delta IR_{k+1}\left({\frac{1}{{N_{k+1}}}-{\bf{W}}_{k}}\right)}\right]{\bf{H}}% _{k+1}.$ (17)

Taking the relationship between $\beta_{k}$ and $\beta_{k+1}$ into consideration, the model update function can be calculated as:

$\displaystyle{\bm{\beta}}_{k+1}={\bm{\beta}}_{k}+{\bf{K}}_{k+1}^{-1}{\bf{H}}_{% k+1}^{\text{T}}\left[{\left({1-\Delta IR_{k+1}}\right){\bf{W}}_{k}+\frac{{% \Delta IR_{k+1}}}{{N_{k+1}}}}\right]({\bf{Y}}_{k+1}-{\bf{H}}_{k+1}{\bm{\beta}}% _{k})={\bm{\beta}}_{k}+{\bf{K}}_{k+1}^{-1}{\bf{H}}_{k+1}^{\text{T}}{\bf{W}}_{k% }({\bf{Y}}_{k+1}-{\bf{H}}_{k+1}{\bm{\beta}}_{k}){}+{\bf{K}}_{k+1}^{-1}{\bf{H}}% _{k+1}^{\text{T}}\left[{\Delta IR_{k+1}\left({\frac{1}{{N_{k+1}}}-{\bf{W}}_{k}% }\right)}\right]({\bf{Y}}_{k+1}-{\bf{H}}_{k+1}{\bm{\beta}}_{k}).$ (18)

When the new samples arrive, the method alters the temporary parameters according to the variation of the imbalance ratio. The model can be updated quantificationally instead of by trial and error. Moreover, in the proposed GAN-WOSELM, the new data is created chunk by chunk, thus reducing the probability of overfitting. So, GAN-WOSELM is better suited for real-time online applications. Taking into account the characteristics of the learning procedure, the whole GAN-WOSELM algorithm is presented in Algorithm 1.

Algorithm 1: Learning Procedure of the GAN-WOSELM
1: Input the training datasets $T_{\textit{train}}$ and express the whole feature as ${\bf{X}}_{k}^{+}$
2: Calculate the stop threshold $\textit{num}_{k}$ and principal component scores
3: Select the two biggest principal component scores $\left\|{p_{1}}\right\|_{\max}$ , $\left\|{p_{2}}\right\|_{\max}$
4: While $\left({\textit{num}_{\textit{ite}}<\textit{num}_{k}}\right)$ do
5: Initialize ${\bf{Z}}$ and formulate the optimum criterion
6: $\mathop{\min}\limits_{G}\mathop{\max}\limits_{D}V(D,G)=E_{{\bf{X}}_{k}^{+}\sim p% _{\textit{data}}({\bf{X}}_{k}^{+})}\left[{\log D({\bf{X}}_{k}^{+})}\right]+E_{% Z\sim p_{\bf{Z}}({\bf{Z}})}\left[{\log(1-D(G({\bf{Z}})))}\right]$
7: Update the discriminator $D$ and generator $G$ , respectively
8: If the recognition rate of $D$ closes to 0.5
9: Confirm transient generated feature chunk ${\bf{X}}_{k,\textit{temp}}^{+}$
10: Calculate $\left\|{p^{\prime}_{1}}\right\|_{\max}$ and $\left\|{p^{\prime}_{2}}\right\|_{\max}$
11: end
12: if $\left({\left\|{p^{\prime}_{1}}\right\|_{\max}<\left\|{p_{1}}\right\|_{\max}}\right% )\&\&\left({\left\|{p^{\prime}_{2}}\right\|_{\max}<\left\|{p_{2}}\right\|_{\max}}\right)$
13: ${\bf{X^{\prime}}}_{k}^{+}={\bf{X}}_{k,\textit{temp}}^{+}$
14: execute next iteration
15: else
16: Rebuild the current sequence fragment
17: end
18: end while
19: Calculate the change of imbalance ratio $\Delta IR_{k+1}$
20: $\Delta IR_{k+1}=IR^{\prime}_{k+1}-IR_{k}=\frac{{\left\lfloor{\frac{{N_{k+1}^{-% }}}{{N_{k+1}^{+}}}}\right\rfloor\cdot N_{k+1}^{+}}}{{N_{k+1}^{-}}}-\frac{{N_{k% }^{-}}}{{N_{k}^{+}}}$
21: Calculate the new weight according to the recursively
22: ${\bf{W}}_{k+1}=\left({1-\Delta IR_{k+1}}\right){\bf{W}}_{k}+\frac{{\Delta IR_{% k+1}}}{{N_{k+1}}}$
23: ${\bf{K}}_{k+1}={\bf{K}}_{k}{+\bf{H}}_{k+1}^{\text{T}}{\bf{W}}_{k}{\bf{H}}_{k+1% }+{\bf{H}}_{k+1}^{\text{T}}\left[{\Delta IR_{k+1}\left({\frac{1}{{N_{k+1}}}-{% \bf{W}}_{k}}\right)}\right]{\bf{H}}_{k+1}$
24: Calculate ${\bm{\beta}}_{k+1}$
25: Return: ${\bm{\beta}}_{k+1}$ and the new input weights and offsets

Remark A: Regarding the two types of neural network: GAN and ELM. We express the node number of one layer as $L_{G}$ and $L_{E}$ . The number of games is indicated by $P$ . The computational complexity of original ELM is $O\left({N_{k}L_{E}^{3}}\right)$ . The computational process of ${\bf{K}}_{k+1}$ requires $O\left({2N_{k}L_{E}^{2}}\right)$ multiplications. ${\bf{\beta}}_{k}$ need no calculation in the sequential learning process. So the novel WOS-ELM is $O\left({2N_{k}L_{E}^{2}\cdot L_{E}\cdot N_{k}\cdot\left({N_{k}+L_{E}}\right)}\right)$ , which can be simplified as $O\left({N_{k}^{3}L_{E}^{3}+N_{k}^{2}L_{E}^{4}}\right)$ . The proposed method also involves $O\left({N_{k}^{2}L_{G}^{4}P}\right)$ from GAN and $O\left({N_{k}^{3}}\right)$ from PCA. So the resulting multiplications are in order of

$\displaystyle O\left({N_{k}^{2}L_{G}^{4}P+N_{k}^{3}+N_{k}^{3}L_{E}^{3}+N_{k}^{% 2}L_{E}^{4}}\right).$ (19)

In contrast, the computational complexity for [15] concentrates mainly on the deviation calculation and online learning process. The computational complexity is $O\left({N_{k}Q+N_{k}^{2}L_{E}^{3}+N_{k}^{2}L_{E}^{5}}\right)$ , where $Q$ is the number of nearest neighbours. The hidden node number of neural networks is much smaller than the sample number, so the computational complexity of GAN-WOSELM usually depends on the length of chunk. Although the computational complexity for the proposed algorithm is greater than that in [15], it is worth noting that the proposed method does the adaptation by itself and reduces overfitting.

4. Experiment results and discussion

This section describes the experiment. Firstly, we list the datasets and evaluation criteria. Then, the results of the different methodologies are compared. GAN-WOSELM is compared with other classical online classification methods. The theoretical demonstration is shown in the last section.

4.1 Datasets description

The GAN-WOSELM method proposed in this paper has mostly been applied to the online prediction of imbalanced data streams. In order to evaluate the effectiveness of the proposed method, there are four UCI datasets with various dimensionalities to be selected [36]. Table 1 presents the specifications of each dataset.

Table 1
Specifications of the datasets

Datasets	Feature	Minority class	Majority class	Minority	Majority	Imbalance	Training	Testing
						ratio	samples	samples
Statlog	18	Class of ‘van’	All other classes	199	647	0.307	94	752
Pima	8	Class of ‘1’	Class of ‘0’	169	547	0.309	170	546
Cardiotocography	22	Class of ‘2’, ‘3’	All other classes	349	785	0.445	200	934
Pen-Based	17	Class of ‘8’, ‘9’	All other classes	259	1233	0.210	300	1192

Aiming at the various prediction problems, time series data must be normalised as follows:

$\displaystyle t^{\prime}=\frac{{t-t_{\min}}}{{t_{\max}-t_{\min}}},$ (20)

where $t$ is an input numeric value. $t_{\min}$ and $t_{\max}$ are the minimum and maximum of the selected feature data, respectively.

4.2 Evaluation criteria

The evaluation criteria of algorithm performance consist of multiple metrics. For dealing with two-class classification problems, the minority class is labelled as the positive sample and majority class are labelled as negative. The essential classifier result is to form a confusion matrix as shown in Table 2.

Table 2
Confusion matrix

Datasets	Predicted result
	Positive	Negative
Positive	TP (True Positive)	FN (False Negative)
Negative	FP (False Positive)	TN (True Negative)

For imbalanced sequences, the overall error is no longer an appropriate performance measure. G-mean and F-measure evaluate generalisation performance better [37]. They are shown as the following two equations.

$\displaystyle\textit{G-mean}=\sqrt{\frac{{TP}}{{TP+FN}}\times\frac{{TN}}{{TN+% FP}}},$ (21) $\displaystyle\textit{F-measure}=\frac{2*\textit{Precision}*\textit{Recall}}{% \textit{Precision}+\textit{Recall}},$ (22)

where

$\displaystyle\textit{Precision}=\frac{TP}{TP+FP},\textit{Recall}=\frac{TP}{TP+% FN}.$

Then, the ROC (receiver operating characteristics) graph gives the overall performance assessment criteria [38]. The ROC graph is obtained by plotting the FPR on the x-axis and the TPR on the y-axis. The FPR is called the false positive rate, which is calculated according to the confusion matrix. The TPR is the true positive rate, whose definition is equal to Recall. They are shown in detail in the following equations.

$\displaystyle\textit{FPR}=\frac{{FP}}{{FP+TN}},$ (23) $\displaystyle\textit{TPR}=\textit{Recall}=\frac{{TP}}{{TP+FN}}.$ (24)

Also, the area under the ROC curve (AUC), is a popular measure for imbalanced data prediction [39].

Figure 2.

Spatial distribution (a), ROC curve (b), G-mean (c) and F-measure (d) of the four methods for Statlog.

Figure 3.

Spatial distribution (a), ROC curve (b), G-mean (c) and F-measure (d) of the four methods for Pima.

Figure 4.

Spatial distribution (a), ROC curve (b), G-mean (c) and F-measure (d) of the four methods for Cardiotocography.

Figure 5.

Spatial distribution (a), ROC curve (b), G-mean (c) and F-measure (d) of the four methods for Pen-Based.

4.3 Experiment results

In order to see how the GAN design influences the performances of the proposed system, we firstly try the different GAN architectures. The hidden node number of generator is 100. For imbalanced data stream forecasts, the OS-ELM, WOS-ELM and SMOTE-OSELM methods are chosen for comparison. The chunk size is chosen as 50 for the four datasets. In the first experiment, we show the prediction result of Dataset 1. Figure 2 shows the evaluation effect of an execution round for the Statlog.

In Fig. 2a, the spatial distribution is shown according to the first two principal components. The blue circle describes the correctly classified samples and the red spots indicate the misclassified ones. Then, we show a black line describing the proposed method and coloured lines with different shapes, which are the contrastive methods. The averaged ROC curves of OS-ELM, WOS-ELM, SMOTE-OSELM and GAN-WOSELM for online prediction are shown in Fig. 2b. The AUC of our approach is 0.9087, which is larger than the others. Figure 2c and d compare the accumulated G-means and F-measures. Obviously, our approach has better generalisation performance compared with the other constructive ELM. The F-measure is almost above 0.95, which is almost closed. It is proven that the proposed method has better capacity of adjustment. This advantage generates a more suitable model with the changing data.

In the second experiment, we show the prediction result of Dataset 2. Figure 3 shows the evaluation value of an execution round for the Pima.

From the spatial distribution of Dataset 2 shown in Fig. 3a, we can see that there are many more correct predictions than incorrect ones. In Fig. 3b–d, if the data dimensionality decreases, the SMOTE-OSELM method loses efficacy. The proposed method has better generalisation performance than the OS-ELM and WOS-ELM methods. If the imbalance ratio is extremely low, the traditional weighted ELM usually leads to excessive learning. It is difficult to control the optimal parameters. However, the GAN-WOSELM method changes the model along with the changed imbalance ratio; therefore, the fluctuation of generalisation performance is not volatile.

In the third experiment, we show the prediction results of Dataset 3. Figure 4 shows the forecast effect of an execution round for the cardiotocography dataset.

The spatial distribution of Dataset 3 is shown in Fig. 4a. Almost all results have the correct predictions, especially for the higher principal component scores. As seen in Fig. 4b, all the ROC curves are beyond the diagonal line and the proposed method has the best one. In Fig. 4c and d, the G-means and F-measures of the four methods are high at first. However, as time goes on, the contrastive hybrid method tends to decline obviously. It is proven that the proposed method has better adjustment capacity. This advantage indicates that the model is more suitable for use with changing data. Although the number of features is high, the generated data have similar meaning.

In the last experiment, we show the prediction results for Dataset 4. Figure 5 shows the evaluation effect of an execution round for the Pen-Based dataset.

The spatial distribution of Dataset 4 (Fig. 5a) shows that the proposed algorithm has excellent prediction ability. In Fig. 5b, the proposed method has a better fit than contrastive ELM. Then, in Fig. 5c and d, it is obvious that the proposed method has a higher G-mean and F-measure than the others. In the proposed method, the generalisation error trend is stable. This phenomenon indicates that due to the model having the ability for fast predictive control, the proposed method is more suitable for regular data.

The results for the four datasets processed with the three contrasting methods indicate that the proposed method gives satisfactory results. The prediction accuracy is significantly improved compared to the OS-ELM, WOS-ELM and SMOTE-OSELM methods. In the proposed method, the generalisation error trend is stable and the F-measure is close to one. It is proven that the proposed method deals with the online prediction of imbalanced data streams well. Of course, the game process of the proposed method requires more time than in SMOTE. With the development of hardware resources, computational time in practical applications is becoming shorter.

5. Conclusion

In this paper, a Weighted OS-ELM method with a Dynamic GAN is proposed. Different from the traditional weighted sequential extreme learning machines, the proposed method determines the weight according to changes in the imbalance ratio. By means of the dynamic GAN, the data generation process does not lose information during resampling. The recursive weight is used for judging the whole network structure, which makes the prediction method can get accustomed to the sequential environment. Then, the prediction process can be automated and the model becomes suitable for non-expert users. In contrast with other methods, our analysis shows that the proposed method has the best generalisation performance. In future, our research will look at how to reduce computation complexity during online prediction of unstructured data.

Footnotes

Acknowledgments

The work was supported by the National Key Research Project of China under Grant No. 2016YFB1001304, and the Fundamental Research Funds for Central Universities under Grant No. 2017TD-19. The authors gratefully acknowledge financial support from the Research Centre for Intelligent Signal Identification and Equipment, Jilin Province.

References

Al-Yaseen

W.L.

Othman

Z.A.

and Nazri

M.Z.

, Real-time multi-agent system for an adaptive intrusion detection system, Pattern Recognition Letters 85(1) (2017), 56–64.

Lin

Jiang

Fan

and Wang

, A stacking model for variation prediction of public bicycle traffic flow, Intelligent Data Analysis 22(4) (2018), 911–933.

Reznakova

Tencer

Plamondon

et al., Forgetting of unused classes in missing data environment using automatically generated data: application to on-line handwritten gesture command recognition, Pattern Recognition 72 (2017), 355–367.

Almeida

T.A.

Silva

T.P.

Santos

et al., Text normalization and semantic indexing to enhance Instant Messaging and SMS spam filtering, Knowledge-Based Systems 108(9) (2016), 25–32.

Wang

Minku

L.L.

and Yao

, Resampling-based ensemble methods for online class imbalance learning, IEEE Transactions on Knowledge and Data Engineering 27(5) (2015), 1356–1368.

and Garcia

E.A.

, Learning from imbalanced data, IEEE Transactions on Knowledge and Data Engineering 21(9) (2009), 1263–1284.

Zhou

Z.H.

and Liu

X.Y.

, Training cost-sensitive neural networks with methods addressing the class imbalance problem, IEEE Transactions on Knowledge and Data Engineering 18(1) (2006), 63–77.

Chawla

N.V.

Bowyer

K.W.

Hall

L.O.

et al., SMOTE: synthetic minority over-sampling technique, Journal of Artificial Intelligence Research, 16(2002), 321–357.

Cao

Zhao

and Osmar

, Hybrid probabilistic sampling with random subspace for imbalanced data learning, Intelligent Data Analysis 18(6) (2014), 1089–1108.

10.

Kong

et al., Boosting weighted ELM for imbalanced learning, Neurocomputing 128 (2014), 15–21.

11.

Blaszczynski

and Stefanowski

, Neighbourhood sampling in bagging for imbalanced data, Neurocomputing 150 (2015), 529–542.

12.

Wong

S.Y.

Yap

K.S.

Yap

H.J.

et al., A truly online learning algorithm using hybrid fuzzy ARTMAP and online extreme learning machine for pattern classification, Neural Processing Letters 42(3) (2015), 585–602.

13.

Wang

and Pineau

, Online bagging and boosting for imbalanced data streams, IEEE Transactions on Knowledge and Data Engineering 28(12) (2016), 3353–3366.

14.

Mirza

Lin

and Toh

K.A.

, Weighted online sequential extreme learning machine for class imbalance learning, Neural Processing Letters 38(3) (2013), 465–486.

15.

Gong

and Gu

, A novel SMOTE-based classification approach to online data imbalance problem, Mathematical Problems in Engineering (2016), 1–14.

16.

Guo

Zhou

C.A.

She

and Xu

, Ensemble based on feature projection and under-sampling for imbalanced learning, Intelligent Data Analysis 22(5) (2018), 959–980.

17.

Han

Zhang

and Xu

, Multivariate chaotic time series prediction based on ELM-PLSR and hybrid variable selection algorithm, Neural Processing Letters 46(2) (2017), 705–717.

18.

Mirza

and Lin

, Meta-cognitive online sequential extreme learning machine for imbalanced and concept-drifting data classification, Neural Networks 80 (2016), 79–94.

19.

Wang

, A generalized ellipsoidal basis function based online self-constructing fuzzy neural network, Neural Processing Letters 34(1) (2011), 13–37.

20.

Sun

Yang

et al., ODOC-ELM: optimal decision outputs compensation-based extreme learning machine for classifying imbalanced data, Knowledge-Based Systems 92(2016), 55–70.

21.

Savitha

Suresh

and Kim

H.J.

, A meta-cognitive learning algorithm for an Extreme Learning Machine classifier, Cognitive Computation 6(2) (2014), 253–263.

22.

Mirza

Lin

and Liu

, Ensemble of subset online sequential extreme learning machine for class imbalance and concept drift, Neurocomputing 149 (2015), 316–329.

23.

Maurya

C.K.

Toshniwal

and Venkoparao

G.V.

, Online sparse class imbalance learning on big data, Neurocomputing 216 (2016), 250–260.

24.

Juarez-Castillo

Acosta-Mesa

H.G.

et al., A feature selection method based on a neighborhood approach for contending with functional and anatomical variability in fMRI group analysis of cognitive states, Intelligent Data Analysis 21(3) (2017), 661–677.

25.

Zhao

and Hoi

C.H.

, Online passive-aggressive active learning, Machine Learning 103(2) (2016), 141–183.

26.

Wang

Zhao

Hoi

C.H.

et al., Online feature selection and its applications, IEEE Transactions on Knowledge and Data Engineering 26(3) (2014), 698–710.

27.

Orabona

Joseph

and Barbara

, Bounded kernel-based online learning, Journal of Machine Learning Research 10(11) (2009), 2643–2666.

28.

Huang

G.B.

Song

et al., Trends in extreme learning machines: a review, Neural Networks 61 (2015), 32–48.

29.

Cao

and Lin

, Extreme learning machines on high dimensional and large data applications: a survey, Mathematical Problems in Engineering (2015), 1–13.

30.

Huang

G.B.

and Chen

, Convex incremental extreme learning machine, Neurocomputing 70(16) (2007), 3056–3062.

31.

Feng

Huang

G.B.

Lin

et al., Error minimized extreme learning machine with growth of hidden nodes and incremental learning, IEEE Transactions on Neural Networks 20(8) (2009), 1352–1357.

32.

Zhang

Lan

Huang

G.B.

et al., Dynamic extreme learning machine and its approximation capability, IEEE Transactions on Cybernetics 43(6) (2013), 2054–2065.

33.

Goodfellow

Pouget-Abadie

Mirza

et al., Generative adversarial nets, in: Advances in Neural Information Processing Systems, 2014, pp. 2672–2680.

34.

Kroese

D.P.

Porotsky

and Rubinstein

R.Y.

, The cross-entropy method for continuous multi-extremal optimization, Methodology and Computing in Applied Probability 8(3) (2006), 383–407.

35.

Bai

Huang

G.B.

Wang

et al., Sparse extreme learning machine for classification, IEEE Transactions on Cybernetics 44(10) (2014), 1858–1870.

36.

Asuncion

, UCI Machine Learning Repository, http://archive.ics.uci.edu/ml.

37.

Guo

Liu

et al., Logistic discrimination based on G-mean and F-measure for imbalanced problem, Journal of Intelligent and Fuzzy Systems 31(3) (2016), 1155–1166.

38.

Fawcett

, An introduction to ROC analysis, Pattern Recognition Letters 27(8) (2006), 861–874.

39.

Zou

Feng

and Jiang

, Improved over-sampling techniques based on sparse representation for imbalance problem, Intelligent Data Analysis 22(5) (2018), 939–958.

Sequential prediction for imbalanced data stream via weighted OS-ELM and dynamic GAN

Abstract

Keywords

1. Introduction

2. Related work

2.1 Hybrid classification methods

2.2 Online prediction

3. The proposed method

3.1 Data reconstruction phase

4.1 Datasets description

Table 1 Specifications of the datasets

Table 2 Confusion matrix

5. Conclusion

Footnotes

Acknowledgments

References

Table 1
Specifications of the datasets

Table 2
Confusion matrix