An overview of complex data stream ensemble classification

Abstract

With the rapid development of information technology, data streams in various fields are showing the characteristics of rapid arrival, complex structure and timely processing. Complex types of data streams make the classification performance worse. However, ensemble classification has become one of the main methods of processing data streams. Ensemble classification performance is better than traditional single classifiers. This article introduces the ensemble classification algorithms of complex data streams for the first time. Then overview analyzes the advantages and disadvantages of these algorithms for steady-state, concept drift, imbalanced, multi-label and multi-instance data streams. At the same time, the application fields of data streams are also introduced which summarizes the ensemble algorithms processing text, graph and big data streams. Moreover, it comprehensively summarizes the verification technology, evaluation indicators and open source platforms of complex data streams mining algorithms. Finally, the challenges and future research directions of ensemble learning algorithms dealing with uncertain, multi-type, delayed, multi-type concept drift data streams are given.

Keywords

Overview ensemble classification complex data streams evaluation technology domain data streams

1 Overview

Traditional machine learning algorithms mainly deal with steady data. Now data appears in the form of streams, which has changed the way people store, communicate and process data. In many new applications, people are faced with constantly changing data streams such as web clickstream analysis, network logs, traffic data monitoring and management, telecom data management etc. [1]. As researchers study data streams, more complex data streams types have emerged, but satisfactory performance cannot be obtained when processing data such as imbalanced, high-dimensional and noisy data. In this context, how to effectively construct an efficient knowledge discovery and mining model has become an important topic in the field of data mining.

Ensemble learning is a research hotspot that aims to data fusion, data modeling and data mining into a unified framework. The classic data streams ensemble classification algorithms including Online Accuracy Updated Ensemble Algorithm (OAUE) [2], Leveraging Bagging (LevBag) [3], etc. there are Streaming Ensemble algorithms(SEA) that deal with concept drift algorithm [4], Dynamic Weighted Majority (DWM) [5], etc. Ensemble algorithms deal with imbalanced data including SMOTE Boost [6], Resample-based Ensemble Framework for Drifting Imbalanced Stream (RE-DI) [7]. Ensemble learning process Big data streams including Vertical Hoeffding Tree(VHT) [8] and HSMiner [9] algorithms. These are some classic ensemble algorithms for complex data streams.

This article will conduct a careful analysis of the complex data stream classification algorithm and compare the data streams classification overview of previous researchers. Lemaire et al. [10] introduced the data streams learning method in the review literature, using single classifier to process the data streams as the main method. The paper also briefly introduced common data sets and evaluation methods. The review of Gomes [11] is a very comprehensive review on data streams ensemble classification from the combination of classification to diversity and dynamic update, it gives a detailed introduction mainly for the concept drift data streams. Krawczyk et al. [12] gave a detailed introduction from steady-state data to dynamic data stream ensemble classification to regression. The paper also mentioned some advanced issues of data stream for discussion. Although the above overview of data streams introduces common data streams processing methods, most of the types of data streams that are not subdivided are still dealing with concept drift as the main research object. At the same time, evaluation techniques and indicators of data streams are only brief mention and no summary. The frame structure of this paper is shown in Fig. 1. And the main contributions are as follows:

For the first time, this article comprehensively summarizes the complex types of data streams that people have studied in recent years and compares the proposed ensemble algorithms. At present, there is no overview to summarize the ensemble algorithms of various types of data streams from the perspective of ensemble methods.

The article also summarizes the development status of field text, graph and big data streams ensemble algorithms in recent years.

This article first comprehensively summarizes the evaluation techniques and evaluation indicators used in the experimental analysis of complex data streams for a summary and comparison. The current review does not provide a comprehensive overview of the evaluation methods.

Fig. 1

Comprehensive classification framework for complex data streams.

2 Introduction

In recent years, in the research field of data streams mining, great progress has been made in obtaining useful models from a large amount of rapidly generated data and traditional static batch processing methods have gradually become unsuitable for non-static learning environments. Transition from traditional steady-state data to dynamic data, the processing of data in a non-static environment means that new requirements are put forward.

2.1 Data streams

A data stream is a potentially unbounded, ordered sequence of data items, arriving over time. Under the framework of fully supervised learning, let S = {x₁, x₂, …, x_t-1, x_t, x_t+1, …} represent attributes value at time t, and y_t is at time t label. The goal of data streams classification is to train the classifier and establish the mapping relationship between the feature vector and class label. Most of the existing streaming data mining solutions are carried out under the assumption that the data is stable. However, in the real world, the generation of data streams is usually carried out in a non-stationary environment, which means that the underlying distribution of data can be changed arbitrarily over time [13]. In a non-static environment, more complex data representations will be encountered. Imbalanced data, large-volume data, multi- instance data and multi-label data will appear in the data streams. Next, we will briefly discuss the challenges of these data streams.

(1) Concept Drift

Nowadays, data appears in the format of a data streams instead of observing data in the format of a static data set in a static environment. In such a non-stationary environment, the concept of data and data distribution change over time, this phenomenon is called concept drift [14]. The following will define the concept drift from the perspective of probability. When the joint probability of two time points t_o and t₁ changes, there will be concept drift, namely Equation (1). $p_{to} (X, y_{i}) \neq p_{t 1} (X, y_{i})$ (1)

According to the speed of drift change, it can be divided into sudden change, incremental, gradual and recurring drift as shown in Fig. 2. If one source of data distribution is replaced by another source of data distribution, it is called sudden type. When you begin to observe the mixture of the two data distributions, this drift is called incremental type. When the data instances of the old data distribution begin to decrease and the data instances of the new data distribution begin to increase, the scene is defined as a gradual type. recurring drift refers to a situation where instances of new concepts or instances of old concepts begin to recur after a certain time [15].

Fig. 2

Concept drift types.

(2) Other Challenges

The real data streams only have concept drift and even class imbalance. A typical example of this situation is technical diagnosis, where the probability of failure increases with the use of time. Sometimes the relationship between the minority class and the majority class changes and the former minority class becomes the majority class [16]. In the application of social media, such as the popularity of Twitter discussion topics. In reality, a small number of categories are often the focus of research.

Non-standard data is mentioned in the literature [12]. In recent years, non-standard data and class structure have attracted more and more attention from the machine learning community. In data streams mining, multi-label and multi-instance learning is still a largely untapped field. Multi-label data is involved in data with multi-label data streams in many real-world applications. The multi-label data streams is a stream that has the same attributes as the multi-label data. Typical multi-label data includes news articles, medical text, etc [17]. Simultaneous multi-instance data is a field of little attention in the streaming mining environment. Most of the work in this field is concentrated in the fields of biological information, image retrieval, acoustic classification, and online video processing.

In the era of big data, with the development of computer science and Internet technology, big data streams are exploding at an exponential rate. According to statistics, Google processes more than hundreds of petabytes of data every day, Facebook generates more than 10 petabytes of log data every month and Taobao generates dozens of terabytes of online transaction data every day [18]. At the same time, most of the data is unstructured, such as text and graph data, which are presented to users. These complex data structures are difficult to represent, and gradually become difficult and hot issues in data stream mining.

2.2 Learning mode

From steady-state data to data in a non-static environment, its learning method is constantly changing the learning process of data as shown in Fig. 3. Traditional learning generally uses a batch learning method, but it can only process a relatively small amount of data. With the advent of the big data era, more learning methods are needed, such as incremental learning, online learning and block-based learning methods, which are commonly used methods in data streams, are also highly efficient processing methods.

Fig. 3

The process of data streams learning.

(1) Incremental Learning

Incremental learning includes receiving and integrating new instances, without the need to perform a complete learning phase from scratch. In this approach, the model will gradually evolve to adapt to changes in the incoming data. There are two ways to update the model: update by data instance and update by window. Incremental algorithms must learn from data faster than batch learning algorithms. Such algorithms must have very low time complexity. The algorithm can be modified to have the ability to incrementally learning. Incremental versions of support vector machines ISVM [19] and LASVM [20]. This online method incrementally selects a set of examples and uses support vector machines to learn from them. The ensemble learning algorithm will also add incremental learning technology. Incremental learning in ensemble classification can better apply to data streams arriving in time order. The classic algorithms include Learn++ [45], heterogeneous Bagging++ [18] and so on.

(2) Online Learning

The main difference between online learning and incremental learning is that instances are continuously obtained from the data stream and can only be processed once without re-storing and reprocessing. The requirements in terms of time complexity are stronger than incremental learning. Ideally, the online classifier implemented on the data stream must have a constant time complexity of O(1). The goal is to learn and predict at least as fast as the data streams arriving. Due to the nature of data streams that need to be processed in time and limited memory, online learning technology is widely used in data streams classification. In the data streams online learning algorithm Online Bagging and Online Boosting [21], the author compares the accuracy and processing time of online technology with batch processing technology through experiments. Algorithm OAUE [2] based on block addition mechanism and online technology. It can estimate the classifier error only for the window of the last instance in a fixed time and memory.

(3) Chunk Learning

Learning instances appear continuously in the form of data blocks. S ={ B1 ∪ B2 ∪ … ∪ Bn } locks appearing in the set are usually equal in size. Blocks are usually of equal size and the construction, evaluation, or updating of classifiers is done when all examples from a new block are available. This distinction may be connected with supervised or semi-supervised frameworks. For instance, in some problems data items are more naturally accumulated for some time and labeled in blocks while an access to class labels in an online setup is more demanding. The block-based learning method is also a common training method for data stream classification. The AWE algorithm [22] trains the classification on consecutive data blocks, and replaces the worst classifier with each update. Learn++.NSE algorithm [23] is one of the extensions of Learn++, which can learn data incrementally through blocks with fixed or different sizes, instead of batch learning without retaining old training examples.

2.3 Data streams classification

Data stream classification is a variant of the traditional supervised machine learning classification task. The main difference between these tasks is that in streaming scenarios, instances are not easy to obtain as part of a large static data set classifier. On the contrary, instances are provided as a continuous stream of data quickly and sequentially over time. Therefore, data stream classifier must be prepared to handle a large number of instances, so that each instance can only be checked once or stored for a short period of time. This section will give a brief summary from the single model classification of the classification algorithm to the ensemble model. The paper focuses on the ensemble model to deal with the data streams in non-static situations. At the same time, in Fig. 4, the transition from traditional classification to data stream classification is compared. It contains common single classifiers and ensemble classification.

Fig.4

Traditional algorithms and data streams algorithms.

(1) Single Classification Model

In the classification process, a single classifier is constantly recursively updating its structure with newly arrived data. At the same time, the single classifier is also used as the base classifier of the ensemble model. Here, we will discuss the commonly used single model classifier.

In single-model data stream classification, decision trees, Naïve Bayes and SVM are classifiers that are often used by researchers. The VFDT algorithm [24] is considered to be the most reference decision tree classifier for mining large-scale data streams. In VFDT, the tree is constructed in an incremental manner and does not retain any instances. Blanco et al. [25] proposed an online hybrid tree (OHyT), which used two different segmentation methods to construct a decision tree and two methods showed good performance in experiments. In terms of using Bayesian data stream, Krawczyk et al. [26] proposed a Bayes-based WNB-CD algorithm to process the data stream, weighted the incoming data block samples and proposed two methods for calculating weights. Recently, Hemalatha et al. [27] proposed a hybrid decision tree IFDB algorithm based on incremental flexible naive Bayes, which uses kernel density estimation to model the continuous attributes of leaf nodes to improve class prediction accuracy. Since support vectors can handle very large data. Tsang et al. [28] proposed a Core Vector Machine (CVM) is a new method based on the concept of the smallest external sphere to extend the core method. And Rai et al. [29] proposed the stream model of the classic StreamSVM algorithm, but it is only used to process the data stream and cannot detect the concept drift in the data stream.

In data stream classification, KNN and neural networks are constantly being valued to explore new mining methods. KNN is a classification algorithm based on distance metric, which is a lazy learning method. Law et al. [30] proposed an incremental classification algorithm ANNCAD that uses multi-resolution data representation to find the adaptive nearest neighbors of test points. The update speed of ANNCAD is fast, and the exponential forgetting method effectively adapts to concept drift and is very suitable for mining data streams. In terms of complex data streams, Roseberry et al. [31] proposed ML-SAM-KNN, which is a multi-label classifier using self-tuning memory for drifting data streams. Short-term memory is used to store recent examples, while long-term memory is used to remember historical and unusual data. In terms of neural networks, Leite et al. [32] proposed an Evolutionary Granular Neural Network (eGNN) algorithm, which can deal with mutations and gradual changes in typical non-stationary environments. EGNN uses fuzzy neurons to build interpretable multi-scale local models for information fusion.

(2) Ensemble Classification Model

A single model usually cannot be classified smoothly in a fast data stream. Therefore, the combination of base classifiers can make the model more robust. The ensemble model uses various combination strategies to predict and classify instances. At the same time, it has advantages in processing various complex types of data streams. There are three main frameworks for the current ensemble methods. That is Bagging, Boosting and Stacking, which are currently the most used structures in ensemble algorithms. Because of their excellent performance, they have received extensive attention from researchers. These three structures and related algorithms will be introduced in detail below.

(a) Bagging Ensemble

Bagging method is a simple and effective method to generate independent model integration. Bootstrap sampling method is used to obtain examples for training, and voting method is usually used to predict. Researchers continue to explore the bagging algorithm and propose many algorithms based on this method. Hothorn et al. [33] proposed double-bagging. In this algorithm, out-of-bag examples are used to train two classifiers in each iteration. The author recommends using this method to deal with variable problems and bias selection methods. Oza et al. [21] proposed the online version of bagging to illustrate that online processing is superior to batch processing in terms of accuracy and runtime. In the field of facial recognition, the bagging algorithm is used to integrate Extreme Learning Machine (ELM) [34] to improve the recognition performance. Random Forest is also an ensemble model under an independent framework. It uses the decision tree without pruning constructed by the CART algorithm as the base classifier, which combines bagging and random feature selection.

(b) Boosting Ensemble

Boosting is a method to improve the accuracy of any given learning algorithm, and AdaBoost is the most representative algorithm in the Boosting family. The main idea of AdaBoost is to focus on instances that were misclassified during previous training. The degree of attention is determined by the weight assigned to each instance in the training set. As people continue to study the AdaBoost algorithm, a bunch of classic algorithms about the boosting family have been extended. In the field of image segmentation, Avidan et al. [35] proposed the SpatialBoost algorithm based on AdaBoost. The analysis results of synthetic images and real images show that it is superior to AdaBoost when spatial reasoning is involved. Cortes et al. [42] proposed DeepBoost, which can use a deep decision tree as a base classifier and achieves high accuracy without overfitting the data. In [37, 38] authors developed boosting-based General Regression Neural Network (GRNN) ensembles. Tkachenko et al. [36] developed a new neural network tool integration in order to improve the accuracy of the task of recovering and predicting missing IoT data. It consists of two successive regression neural networks (GRNN) and a successive geometric transformation model (SGTM) neural structure. On two consecutively connected general regression neural networks, the SGTM neural structure is used as a supplement to improve the accuracy of the prediction results. In [37], the missing data management tasks in intelligent systems are considered. A possible recovery method for predicting partial or complete loss of data based on the extended input SGTM neural structure is proposed to improve the two GRNN ensemble. The accuracy of the weighted summation process of the output displacement of the GRNN network is improved.

(c) Stacking Ensemble

Another well-known ensemble method is Stacking, which is based on a meta-learning method. The model uses the following steps: (a) Zero-level data. All basic learners run on the original data set, (b) One level data. After level zero, the prediction made by the classifier is considered new data, (c)Final prediction. Another learning process is to use the first-level data as the new input and output to obtain the final prediction. Ortiz-Díaz et al. [38] proposed the Fast Adaptive Stacking of Ensembles-Active Learning (FASE-AL) algorithm, which uses active learning to generalize classification models with unlabeled instances. FASE is a stacking integrated algorithm used to detect and adapt the model when the input data stream has conceptual drift. Ding et al. [39] proposed a new stacking algorithm, which defines cross entropy as the loss function of the classification problem. The training process uses a neural network with stochastic gradient descent technology. Izonin et al. [40] describes a prediction method using a new, stacking-based GRNN ensemble model. Each member of the developed ensemble processes its own dataset, where the vectors of the original set of data are randomly shifted relative to the current point. The authors chose SGTM neural-like structure as a meta-algorithm for the formation of the result of the ensemble. Yang et al. [41] proposed a new method based on the integration of multiple heterogeneous pre-trained deep convolutional neural network models (P-DCNN) and superposition algorithms, which can achieve high accuracy in inverse synthetic aperture radar (ISAR) images under the condition of small sample sets Automatic recognition of space targets.

(d) Combination strategy

Combining the predictions of the overall membership can improve overall performance. Through appropriate combination methods, accurate predictions can be made for difficult-to-classify instances. However, people spend a lot of energy to generate different sets of classifiers, but little is known about the method of combining the output of the classifiers. Here we mainly introduce the combination strategies of common weighting and complex meta-learning methods.

The weighted method can combine basic model outputs by assigning weights to each basic model. Majority voting is the simplest weighting method for classification problems. The category predicted by the sample is the category with the most votes. For example, the OzaBag [21] and LevBag [3] algorithms both use the majority voting method. At the same time, complex weighted voting is often favored by researchers in data stream classification. Weighted voting is used in the AWE algorithm [51], but it does not perform well in terms of using mutational concept drift. And the AUE2 [53] algorithm uses a non-linear weighting function, while the OAUE algorithm [2] uses a non-linear and linear weighting function respectively.

Meta-Learning is a process of learning by a learner. The meta-learning model is different from the standard machine learning model because it contains multiple learning stages. In the meta-learning, the output of the individual classifier is used as the input of the meta-learner to produce the final output. A typical example of meta-learning is the Stacking algorithm, which creates a meta-data set and generalizes the meta-classifier from it, and is responsible for combining the base classifier’s predictions into the final prediction during the prediction process. The Stacking combination strategy used in data streams classification is the restricted hoeffding trees integrated algorithm proposed by Bift et al. [43], which uses the ADWIN change detection method to set the learning rate of the perceptron for the data stream, and uses ADWIN when the ensemble members are no longer performing well to reset them.

2.4 Methods

The research methodology used in this review is PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) [44], which can make our review articles more rigorous and logical.

(1) The PRISMA Protocol

PRISMA is used to summarize the existing literature report results in a specific research field to get the final result. The program includes a four-stage flow chart (i.e. identification, screening, eligibility and inclusion), which helps to ensure consistent and accurate reporting of all types of system reviews and meta-analysis studies [44]. By using this program, the conclusions and results will be free from the bias of review research, but most reviews may be affected by selective results reporting. In addition, by defining relevant query criteria, various sources with qualification criteria can be used to exclude irrelevant articles.

The four-stage flow chart will first determine the source of the article, and then filter the search results of duplicate and irrelevant articles based on the title and abstract. Then the full-text papers are excluded according to the eligibility criteria to obtain the included research for further analysis.

(2) Inclusion Criteria and Database

Eligibility criteria are indispensable in assessing the effectiveness, applicability and comprehensiveness of the review [44]. It is worth noting that the design of exclusion criteria is incremental. For example, if an article is excluded by exclusion criterion 1, the article will be automatically excluded without further verification of additional exclusion criteria.

Inclusion Criteria (IC)

IC-1-The title, abstract or keywords of the paper have one of the following terms:

“Ensemble learning” or equivalent expressions;

“Concept drift data streams” or equivalent expressions;

“Imbalanced data streams” or equivalent expressions;

“Multi-label data streams” or equivalent expressions;

“Multi-instance learning” or equivalent expressions;

IC-2- Research Article or Proceedings / Conference Article;

IC-3 - The paper is written in English;

Exclusion Criteria (EC)

EC-1- The paper is not available;

EC-2- Study is not written in English;

EC-3- The paper does not consider Ensemble learning;

EC-4- The paper does not consider specified algorithm type.

The search strategy is to search for data by entering search strings based on IC-1, IC-2, and IC-3 in the following databases: IEEE explorer and SpringerLink. These two databases are considered to be important and reliable sources of high-quality publications in the field of computer science and engineering. According to the above queries entered in the two databases, a total of 36645 potentially relevant articles were found. The five data stream types studied in this paper are steady, concept drift, imbalanced, multi-label and multi-instance data streams. 27223 articles were deleted by screening titles, abstracts and the remaining 9,422 articles were evaluated in detail. 370 of the 9422 articles were duplicates and were therefore deleted. Therefore, there are 9,052 studies left and 8,902 irrelevant articles according to the exclusion criteria (EC-1, EC-2, EC-3, Ec-4). Finally, the remaining 150 articles were manually screened to select the ensemble classification algorithm except for the 31 corresponding data streams. This review will review the performance of 31 documents from the algorithm and summarize the evaluation results. Figure 5 shows the flow chart of PRISMA’s current system review. Where N represents the total number of current papers and n represents the number of components.

Fig. 5

PRISMA flowchart diagram for the current systematic review.

3 Ensemble classification of complex data streams

The ensemble learning method is a good data classification method. Because a single classifier cannot adapt to the model well in the fast-coming and changing data stream. Through the combination of multiple classifiers, the model can be more adapted to the complex data streams. Among them, the concept drift in the complex data stream is currently the most researched problem. Ensemble learning has different implementation strategies for different complex data streams, such as using key technologies such as online learning, block-based learning and incremental learning to make the model fit better. At the same time, Table 6 summarizes the performance comparison of ensembled algorithms for processing complex data streams.

3.1 Steady data streams

The number of steady data streams is the most stable of all data streams. It does not need to deal with concept drift, imbalanced, multi-labeling and other issues. It is not gradually the hotspot of people’s research, but for the integrity of the content. Some classic algorithms are worthy of research and discussion. It is an exploration of people’s early research on data. Ensemble learning is divided into block-based learning and online learning technology when processing steady-state data streams. The specific algorithm will be introduced below.

3.1.1 Method based on block learning

In the steady-state data stream, the researchers study static data and the data is trained in batches or blocks. Compared with online learning, the use of block-based algorithms may not be studied much. This section will review its classic algorithms.

The early classic Learn++ [45] is an ensemble algorithm based on neural network classifiers, which has been mentioned in many experimental comparison algorithms. The algorithm is to build a new neural network model on the incoming block data for incremental training set and then use the majority voting decision-making mechanism to make predictions. Minku et al. [46] introduced the use of negative correlation learning Negative Correlation Learning (NCL) in incremental learning. Negative correlation learning is a successful application in constructing neural network ensemble. The author proposed two models of fixed size and growing NCL(GNCL) to explore their degree of forgetting knowledge. Ensemble classifier is a main direction of incremental learning research and there are many ensemble incremental learning methods. Among them Learn++ derived from AdaBoost is very special, while the heterogeneous Bagging++algorithm [47] based on the analysis of the advantages and disadvantages of Learn++, the author proposes an incremental ensemble algorithm that uses bagging to generate an incremental ensemble for each block of data. The algorithm uses bagging to build a new model from the incoming data block. Experiments show that algorithm is superior to Learn++and negative correlation learning algorithms. Kidera et al. [48] proposed an incremental ensemble model based on the AdaBoost.M1 algorithm. In the model, the long-term memory resource allocation network Resource Allocation Network With Long-Term Memory (RAN-LTM) developed to achieve stable incremental learning is used as the classifier. At the same time, the author proposes a new weighted majority voting method to update the weight of the classifier.

3.1.2 Method based on online learning

Online learning is a commonly used technology in ensemble learning. It is a hot method for everyone to study whether it is in static or non-static data. The online learning algorithm processes the arrived instance once, without storing and processing and can make a faster response to the data stream classification.

The application of online technology in steady data is relatively extensive. The OzaBag algorithm [21] requires that the entire training set is immediately available, but in some cases random access to the data is required. The proposed algorithm solves this problem. It uses each base classifier to update with k copies of newly arrived data, where k follows the k Poisson (1) distribution and finally is classified by unweighted voting, which is equivalent to the batch version in terms of accuracy. The OzaBoost ensemble algorithm [21] maintains a fixed-size set. These classifiers are trained according to the received instance and each new instance updates the classifier in sequence. The instances in the sequence that were misclassified by the previous classifier will update their weights for the latter classifier to attention. When the classifier misclassifies an instance, the Poisson distribution parameter related to the instance is multiplied by 1/2ɛ, and becomes λ (1/2 (1 - ɛ)) when it is correctly classified, where ɛ is the classification error of the base model.

Random Forests (RFs) are often used in many computer visions and machine learning applications. However, in most applications, RFs are used offline and Online Random Forest (ORF) algorithm [49] is a combination of OzaBag, random feature selection and new tree growth methods. It allows online decision trees to be built and the author also added a time weighting scheme. Bifet et al. [3] proposed a new algorithm LevBag, which randomizes the weights of input stream instances to improve the accuracy of the ensemble. At the same time, the simplicity of the bagging technique is combined with adding more randomness to the input and output of the classifier. According to the sequential learning mechanism. Zhai et al. [50] proposed an efficient online extreme learning machine ensemble algorithm called Ensemble of Online Extreme Learning Machine (E-OS-ELM) for large data set classification, the algorithm is based on online sequential learning (OS- ELM) algorithm trains the base classifier, and uses a simple voting strategy to ensemble the trained base classifier.

3.1.3 Chapter summary

The above two subsections review the classic algorithm of static data streams, which is based on block and online ensemble classification. Although steady data streams have gradually ceased to be the scope of everyone’s research, their algorithm ideas are still worthy of discussion by researchers. Among the algorithms introduced above, we have selected several classic algorithms to quantitatively analyze the performance of the algorithm. From Table 1 (experimental data from their respective papers), it can be observed that under the same data set, the block-based algorithm Bagging++algorithm is superior to GNCL in terms of accuracy and running time. At the same time, the online-based LevBag [3] randomizes the weight of the input stream, which improves the accuracy of the ensemble classifier and the time is less than that of the OzaBag [21] algorithm.

Table 1
Algorithms comparison

Algorithm Datasets Accuracy Time

GNCL [46] Mushroom 98.87-99.48 74.98

Bagging++ [47] Mushroom 93.90-99.94 9.17

OzaBag [21] CovType 74.71 16.05

LevBag [3] CovType 87.81 6.98

Algorithm	Datasets	Accuracy	Time
GNCL [46]	Mushroom	98.87-99.48	74.98
Bagging++ [47]	Mushroom	93.90-99.94	9.17
OzaBag [21]	CovType	74.71	16.05
LevBag [3]	CovType	87.81	6.98

3.2 Concept drift data streams

Another biggest challenge in dealing with data in a dynamic environment is concept drift. This drift will occur when the concept involved in the collected data changes after a period of stability. Nowadays, researchers’ research on concept drift has been continuously increasing in the past ten years. This section mainly deals with concept drift data streams from the perspective of ensemble learning methods, using ensemble learning to ensemble block-based, online-based and other latest processing concepts. The technique of drift is discussed.

3.2.1 Method based on block learning

The idea of block-based ensemble is to divide the incoming data stream into data blocks to update the base classifier, replace the weak classifier with a new classifier. Then it uses a combination strategy to predict the result. The block-based method is a common method to adapt concept drift.

Wang et al. [51] proposed an Accuracy Weighted Ensemble (AWE) algorithm for classifier ensemble. It can adapt to potential concept drift, but there are problems with gradual drift in order to improve the algorithm. Deckert et al. [52] proposed a new concept drift processing algorithm Batch Weighted Ensemble (BWE), which is used to deal with sudden and gradual concept drift in labeled data with decision attributes. The learning examples are processed into blocks of the same size. Then drift detector is introduced into the ensemble framework, which has better accuracy than the performance of the standard precision weighted ensemble classifier. Brzezinski et al. [53] proposed a block-based data ensemble classification algorithm Accuracy Updated Ensemble (AUE2), which aims to respond to different types of concept drift. The main novelty of this algorithm is to combine the ensemble weighting mechanism with the incremental training of the component classifier. It is different from the linear function used in AWE. The weighting function in the article uses the calculation formula of the nonlinear weighting function Equations (2)–(4). ${MSE}_{ij} = \frac{1}{| B_{i} |} \sum_{{xy} \in B_{i}} (1 - f_{y}^{j} (x))^{2}$ (2)

${MSE}_{r} = \sum_{y} (1 - p (y))^{2}$ (3)

$ω_{ij} = \frac{1}{{MSE}_{r} {+ MSE}_{ij} + ɛ}$ (4)

Where ω_ij is the weight of the base classifier and the function $f_{y}^{j} (x)$ represents the posterior probability of the base classifier predicting that x is class y.

Some current methods believe that it is inappropriate to update the ensemble while processing the continuous stream, such as infinite growth ensemble or deleting all its basic learners when trying to overcome concept drift. Bertini Junior et al. [54] proposed the Iterative Boosting Streaming Ensemble (IBS) algorithm, which applies boosting to block data and maintains ensemble by adding a certain number of basic learners. Elwell et al. [23] proposed the Learn++ for Non-Stationary Environments (Learn++.NSE) algorithm inspired by the boosting algorithm. It learns from continuous blocks of data without making any assumptions about the nature or drift speed. It can learn from the environment of constant or variable drift rate, adding or deleting concept classes and periodic drift. This method is gradual learning, just like the Learn++family member algorithm, there is no need to access the previously learned data.

The description of concept drift in the traditional data stream mining stage requires the availability of labeled samples, but it is not feasible to incorporate tags into simplified transactions in terms of time and resource utilization. Abdualrhman et al. [55] proposed a deterministic concept drift detection (DCDD) method based on ensemble classifiers. The ensemble structure uses AdaBoost. This drift detection method can describe concept drift regardless of the label assigned to the sample. This method has a high drift detection rate, a minimum false warning rate and a miss rate, and can detect concept drift in a scalable manner.

3.2.2 Method based on online learning

The online ensemble algorithm is designed to learn in a class label environment that is available after each instance, instead of learning in a block manner. With the arrival of class labels online, the online algorithm can be more effective than in the processing block environment. Respond quickly to concept drift. It is a model that is continuously revised according to sequential learning.

Weight has always been the key content of everyone’s research and through continuous innovation of weighting methods to adapt to concept drift Kolter et al. [5] proposed the Dynamic Weighted Majority (DWM) algorithm, it will maintain the weight of a base classifier Pool. It increases or deletes base classifiers according to the performance of the global algorithm. Each base classifier has a weight. If the base classifier makes a prediction error, it will use the β multiplication factor to reduce the weight of the base classifier. Experiments prove that the classifier can keep learning well in the concept drift environment. OAUE [2] is a very classic algorithm, which ensembles the weighting mechanism based on the block method and online learning. This method converts the block-based ensemble to an online learner method, which trains each stored instance and weights the classifier. The innovation puts forward an efficient weighting function of the base classifier. And recently Aancy et al. [58] proposed a dynamic streaming data mining method based on adaptive online learning rules (SOAR). SOAR ensembles the basic characteristics of block and network ensemble. Then it updates the weight of each classifier based on its quality. It promotes adaptive windows to deal with gradual and sudden concept drift.

Diversity for Dealing with Drift (DDD) algorithm is proposed by Minku et al. [56] according to different types of drift, different ensemble levels can get the best priority accuracy. DDD maintains ensemble systems with different levels of diversity which uses the advantages of diversity to deal with drift. And it uses information from old concepts to help learn new concepts. Diversified Online Ensembles Detection (DOED) proposed by Sidhu et al. [57]. This method retains two ensembles of weighted classifiers. One is the ensemble of low diversity and the other is the ensemble of high diversity. Classifiers update according to the classification accuracy of new data instances. In recent years, some researchers have combined dynamic weighting and diversity to improve extremely high accuracy. Sidhu et al. [59] proposed an online ensemble method, Diversity Dynamic Weighted Majority (DDWM) algorithm, which has different concept distributions. This method maintains two sets of weights with different levels of diversity. Updating or deleting an ensembled classifier is based on the classification accuracy. Adding a new classifier is based on the final global prediction of the algorithm and ensemble global prediction of any instance.

3.2.3 Chapter summary

Concept drift in data streams is currently the most extensive research direction. Ensemble learning is a powerful tool to deal with concept drift. The above subsections summarize the concept drift ensemble classification algorithm from a block-based and online perspective. We carefully analyze their robustness through experimental indicators from the following classic algorithms. Due to AUE2 [53] combines AWE’s [51] ensemble weighting mechanism and incremental training of component classifiers, it can be observed in Table 2 that the AUE2 algorithm improves the classification accuracy and reduces the operation time compared to AWE algorithm. Based on the online OAUE [2] algorithm, a new weighting function is proposed to improve the accuracy compared with DWM [5], but on the contrary it increases the running time and space complexity.

Table 2
Algorithms comparison

Algorithm Datasets Accuracy Time

AWE [51] SEA 87.73 0.67

AUE2 [53] SEA 89.19 0.61

DWM [5] Hyper 89.94 0.21

OAUE [2] Hyper 90.43 0.22

Algorithm	Datasets	Accuracy	Time
AWE [51]	SEA	87.73	0.67
AUE2 [53]	SEA	89.19	0.61
DWM [5]	Hyper	89.94	0.21
OAUE [2]	Hyper	90.43	0.22

3.3 Imbalanced data streams

In addition to concept drift problem, non-stationary data is also affected by the complexity of the data. In particular, the class imbalance that is often encountered, the main feature is that the number of samples in a certain category is significantly more than the number of samples in other categories. Class imbalance is even an obstacle to static data learning. Because the classifier is biased towards most classes and tends to misclassify instances of a few classes.

Many methods have been proposed in the literature to deal with learning from imbalanced data. Such as data sampling methods, cost-sensitive algorithms, single-class classifiers and ensemble classification methods. This section mainly focuses on the processing of imbalanced data around ensemble learning methods. Data preprocessing and cost-sensitive algorithms are discussed in conjunction with ensembled learning.

3.3.1 Data preprocessing method

In imbalanced data, researchers hope to reduce the degree of imbalance between minority samples and majority classes at the data level, in which data resampling is a representative method to obtain uniform class distribution. The methods based on resampling mainly include under-sampling, over-sampling and mixed sampling of the combination of the two. Under-sampling is to remove some instances from most classes and oversampling is to add a few instances of classes.

Synthetic Minority Over-sampling Technique (SMOTE) algorithm [60], it uses linear interpolation to create minority samples. In recent years, ensemble learning is combined with SMOTE to improve the classification accuracy. The SMOTE Boost algorithm proposed by Chawla et al. [6] successfully utilizes the advantages of Boosting and SMOTE algorithms. The Boosting algorithm improves the accuracy of the classifier by focusing on hard-to-learn examples while SMOTE only focuses on a few realities. Embedding SMOTE into the Boosting algorithm pays more attention to the instances of the minority class. At the same time, SMOTE can increase the instances of the minority class.

Wang et al. [61] improved the algorithms oversampling-based online bagging (OOB) and under-sampling-based online bagging (UOB). The author uses resampling and time attenuation metrics to improve the two ensemble learning methods of OOB and UOB to overcome the problem of online class imbalance. Finally, combining their respective advantages, the weighted ensemble methods WEOB1 and WEOB2 are both more accurate than OOB. Its robustness is better than UOB. Recently, Zyblewski et al. [63] proposed a framework that combines dynamic ensemble and preprocessing technology to deal with highly imbalanced data streams. Then they proposed a sampling method that uses a hierarchical bagging classifier to replace the minority and majority classes. Experiments show that the method of combining dynamic ensemble selection and data preprocessing is better than the latest online and block-based methods.

The problem of concept drift is also faced in the imbalanced data streams, which is an emerging content of current research. Ancy et al. [62] proposed the Handling Imbalanced Data with Concept Drift (HIDC) method. The algorithm focuses on the preprocessing of data and the classification process of stream data. When faced with class imbalance, oversampling and under-sampling techniques are used. When drift occurs, the implicit weighting scheme is applied to the base classifier to compare and replace the worst classifier. Zhang et al. [64] proposed Resembling-based Ensemble Framework for Drifting Imbalanced Stream (RE-DI) ensemble framework. The ensemble framework consists of a long-term static classifier that handles gradual changes and multiple dynamic classifiers that handle sudden conceptual drift. The resampling buffer is used to store instances of the minority class to solve the imbalanced distribution.

3.3.2 Cost-sensitive learning method

In addition to changing the distribution of data to reduce the impact of class imbalance on classification, it can also be considered from the level of algorithms. It mainly includes cost-sensitive learning and ensemble learning. The misclassification costs of different class are introduced in the learning and the higher misclassification costs are assigned to the minority class to improve its generalization performance. Cost-sensitive classification considers the different costs of different misclassification types. The cost matrix encodes the penalty for classifying samples from one category to another. Let C (i, j) denote the cost of predicting an instance in category i as category j. Here a cost matrix such as Table 3 is defined for two classifications. When dealing with the problem of category imbalance, the cognitive importance of positive examples is higher than that of negative examples. Therefore, the cost of misclassifying a positive instance is greater than the cost of misclassifying a negative instance, namely C (+ , -) > C (- , +) and making the correct classification usually does not bring any penalty. Then the cost-sensitive learning process seeks to minimize the number of costly errors and the total misclassification cost.

Table 3
Confusion matrix

Category Positive Negative

Positive C (+ , +) C (+ , -)

Negative C (- , +) C (- , -)

Category	Positive	Negative
Positive	C (+ , +)	C (+ , -)
Negative	C (- , +)	C (- , -)

Sun et al. [65] combined cost-sensitive and Boosting algorithm. The paper comprehensively analyzed the advantages and disadvantages of AdaBoost algorithm in solving class imbalance problems. Then they explored three cost-sensitive boosting algorithms, namely AdaC1, AdaC2, AdaC3.The cost term is introduced into the learning framework of AdaBoost. Further analysis shows that the proposed algorithm conforms to the phased additive modeling in statistics to reduce the cost index loss. At the same time, Tao et al. [66] proposed a cost-sensitive ensemble algorithm for support vector machines based on cost weights. In this method, in order to ensure the consistency of the optimization goal of the weak learner and the boosting scheme, the author not only uses the cost-sensitive support vector machine as the basic weak learner, but also modifies the standard boosting scheme to be cost-sensitive. In order to ensure that the continuous classifier has more training examples, especially the few classification examples at the boundary, an adaptive sequential misclassification cost weight determination method is proposed.

The ensemble of neural networks to deal with imbalanced data is also a hot topic of research. Wong et al. [67] proposed two new cost-sensitive methods, namely, Cost-Sensitive Deep Neural Network (CSDNN) and Cost-Sensitive Deep Neural Network Ensemble (CSDE) to Solve the problem of class imbalance. Among them CSDE is ensemble learning version of CSDNN. In order to improve the generalization performance of CSDNN, random under-sampling and hierarchical feature extraction of hidden layers of deep neural network are used in CSDE. Recently, Loezer et al. [68] proposed a cost-sensitive Adaptive Random Forest (CSARF) algorithm and compared it with the adaptive data set on 6 real worlds and 6 different analogs. Random forest (ARF) and random forest with resampling (ARFRE) are compared. And the results show that CSARF is better than ARF in average recall rate and average F1.

3.3.3 Chapter summary

This chapter summarizes the common handling methods for imbalanced data streams. Using data preprocessing and cost-sensitive algorithms to deal with imbalanced data classification that is common in daily life. Table 4 shows that the proposed RE-DI algorithm which uses an ensemble classifier composed of a static classifier and multiple dynamic classifiers sliding windows. The algorithm is based on a new resampling method to process unbalanced data. It can be observed from the table that the experimental indicators AUC and Accuracy under the same data set show that it is better than the other three algorithms.

Table 4
Algorithms comparison

Algorithm Datasets AUC Accuracy

UOB [61] PAKDD 60.7 42.4

OOB [61] PAKDD 63.5 70.7

ARF [68] PAKDD 63.2 80.1

RE-DI [64] PAKDD 65.8 80.3

Algorithm	Datasets	AUC	Accuracy
UOB [61]	PAKDD	60.7	42.4
OOB [61]	PAKDD	63.5	70.7
ARF [68]	PAKDD	63.2	80.1
RE-DI [64]	PAKDD	65.8	80.3

3.4 Non-standard data streams

In recent years, non-standard data has attracted more attention from machine learning scholars. Researchers are paying more attention to the learning of multiple goals. The data source may be accompanied by multiple class labels or the composed data is multiple instances, so it needs to be able to effectively adapt to these Examples of learning methods. Although most of the current research focuses on static, non-streaming frameworks, there are also some researches that focus on more complex data streams. This summary will summarize the research progress of multi-label and multi-instance classification of non-standard data streams in recent years, as well as their respective learning algorithms using ensembled methods.

3.4.1 Multi-label data streams

Formally multi-label streams classification problem is to train a model with sub-label Y ⊆ L for each instance in a high-speed data stream, where L is a label set. Although multi-label classification has been studied in traditional database mining scenarios, multi-label data stream classification is a relatively new concept and has not been fully resolved. Multi-label classification methods are usually divided into two categories. (1) problem transformation (2) algorithm adaptation. The problem conversion method converts a multi-label problem into one or more single-label problems.

Read et al. [69] proposed Ensemble of Classifier Chains (ECC) to deal with multi-label classification. This method is developed on the basis of the binary correlation method, which reduces the computational complexity compared with the more complex methods currently used. By passing the label-related information along the classifier chain, the author overcomes the shortcomings of the binary association method and obtains better prediction performance while maintaining low computational complexity. Wang et al. [70] proposed the Streaming Weighting ML-KNN based Ensemble Classifier (SWMEC) method, which is an ensemble model of multi-label data stream classification based on ML-KNN. The weighted ensemble model proposed by the author uses multi-label data stream to effectively update the model.

In recent years, applications such as text classification, web and social media mining have increasingly demanded multi-label classification algorithms. Nguyen et al. [71] introduced a scalable multi-label data classification ensemble method (MVI) based on online variable reasoning, in which random projections are used to create an ensemble system. As a second-order generation method, the classifier can effectively use the underlying structure of the data in the learning process. At the same time, multi-label is driven by the challenge of concept drift. Sun et al. [72] proposed the multi-label ensemble with Adaptive Windowing (MLAW) algorithm to design an effective ensemble paradigm for multi-label data stream classification. The algorithm deploys a novel change detection based on the Jensen-Shannon divergence to identify different types of conceptual drift in the data streams. In [73], a sliding window mechanism is also used to establish an ensemble model based on the COINS algorithm. The semi-supervised technology is applied to the problem of multi-label data stream classification. At the same time, a new label emergence detection algorithm is proposed and applied to data stream classification to refine the model and improve the accuracy of the classification algorithm.

3.4.2 Multi-instance data streams

In traditional supervised learning, there is only a single instance. This instance can only be identified by one label as shown in Fig. 6(a). However, in multi-instance learning, data structure is more complicate. Each object or target variable is defined by an uncertain number of feature vectors as shown in Fig. 6(b). In multi-instance learning, there is the concept of a package. A package is a group of instances and the associated class label belongs to the package. The use of ensemble learning methods to process multi-instance data is also a research direction and it has excellent performance in the evaluation indicators of classification tasks.

Fig. 6

Single instance learning and Multi instance learning.

Bjerring et al. [74] proposed a randomized ensemble method Multi-Instance Rule Induction (MIRI) which was improved based on the MITI algorithm [69]. It is a single learner of decision tree, specially designed for multi-instance classification, but the splitting criterion of the algorithm has a significant impact on accuracy. Through modification, the algorithm is changed into a regular learner. Finally, the author proposes a randomization method resulting in an ensembled model. In recent years, neural network-based multi-instance classification algorithm BP-MIP and multi-instance regression algorithm BP-MIR have been proposed one after another. Zhang et al. [76] introduced neural network ensemble technology to solve the multi-instance learning problem. They constructed BP-MIP ensemble and BP-MIR ensemble respectively. Multi-instance neural network ensemble is better than single multi-instance neural network in solving multi-instance problems.

In the image target detection collection, Babenko et al. [76] proposed an improved online boosting scheme Online MILBoost (OMB). They believe that once a packet is marked as a positive packet, then all instances in it are also positive, so it can be used for training. You can use a stream process similar to standard online boosting to update the ensemble structure with a new model. For the same target detection task, Wang et al. [77] proposed a semi-supervised online weighted multi-instance learning ensemble method Semi-Weighted Multi Instance Learning (Semi-WMIL). In order to make full use of the target and its surrounding background information, combined with semi-supervised learning, multi-instance learning and Bayes theorem, a new single target detection tracker is proposed. In the parameter update stage of each frame, the tracker uses the inconsistency function of the block-based labeled and unlabeled training samples when selecting the optimal base classifier.

3.4.3 Chapter summary

This section summarizes an emerging direction that is currently being researched, namely multi-label and multi-instance algorithms. The fields involved in this direction are mainly image, video and so on. Due to the complex characteristics of its own data source, its evaluation indicators are also diversified. As can be seen in Table 5, the proposed algorithm MLAW [72] is superior to the first three algorithms in the two indicators Hamming loss and Subset accuracy due to its periodic change detection mechanism that can reuse the previous concepts.

Table 5
Algorithms comparison

Algorithm Datasets Hamming loss Subset accuracy

MLOzaBag [72] Random Tree 0.0934 0.689

ML-SAM-knn [72] Random Tree 0.0541 0.550

MHT [72] Random Tree 0.0614 0.574

MLAW [72] Random Tree 0.0135 0.792

Algorithm	Datasets	Hamming loss	Subset accuracy
MLOzaBag [72]	Random Tree	0.0934	0.689
ML-SAM-knn [72]	Random Tree	0.0541	0.550
MHT [72]	Random Tree	0.0614	0.574
MLAW [72]	Random Tree	0.0135	0.792

Table 6

Comparison of ensemble classification algorithms for complex data streams

Algorithms	Data stream types	Comparison algorithms	Datasets	Advantages and disadvantages
Growing NCL [46]	steady data streams	MLP, Fixed size NCL	Letter, Vehicle, OPt. Digits, Adult, Mushroom	Advantages: The fixed NCL has good generalization ability, and the increased NCL can easily overcome the influence of strong forgetting. Disadvantages: Fixed NCL is easier to be forgotten than increased NCL, and increased NCL has poor generalization ability.
Bagging++ [47]	steady data streams	Learn++, GNCL, BPNN	Adult, Mushroom, Austra, Bupa, Cancer 1	Advantages: Compared with Learn++and incremental NCL, Bagging++ has higher accuracy and faster learning. Disadvantages: The algorithm of heterogeneous ensemble is too simple and rough.
AdaBoost RAN-LTM [48]	steady data streams	AdaBoos.M1	Banana, Image, Xnumeral	Advantages: The model can learn asymptotically without serious forgetting. Disadvantages: When creating a new classifier, how to update the weight function of the classifier output is more ambiguous.
OzaBag [21]	steady data streams	Naïve Bayes, Bagging, Online MLP	Balance, Promoter, Census, Forest Covertype	Advantages: Naïve Bayes base classifier has the same performance as bagging processing. Disadvantages: The accuracy of MLP base classifiers is lower than that of batch bagging.
OzaBoost [21]	steady data streams	Naïve Bayes, Boosting, Online MLP	Balance, Promoter, Census, Forest Covertype	Advantages: The accuracy of the Bayes classifier is higher than the Boosting algorithm. Disadvantages: Boosting performance in batch processing of small data sets is better than Online Boosting.
LevBag [3]	steady data streams	ADWIN Bagging, Hoeffding	Forest Covertype, Poker-Hand, Rotating	Advantages: Compared with bagging, this algorithm has two randomization improvements: adding resampling and using output monitoring code. Disadvantages: Although the accuracy rate has been improved, the speed has not been improved.
ORF [49]	steady data streams	Offline RF, Online RF	DNA, g50c, Letter, Mushroom, Salt-age, USPS	Advantages: The algorithm is stable and has stronger robustness to noise, and real-time operation is easy to implement. Disadvantages: The algorithm converges to the offline version but lacks relevant theoretical proofs.
E-OS-ELM [50]	steady data streams	MRAN, GAP-RBF	Letter, Shuttle, KDD-10%, Forest	Advantages: fast learning speed and high accuracy compared with other algorithms. Disadvantages: The process is too complicated.
BWE [52]	concept drift data streams	AWE	ele2, hyperplane, STAGGER	Advantages: The introduction of drift detection methods from the integrated structure can well adapt to sudden changes and gradual type drifts, which makes up for the shortcomings of AWE in dealing with sudden changes. Disadvantages: The experiment only compares an algorithm, and at the same time, it is slow to reflect the first change on the data set STAGGER.
Learn++.NSE [23]	concept drift data streams	SEA, DWM, AdaBoost	Gaussian, SEA, Nebraska Weather Prediction	Advantages: The algorithm can adapt to the distribution of any data stream, and at the same time, it conforms to the characteristics of incremental learning and allows learning new knowledge to temporarily forget the old. Disadvantages: In terms of running time, as the amount of data continues to increase, the algorithm is slower than the SEA algorithm.
AUE2 [53]	concept drift data streams	ACE, DDM, LevBag, Oza, DWM, NSE, NB	HYP, RBF, SEA, LED, Elec, Poker, Airlines, Conv	Advantages: The algorithm can deal with various types of drift, and does not depend on the size of the data block and proposes a new weighting function. Disadvantages: There is no detailed discussion on the size of the ensemble in the experiment. Whether the base classifier is only effective with the Hoffdingg tree.
IBS [54]	concept drift data streams	SEA, CZE, learn++.NSE, EBI, WMA	SEA, Circle, LED, Electricity, KDDX99,	Advantages: The algorithm can better adapt to drift when learning knowledge classifiers in batches, and also has a deletion mechanism. Disadvantages: The algorithm itself is still a passive detection drift and it is not clearly stated that the algorithm adapts to the type of drift.
DCDD [55]	concept drift data streams	MD3	Poker Hand	Advantages: In order to overcome the error detection of current drift detectors, the proposed deterministic model detection method has better applicability. Disadvantages: Few comparison algorithms and data sets are used.
DWM [5]	concept drift data streams	Naïve Byes, DWM-NB, DWM-ITI	Stagger, SEA, Electricity, UCI	Advantages: The algorithm proposes a new weighting idea, from the global performance to the local to update the base classifier, which can adapt to drift well. Disadvantages: There is no formal theoretical explanation of the weighting formula, and there are fewer applications of comparison algorithms when setting up experiments.
DDD [56]	concept drift data streams	EDDM, DWM	Circle, SineV, Line, Plane, Bool	Advantages: Use the advantages of diversity to deal with concept drift, and use old concept information to help learn new knowledge. Disadvantages: There are further improvements to deal with repetitiveness or periodicity and sudden drift.
OAUE [2]	concept drift data streams	AUE, AWE, LevBag, DWM, ACE	Airlines, CovTppe, Hyper, Poker, RBF, Wave 2	Advantages: The algorithm analyzes the advantages and disadvantages of block-based and online technologies. The proposed algorithm is a combination and improvement of the two, which can better adapt to drift. Disadvantages: It is a passive adaptation to concept drift, and there is no detection of drift.
DOED [57]	concept drift data streams	EDDM, DWM	Stagger, Electricity, Breast cancer	Advantages: diversified ensemble is used, and two weighted ensembles, namely high score and low score ensemble are retained, and the response to gradual changes is fast. Disadvantages: A small amount of data does not have a very good accuracy for the algorithm’s model, and the accuracy can only be maintained as the data increases.
SOAR [58]	concept drift data streams	Em, AWOE	Weather	Advantages: The algorithm combines diversity, self-adaptive windows, and association modes, which can handle gradual changes and sudden changes at the same time, with high accuracy. Disadvantages: Too few data sets are used and only one is used for experiments, which is not extensive enough.
DDWM [59]	concept drift data streams	DWM, EDDM, NB	Stagger, SEA, Electricity, Breast Cancer	Advantages: Two basic classifiers are used, which are well adapted to concept drift. Disadvantages: Too few comparison algorithms.
SMOTE Boost [6]	imbalanced data streams	SMOTE, Boost, AdaCost	Satimage, Phoneme, KDDSCup-99	Advantages: The algorithm successfully uses the advantages of SMOTE and Boost, making SMOTE pay more attention to fewer categories and improve the classification accuracy. Disadvantages: For the integrated base classifiers that are too single, common classifiers can be used for comparison.
HIDC [62]	imbalanced data streams	DFGW-IS	Weather	Advantages: The algorithm uses two parts, data sampling and dynamic weighting, to deal with imbalanced data streams and drift issues. Disadvantages: There are too few data sets and comparison algorithms in the experiment.
RE-DI [64]	imbalanced data streams	OOB, UOB, LB, ARF	Covtype, poker, GMSC, HYP	Advantages: Effective use of resampling and integrated dynamic classifiers to deal with drift has achieved good results. Disadvantages: The algorithm uses block-based training classifiers, and sampling may cause a lot of time loss.
AdaC1, AdaC2, AdaC3 [65]	imbalanced data streams	AdaBoost, CSB2, AdaCost	Cancer, Hepa, Pima, Sick	Advantages: AdaC2 enables the standard algorithm to optimize the error rate according to the theorem to optimize the cost error rate. Disadvantages: There are fewer comparison algorithms in imbalanced data streams.
CSDE [67]	imbalanced data streams	AdaCost, MetaCost, SMOTE Boost	KDD98, Customer churn, Direct Marking 3	Advantages: CSDE uses random under-sampling and hierarchical feature extraction of hidden layers of deep neural networks. It has better generalization and is not easy to overfit. Disadvantages: The running time of the experimental algorithm needs to be further optimized to improve the running speed.
CSARF [68]	imbalanced data streams	ARF, ARF_RE	Electricity, Airlines 4 , Santander	Advantages: The proposed combination of error-based cost and random forest. Experiments show that it is superior to other algorithms in various indicators. Disadvantages: The influence of concept drift data streams that still exists in imbalanced data streams is not considered in this article.
ECC [69]	multi-label data streams	CLR, EBR, EPS, IBLR	Music, Scene, Yeast, Genbase	Advantages: reduced computational complexity, using the ensemble of classifier chains to eliminate disordered chains. Disadvantages: The space complexity is high, although the time consumption is shortened.
SWMEC [70]	multi-label data streams	SMART, MLKNN	TMC2007	Advantages: The algorithm has higher accuracy and lower storage cost. Disadvantages: Too few algorithms, especially integrated learning to deal with Multi-label data streams.
MVI [71]	multi-label data streams	BR, CC, RT, ML	Enron, IMDB, TMC	Advantages: time cost is not high, with stable performance. Disadvantages: The experimental part of the algorithm is too short and does not consider Multi-label data streams of concept drift data streams.
MLAW [72]	multi-label data streams	MHT, MLOzBag, ML-SAM-knn	IMDB, RBF, TMC2007	Advantages: It can handle different types of concept drift data streams. Disadvantages: As ensemble continues to increase, pruning strategies should be considered.
MIRI [74]	multi-instance data streams	— — — —	Eastwest, musk, westeast	Advantages: The ensemble based on decision tree has good classification performance. Disadvantages: Since the experiment is done for comparison with its own structure changes, there is no comparison with other algorithms.
BP-MIP, BP-MIR Ensemble [75]	multi-instance data streams	EM-DD, Citation-kNN	Musk data	Advantages: The early ensemble of two types of neural networks has good robustness. Disadvantages: The early processing of Multi-instance data streams lacked the measurement of other indicators.
OMB [76]	multi-instance data streams	OAB, SemiBoost, Frag MILTrack	Sylvester and David, Occluede Face	Advantages: The ensemble of Multi-instance data streams learning combined with online Boost has stable performance. Disadvantages: For target tracking, there are some problems, such as long-term occlusion of objects or completely leaving the scene, which may cause learning errors.
Semi-WMIL [77]	multi-instance data streams	WMIL, ODFS, RTCT, IVT, DFT	Basketball, Cliffbar, Couple	Advantages: The proposed semi-supervised WMIL tracker can ensure the effectiveness in online practical applications. Disadvantages: In the ensemble process, the selection criteria of the base classifiers need to be further improved.

4 Domain data streams ensemble classification

Data streams classification has been extended to many fields by researchers to conduct streams data mining and extract useful information for research applications. As the Internet age continues to touch all walks of life, people’s demand for classification tasks is also constantly expanding. In stream mining tasks, text classification, graph data classification and big data classification. These fields are also advanced problems of data stream classification that researchers are studying. Due to the significant changes in the representation, structure and magnitude of data. More complex algorithms are required for analysis. This section will analyze the results of these data streams in recent years.

4.1 Text data streams

Text classification in the Natural Language Processing (NLP) has always been a hot research direction. Popular social platforms like Twitter and Weibo will analyze news content, social interactions and advertisements. They can recommend more personalized services to users. However, text stream classification is still an emerging direction of text classification. The timely arrival of the data streams involved and the characteristics of one-time processing need to be considered by researchers. The current text streams classification methods are also relatively diverse, such as clustering methods, distributed processing of large-scale text streams and ensemble methods.

Trofimov et al. [78] analyzed the limitations, challenges and solutions of distributed text stream s classification. Text stream classification is an important problem that is difficult to solve on a large scale. Batch processing systems are widely used for text classification tasks, but they cannot provide low latency. Distributed stream processing systems can provide low latency, but they do not support the same level of fault tolerance and determinism as batch processing systems. In this work, the author demonstrates how distributed stream processing features affect the results of a typical text classification data stream. The analysis shows a trade-off between fault tolerance and reproducibility and performance.

Text streams mining based on concept drift is a very challenging research topic. The detection of concept drift is a computationally complex problem. Song et al. [79] proposed a new ensemble model Dynamic Cluster Forest (DCF) for text stream classification with concept drift. The ensemble model proposed in this paper is based on multiple Clustering Trees (CTs). In particular the DCF model has two novel strategies: (1) an adaptive ensemble strategy that dynamically selects discriminative CTs according to the inherent characteristics of the data stream. (2) a dual voting strategy that simultaneously considers the reliability and accuracy of the classifier. At the same time, Yang et al. [80] proposed unlabeled keywords to classify the text stream to reduce the burden of manual labeling. They use keywords and unlabeled documents to build a basic text classifier to classify the text streams and use the ensemble classifier algorithm to deal with the problem of concept drift in the text data stream. Most research on stream data classification is based on the assumption that the data can be fully labeled. However, in real applications, manually marking the entire stream for training is impractical and time-consuming. In data streams environment, it is common to have only a small portion of positive data and a large amount of unlabeled data. In this case, applying the traditional stream algorithm directly to the positive unlabeled stream may not work well or lead to poor performance. Pan et al. [81] proposed a dynamic classifier ensemble method for the classification of positive text streams and unlabeled text streams (DCEPU). In the classification stage, the problem is solved by constructing a suitable evaluation set and designing a new dynamic weighting scheme.

Traditional machine learning methods use features such as word frequency to represent documents in a vector space. It has limitations in processing the order and semantics of words. Upadhyay et al. [82] proposed a weighted classifier ensemble algorithm to solve the text classification problem. When combining classifiers, this algorithm associates the predictions of each classifier with different weights instead of using majority voting as a combination method. The particle swarm optimization algorithm is used to minimize the loss function of the training data to obtain the optimal weight. Samami et al. [83] proposed an optimal text classification method based on natural inspired algorithms and ensembled classifiers. In this model, an optimization algorithm based on biogeography and an ensembled classifier are used for feature selection. Compared with a single classifier, use of an ensemble classifier for classification can provide better performance for optimal text classification.

4.2 Graph data streams

The problem of graph classification appears in the context of many classic fields, such as chemical and biological data, networks and communication networks. In recent years, with the emergence of Internet applications, people have become more and more interested in dynamic graph streaming applications. Such network applications are defined on the basis of a large number of nodes. Applications such as (1) Decompose the communication patterns of users in a social network within an appropriate time window into a set of disconnected graphs, which are defined in a large number of user node domains. (2) The user’s browsing mode is usually a sub-graphic of the web map. The most challenging situation in the classification of graph data streams is that data only exists in the form of graph streams. Therefore, it is necessary to detect changes in the data stream to perform real-time classification in the classification of graph data streams.

The dynamic change of the label may indicate an important event or activity pattern. Aggarwal et al. [84] studied the differential classification problem in the graph stream in the paper, in which predicted important classification event is the change of the node classification label. Unlike steady ensemble classification problems. This method focuses on detecting changes in node classification dynamically and in real time, instead of actually classifying nodes. The differential data classification problem can also be considered as a general form of the non-dispersive event detection problem, in which node labels are used to supervise the detection process. The experimental results show the effectiveness of this method.

In many practical applications, graphs can be defined on a very large set of nodes. The large field size of the underlying graph makes it difficult to learn the summary structure information of the classification problem, especially in the case of data. To meet these challenges, Aggarwal [85] proposed a probabilistic method for constructing “in-memory” summary of underlying structure data. The digest uses a two-dimensional hashing scheme to determine the authentication mode in the underlying map. The author provided the probability boundary of the model quality determined by the process and proved the quality experimentally on some real data sets.

There is also an imbalanced problem in the classification task of graph data streams. Pan et al. [86] proposed a classification model to deal with imbalanced graph data with noise which is graph ensemble boosting. It uses an ensemble-based framework to graph data streams divided into blocks containing many types of imbalanced noise graphs. A boosting algorithm is proposed for each independent block, which combines discriminative subgraph mode selection and model learning as a unified graph classification framework. In order to solve the problem of concept drift in the graph data stream, an instance-level weighting mechanism is used to dynamically adjust the instance weights. Through the instance-level weighting mechanism, the enhanced framework can emphasize difficult graph samples. The classifier constructed from different graph data blocks which constitute an ensemble for the classification of graph data streams.

In the chemical industry, there are often fault classifications. Liu et al. [87] proposed an ensemble FDA model based on manifold-preserving sparse graphs. First, on the premise of maintaining the basic manifold structure of the training samples. Some mislabeled samples are filtered using the sparse graph of the retained manifold. Secondly, in order to improve the model’s robustness to left-standard error samples, the FDA-based Bagging algorithm was used to construct multiple sub-classifiers and these sub-classifiers were combined to form a robust classifier. Experiments proved the robustness of the algorithm model.

Su et al. [88] proposed a new method of multi-label classification, which relies on the ensemble learning of a set of random output graphs on multiple labels. And a structure of kernel-based output learner as the base classifier. For ensemble learning, the difference between the output graphs provides the required diversity of base classifiers and improves performance as the ensemble scale continues to increase. The author studied different methods of forming ensemble predictions, including majority voting and two methods of reasoning about the graph structure before or after merging the basic model into the ensemble. The behavior of the multi-standard ensemble is explained theoretically from the two aspects of the diversity and consistency of the micro-standard prediction and the previous research on the single-target ensemble is promoted.

4.3 Big data streams

In recent years, big data has become a key field representing the modern information and digital world. A large amount of data is being generated every moment in life. In the latest development, IBM uses the “5Vs” model to describe big data. In addition, some classic mining algorithms need to transfer the entire data set multiple times. Although distributed data mining and data stream mining have been extensively studied, there are relatively few studies on distributed data stream mining methods.

Big data streams not only continuously receive large amounts of data, but may also have different types of characteristics. In addition, concepts and functions tend to evolve throughout the process. Haque et al. [9] designed a method based on multi-layer ensemble called HSMiner to solve the above-mentioned challenges that is to mark instances in the ever-changing big data stream. When classifying big data streams, HSMiner may face scalability issues. To solve this problem, the author proposes three methods to use MapReduce-based parallelism to build these massive AdaBoost ensemble.

The Vertical Hoeffding Tree (VHT) algorithm [8], which is the first distributed data algorithm for learning decision trees which can be used to perform classification tasks on large data streams arriving at high speed. VHT is a novel method of vertical distribution of decision trees. The algorithm is implemented on Apache SAMOA, so it can run on real-world clusters. Through detailed experiments and compared with the centralized sequential tree model, it is shown that VHT can handle dense and sparse instances with thousands of attributes, and the accuracy is very small. In large-scale imbalanced data, Lin et al. [89] proposed an ensemble random forest algorithm based on Apache Spark, which can be used for large-scale imbalanced classification of insurance business data. The experimental results show that the ensemble random forest algorithm is more suitable for insurance product recommendation or potential customer analysis than traditional SVM, logistic regression and other strong classifiers. Combining the proposed bootstrap under-sampling algorithm with KNN, it can be used for the preprocessing of imbalanced classification algorithms. The ensemble learning algorithm combined with bootstrap sampling preprocessing can further reduce learning.

In dealing with large-scale imbalanced data aspect, Mwangi et al. [90] proposed an ensemble model of RUSBoost combined with cost-sensitive CNN. The combination of RUS and boosting can overcome the imbalanced performance of the RUS algorithm at the large data level, while the convolutional neural adding a cost layer to the network can achieve the purpose of allowing the classifier to automatically learn the cost during the training process, thereby improving the performance level of the model.

Real-time mining of ever-changing data streams faces new challenges. Such as ever-increasing data volume, speed and volatility require real-time data processing, the ability to react quickly and adapt to changes. Marron et al. [91] proposed a random forest ensemble based on random hoeffding tree real-time data streams classification method which using vector SIMD and the multi-core capabilities available in modern processors to provide the required throughput and accuracy.

The rise of the Internet of things and other sensor networks has created many vertically distributed and high-speed data streams, which require specialized algorithms to realize true distributed data mining. Denham et al. [92] proposed a new hierarchical distributed stream miner Hierarchical Distributed Stream Miner (HDSM), which can learn the characteristics of different data streams in the case of the smallest data transmission to the central location. relationship. Experimental evaluation shows that compared with the previously proposed distributed stream mining method, the classification accuracy has been significantly improved.

5 Complex data streams evaluation methods

The correct evaluation of classifiers or regression models is a key problem of machine learning, as shown in Fig. 7. The use of correct and appropriate evaluation measures is also an important part of the experiment. Only through accurate data can evaluation measures show the advantages of the proposed algorithm. Performance evaluation is the last step of classification. The performance of the classifier is indispensable in the classification process. The quality of the data to be classified determines the quality of the algorithm used. The following will summarize the verification techniques of data stream classification that researchers often use in the data stream environment, their unique performance evaluation methods for complex data streams and a detailed discussion on the software platform they use.

Fig. 7

Data evaluation process framework.

5.1 Validating methods

In the context of steady and batch learning, the most commonly used scenario for estimation measures is cross-validation. Due to the limited amount of traditional data, cross-validation is used to increase data usage and its biggest disadvantage is time-consuming. However, it is not directly applicable in a data environment with strict calculation requirements and conceptual drift. Although the number of streaming learning algorithms continues to increase, the indicators and experimental design for evaluating the quality of learning models are still open issues. The evaluation technology is proposed to determine which examples are used to train which examples are used to test the learning model. This section summarizes the widely used verification techniques in data experiments in many documents.

Holdout: Holdout is an independent test set. To use this evaluation method, two data subsets are required: a training data set and an independent Holdout set. According to this arrangement, at any given moment when you want to evaluate the model, there is a set of tests that the model has not used before. By testing the learning model on such a constantly updated set, an unbiased estimate of the model error is obtained. Gama et al. [93] used this verification technique in the ultra-fast forest tree system (UFFT). At the same time, many researchers gave a detailed introduction to the Holdout evaluation method in their review papers [12]. Hold has been applied to data streams scenarios.

Prequential: This method is the most commonly used evaluation technique in data streams. First it uses each instance to test the model and then trains the model. To observe an example, the current model will make a prediction; when the system receives feedback from the environment, the loss function can be calculated. At the same time, in order to make the error estimation more robust to this situation, it is necessary to implement an appropriate forgetting mechanism that combines a sliding window or a fading factor. Brzezinski et al. [2] used a combination of window and attenuation factor as the forgetting condition when performing the experimental setting of the OAUE algorithm. The window size was d = 1000 and the attenuation factor α=0.01, which was verified by adjusting these two variables. algorithm. As well as the IBS algorithm of Bertini Junior et al. [54], it uses the same window and attenuation factor combined evaluation method for experiments. At the same time, the Prequential Accuracy indicator is widely used in supervised data stream classification algorithms, such as DDD algorithm [56] and DOED algorithm [57].

Cross-validated prequential: Many stream data currently faced by society are generated from many sources, and at the same time have real-time and large-scale features. Bift et al. [94] proposed a new distributed K-fold verification method for big data streams evaluation methods. When using real data to train the classifier, the priority evaluation method can only run one experiment compared to cross-validation, while standard cross-validation handles each fold of the stream independently, so the concept drift that occurs in the data stream may be missed. In order to overcome this challenge, the author proposes three strategies: (1) cross-validation (2) split-validation (3) bootstrap validation. The theory proves that the error of using the K-fold method is lower than that of the priority method. The error is equal to the true error minus the estimated error. For all classifiers c, m test cases, for all δ ∈ (0, 1], as in Equations (5) and (6). $\begin{matrix} Pr (| trueerror - estimated . error |) \\ ⩽ \sqrt{\frac{ln (2 / δ)}{2 m}} ⩾ 1 - δ \end{matrix}$ (5) with the same high probability, having more instances will reduce the discrepancy between the true error and the estimated error. $\begin{matrix} E [{| {trueerror}_{k} - {estimatederror}_{k} |}^{q}] ⩽ \\ E [{| {trueerror}_{1} - {estimatederror}_{1} |}^{q}] (\forall q ⩾ 1) \end{matrix}$ (6)

where E (|trueerror_k - estimatederror_k|^q) is no larger for the prequential k– fold strategy than for a prequential evaluation strategy.

After the above discussion, the error of the priority k-fold strategy is smaller than that of the priority method. This evaluation method is used in the experimental verification of the adaptive random forest (ARF) algorithm proposed by Gomes et al. [95]. Verification latency: A large part of the research of streaming mining is to classify the availability of real labels immediately after making a prediction. But in many practical situations, the tag arrives with a non-negligible delay. This raises the question of how to evaluate the classifier in this situation. Gomes et al. [95] conducted a timely setting and delayed setting to simulate the experiment in the evaluation test of the ARF algorithm, and proposed a method to evaluate the classification performance, assuming that there is a problem setting that the label is not easy to obtain. It is identified as a “delayed label setting” and its definition is very simple: it evaluates learning based on its ability to correctly predict an instance, and this basic fact can only be used after a fixed unit of time. Grzenda et al. [96] proposed a new method for evaluating data streams label delay, namely continuous re-evaluation. It is applied to reference data streams and distinguishes stream mining techniques based on the ability of stream mining techniques to refine predictions based on newly arrived instances.

5.2 Evaluation indicators

As people pay more attention to streaming data stream mining nowadays, various algorithms for complex data streams have been proposed one after another. However, a good algorithm for processing data stream classification requires corresponding evaluation indicators to verify. This section is a discussion on the evaluation indicators of complex data streams that have been widely studied. Table 8 summarizes the evaluation indicators of complex data streams algorithms. Different types of data streams have their own unique indicators. Using these to measure the performance of the algorithm allows researchers to clearly understand the algorithm itself.

Table 7
Confusion matrix

Category Positive sample Negative sample

Positive Sample True Positive (TP) False Negative (FN)

Negative Sample False Positive (FP) True Negative (TN)

Category	Positive sample	Negative sample
Positive Sample	True Positive (TP)	False Negative (FN)
Negative Sample	False Positive (FP)	True Negative (TN)

Table 8

Evaluation indicators for complex data streams algorithms

Data stream type	Algorithms	Datasets	Data stream evaluation indicators
			ACC	RT	M	K	DDR	FM	P	R	G	AUC	HL	RL
Steady data streams	AdaBoost RAN-LTM [48]	Banana, Image, Xnumeral	√
	Bagging++ [47]	Adult, Mushroom, Austra	√	√
	OzaBag [21]	Balance, Promoter, Census	√	√
	OzaBoost [21]	Balance, Promoter, Census	√	√
	LevBag [3]	LED, Poker-Hand	√	√	√	√
Concept drift data streams	SEA [4]	Adult, Seer	√
	AWE [51]	SEA	√
	IBS [54]	SEA, Hyperplane, LED	√
	DCDD [55]	Poker-Hand					√
	DWM [5]	Stagger, SEA, Electricity	√	√
	DDD [56]	Circle, SineV, Electricity	√	√
	OAUE [2]	CovType, LED, Wave, RBF	√	√	√
	DOED [57]	Vehicle, Sick, Labor, Zoo	√	√	√	√
	SOAR [58]	Weather						√	√	√
	DDWM [59]	SEA, Hyperplane, Breast cancer	√	√	√	√
Imbalanced data streams	SMOTE Boost [60]	Satimage, KDDCup-99Intrusion						√	√	√
	OOB, UOB [61]	Gearbox, Small building	√							√	√
	HIDC [62]	Weather						√	√	√	√
	RE-DI [64]	Poker23, PAKDD, GMSC	√									√
	SVM-CSE [66]	German, Ycast, Wpbc						√			√
	CSDE, CSDNN [67]	CCSF, DM, KDD98, Churn									√	√
	CSARF [68]	Santander, Electricity, Airline						√		√		√
Multi-label data streams	ECC [69]	Music, Scene, Yeast, Genbase	√					√					√
	SWMEC [70]	TMC2007						√						√
	MLAW [72]	IMDB, RBF, Medical Mill		√									√
Multi-instance data streams	MITI [74]	Eastwest, musk, fox	√
	Semi-WMIL [77]	Basketball, Cliffbar, Couple	√
	OMB [76]	Sylvester and David, Occluede Face	√							√
Big data streams	VHT [8]	Electricity, Particle	√	√
	HSMiner [9]	ForestCover, PAMAP2	√	√				√			√
	RUSBoost+Cost-Sensitive CNN [90]	WHOI-Plankton, ImageNet							√	√	√
	HDSM [92]	EEG, GAS, FLT, SEA	√

Remarks: Running Time: RT; Memory: M; Kappa Static: K; Drift Detection Rate: DDR; F-Mean: FM; Precision: P; Recall: R; G-mean: G; Hamming Loss: HL; Ranking Loss:RL.

(1) Traditional Classification Evaluation Indicators

In the traditional machine learning classification of steady-state data streams, the most important indicators of the classification algorithm by researchers are focused on accuracy, error rate, memory and running time, regardless of the four indicators of regression or classification tasks are the top priority. Changes in some of these indicators reflect the criteria for the quality of the proposed algorithm. Running time is also called time complexity and it is usually calculated in seconds. However, in order to measure the performance of the research algorithm more comprehensively, more complex evaluation indicators, namely Precision and Recall are proposed, which can better evaluate traditional classification or regression tasks. Precision is used to measure the precision of the classifier, which means that the positive samples that are identified as positive account for the proportion of all samples that are identified as positive. The greater the accuracy rate, the higher the accuracy of the classifier. Recall is used to measure the recall rate of the classifier, which means that the positive samples that are identified as positive account for the proportion of all positive samples. The higher recall rate, the better the classification performance of the classifier.

(2) Data Streams Classification Evaluation Indicators

In the data stream classification scenario, various complicated situations will be encountered, such as different concepts of the classification before and after, and the number of tags of the number stream is not the same. The evaluation of these complex situations requires researchers to study in different categories. The following will summarize the classification and evaluation indicators of non-static data streams used by researchers in the literature in the classification of non-static data streams. In a non-static environment, the traditional classification evaluation index can still be used as a general evaluation index. There are new indicators for the evaluation of time in a non-static environment. Bift et al. [97] proposed RAM-Hours, which is a one-dimensional amount of computing resources used by the streaming algorithm based on the cloud computing service rental cost option.

Concept drift data streams: In data stream containing concept drift, it is sometimes necessary to actively detect whether drift occurs and then process it according to the designed algorithm, so that the classifier can learn properly without reducing the accuracy all the time. Abdualrhman et al. [51] used the drift detection rate and drift detection error rate in the DCDD algorithm. The first calculation is (the number of correctly detected drifts/the total number of detected drifts) and the latter is (the number of falsely detected drifts/the total number of detected drifts).

Imbalanced data streams: In the unbalanced data stream, traditional indicators are no longer suitable for unbalanced scenarios. The classifier always tends to most categories, so for the imbalanced classification performance evaluation, related scholars have proposed the commonly used F-measure and G-mean. F-mean is related to recall rate (Recall) and precision rate (Precision), it can correctly evaluate the classification performance of the classifier for multi-class and minority samples. G-mean is proposed by Kubat et al. [98]. The geometric mean is often used in the field where the data stream of unbalanced label attributes is located. In order to express these two indicators, the confusion matrix Table 7 is mainly used to represent the above indicators and their calculation formulas are Equations (7)–(10). $F - measure = Recall \times Precision$ (7) $Recall = \frac{TP}{TP + FN}, Precision = \frac{TP}{TP + FP}$ (8)

It can be seen that the size of the F-measure value related recall and precision. Only when the recall rate and precision rate are both large, the value of F-measure will increase accordingly. Thus F-measure can correctly evaluate the classification of the classifier for majority-class and minority-class samples performance. $G - mean = \sqrt{{Accuracy}^{+} \times {Accuracy}^{-}}$ (9) ${Accuracy}^{+} = \frac{TP}{TP + FN} {Accuracy}^{-} = \frac{TN}{TN + FP}$ (10) G-means comprehensively considers the classification accuracy of the two types of samples. Compared with the accuracy rate, it can better measure the imbalance of the classification method effect on the data set.

Kappa statistics is a frequently used statistical method in uneven data. Among them, standard Kappa K is also a commonly used evaluation index, and Bift et al. [97] et al. proposed the generalized Kapp M, which is more suitable for unbalanced data than standard K. Unbalanced data on the chart, receiver operating characteristic Receiver Operating Characteristic (ROC) and Area Under ROC Curve (AUC) are also commonly used methods to evaluate the performance of the classifier.

Multi-label data streams: In streaming data mining tasks, multi-label data stream-related algorithms are always a less researched direction. Due to its complexity, it has different requirements for the design of experiments, but it still has its unique classification performance evaluation method. Many literatures involved corresponding evaluation indicators. Such as Hamming loss and Sub-Accuracy are proposed in [99]. Hamming Loss considers prediction errors and loss errors. Then it evaluates how often an instance tag pair is misclassified. Subset Accuracy is a very strict accuracy metric. If all the labels predicted by the classifier are correct, the classification is considered correct.

Big data Streams: Now that everyone is in the era of information explosion, the rapid arrival of high-speed big data streams requires new indicators to measure classification performance. Speedup and Throughout are used in the experiment of VHT algorithm [8]. SpeedUp is a comparison between the speeds of algorithms commonly used in big data algorithms. Throughput is the description of throughput, that is the number of classified instances per second. In the experiment of HDSM algorithm [92], the average CPU time MTBC (mean resource CPU Time), MDTV (maximum data transmission) and other performance indicators are proposed as the measurement indicators of the algorithm.

In addition to the above commonly used evaluation indicators, some people have also proposed more complex methods to evaluate other characteristics of the algorithm. Shake et al. [100] proposed recovery analysis, which provides a concept learner with the ability to quickly discover concept drift and take appropriate measures to maintain the quality and generalization performance of the model. Not only the effect of the model in reducing errors in the new decision space is considered, but also the time required.

5.3 Open source software platforms

Massive Online Analysis(MOA) 5 : The MOA framework is based on the WEKA library, which provides a powerful tool for data flow analysis. MOA provides many data stream learning algorithms, including ensemble classifiers. In addition to learning algorithms, MOA also provides data stream generators (such as SEA, AGR), evaluation methods (such as holdout, prequential) and evaluation indicators (CPU time, RAM-hours, kappa). MOA can be used through GUI or command line, which is convenient for batch running tests and evaluating the performance of own algorithms. MOA is implemented in Java and has many common features with the WEKA framework. Therefore, researchers can expand, call and modify the source code according to their own ideas to realize their own data flow ideas.

Scalable Advanced Massive Online Analysis (SAMOA) 6 : SAMOA provides a set of distributed stream algorithms for the most common data mining and machine learning tasks, such as classification, clustering and regression, as well as programming abstractions to develop new algorithms that can be run on the distributed stream processing engine (DSPE). It has a pluggable architecture that allows it to run on multiple DSPEs (such as Apache Storm, Apache S4 and Apache Samza). In addition, Samoa also provides users with some adaptive data flow learning algorithms, such as vertical Hoeffding tree methods for distributed processing. A Multi-label Extension to WEKA (MEKA) 7 : MEKA is an open source Java framework based on the famous Weka library. Meka provides a convenient interface for practical applications, as well as a wealth of multi-label classifiers, evaluation indicators and multi-label experiments and development tools. It supports multi-label and multi-target data, including incremental and semi-supervised contexts. Meka is easy to use from the command line interface (CLI) or graphical user interface (GUI). Any new Meka classifier can also be combined in any existing Meka ensemble scheme and any Weka-based classifier. Meka also supports semi-supervised and incremental classification in a multi-label context, considering the general multi-objective context, making it a good platform for working in these fields.

JUBATUS 8 : JUBATUS is a combination of a distributed processing framework and a streaming machine learning library. It is the first open source analysis platform that can better analyze massive data streams. By defining three basic operations, namely update, mixing and analysis, JUBATUS adopts a loose model sharing architecture to learn and share machine learning models more effectively. The functions of JUBATUS include the following aspects: online machine learning library, feature vector converter and distributed fault-tolerant online machine learning framework. The goal of JUBATUS is to minimize the size of the model and the number of mixing operations while maintaining high accuracy.

Advanced Data Mining and Machine Learning System (ADAMS) 9 : It is a workflow engine designed to prototype complex prototypes and maintain knowledge workflows. ADAMS itself is not a data flow learning tool, but it can be used in conjunction with other frameworks such as MOA and SAMOA and WEKA to execute data streams analysis. ADAMS itself is not a data streams learning tool, but it can be used in conjunction with other frameworks such as MOA, SAMOA and WEKA to execute data streams analysis.

6 Research directions and prospects

Although many types of data streams now have their own solutions, there are still many types of data streams that need to be explored and studied by researchers at the algorithm level. For example, mining delayed label streams, processing multiple types of concept drift data, processing multiple parallel existing streams and ensemble learning algorithms processing uncertain data streams, etc. These issues will be carefully analyzed below.

(1) The challenge of uncertain data streams ensemble classification

Mining uncertain data stream classification is a current research direction that is rarely studied. Currently available data streams classification algorithms are dedicated to accurate data mining, but uncertain data is very common in real applications. In the sensor network collection and transmission process, information such as humidity, temperature and weather contains a lot of uncertainty. In recent years, there has been a lot of research on uncertain data. For its classification research, especially the direction of ensemble classification. It is a scarce field. The author intends to use a detection technology to detect the uncertainty of the attribute value in the data stream and intends to use the decision tree as the base classifier for ensemble to realize the classification of the uncertain data stream.

(2) The challenge of multi-type streams ensemble classification

Nowadays, most stream ensemble algorithms only have a single type stream. However, in some applications, such as event history analysis involving data review, there may be multiple parallel streams. In such multiple streams, the same data event may appear at different times in each stream and may have different descriptions. This brings up several interesting and new challenges, such as how to aggregate information about the same event available in different streams, how to predict the moment when an event occurs in one of the streams and know the knowledge of other streams and whether to develop a commitment new ensemble for handling multiple streams. The author is thinking of using sliding window technology to incrementally learn data sources, and in order to improve accuracy, use the ensemble of heterogeneous base classifier structures to adapt to different data sources.

(3) The challenge of delayed data streams ensemble classification

Most of the current frameworks assume that the access to the target value is immediate or without too much delay. However, in real life, many are delayed predictions. An example is weather forecast, our forecast will be evaluated in the future. This is related to the problem of label delay, even if we can access such information, it will not be available immediately after the arrival of the new instance. When it is not possible to quickly mark all the examples, it is still possible to obtain the true target value for a limited number of examples at a reasonable cost. The author thinks that active learning can be combined with ensemble methods, active technology is used for sampling, and ensembled methods can be used to better train and improve accuracy.

(4) The challenge of multi-type concept drift ensemble classification

As the data generated in people’s production and life is continuous, at the same time, various new concepts are mixed in the data. In the data streams classification process, most algorithms still focus on detecting a certain type of concept drift, but this situation is in line with the actual status quo. More efficient algorithms should be studied to deal with multiple types of concept drift to continuously improve the accuracy of classification. The author is thinking about the next step, using online learning, because it has incremental characteristics that can recall previously learned knowledge and consider the diversity of ensemble classifiers to improve classification accuracy to deal with drift problems.

7 Conclusion

Predicting the data streams arriving at high speed is inherently difficult. Ensemble learning methods can make the data better fit the model and improve performance. This article reviews the existing complex data streams ensemble classification algorithms. First outline the basic background of data streams and methodology. Then we discuss in detail the ensemble method of the complex data streams, the application domain data streams and its evaluation technology. Finally, in view of the challenges faced by the ensemble classification of complex data streams and the next research direction is proposed, so that these problems can be solved or have more research significance. Ensemble learning has been proven to greatly improve the performance of things prediction, but the comparison of algorithms is adopted in this review. The ensemble structure becomes more and more complex, and the running time becomes slower. Since the algorithm will be affected by factors such as ensemble size, parameters and framework. So ensemble algorithm requires researchers to optimize the model, such as pruning strategies and so on.

At the same time, this review also has some shortcomings. The article’s introduction to the domain data streams is not comprehensive enough, such as the application of intrusion detection and sensor data streams which need further improvement.

Footnotes

Acknowledgments

This work was supported by the National Nature Science Foundation of China (62062004), the Ningxia Natural Science Foundation Project (2020AAC03216) and the Graduate Innovation Project of North Minzu University (YCX21085).

²Generator scripts and dataset links available at:

⁴

References

Ding

, Han

, Li

, Review of concept drift data streams mining techniques, Computer Science 3(2) (2016), 24–2.

Brzezinski

D.S.

, Stefanowski

, Combining block-based and online methods in learning ensembles from concept drifting data streams, Information Sciences 265 (2014), 50–67.

Bifet

, Holmes

, Pfahringer

, Leveraging Bagging for evolving data streams, Proc of the 2010 European conference on Machine learning and knowledge discovery in databases, Berlin, Germany: Springer (2010) 135–150.

Street

, Kim

, A streaming ensemble algorithm (SEA) for large-scale classification, Proc of the Seventh ACM SIGKDD International Conference on Knowledge Discovery and Data Mining 6 (2001), 377–382.

Kolter

J.Z.

, Marcus

, Maloof Dynamic Weighted Majority: An Ensemble Method for Drifting Concepts,”, Mach Learn 8(36) (2007), 2755–2790.

Chawla

N.V.

, Lazarevic

, Hall

L.O.

, et al., SMOTEBoost: Improving prediction of the Minority Class in Boosting, Proc of Knowledge Discovery in Databases: PKDD 2003, Berlin, Germany: Springer (2003), 107–109.

Zhang

, Liu

, Wang

, et al., Resample-based ensemble framework for drifting imbalanced data streams, IEEE Access 7 (2019), 65103–65115.

Kourtellis

, Morales

G.D.F.

, Bifet

, et al., VHT: Vertical hoeffding tree, Proc of 2016 IEEE International Conference on Big Data (Big Data), Washington, USA: IEEE, (2016), 915–922.

Haque

, Parker

, Khan

, Thuraisingham

, Evolving big data stream classification with MapReduce, Proc of the 2014 IEEE 7th International Conference on Cloud Computing (CLOUD). Anchorage, USA: IEEE, (2014), 570–577.

10.

Lemaire

, Salperwyck

, Bondu

, A survey on supervised classification on data streams, Business Intelligence 205 (2015), 88–125.

11.

Gomes

, Barddal

M.J.P.

, Enembreck

, et al., A survey on ensemble learning for data stream classification, ACM Computing Surveys 50(2) (2017), 1–36.

12.

Krawczyk

, Minku

L.L.

, Gama

, Ensemble learning for data stream analysis: a survey, Information Fusion 37 (2017), 132–156.

13.

Sun

, Wang

, Liu

, et al., Online Ensemble Using Adaptive Windowing for Data Streams with Concept Drift, International Journal of Distributed Sensor Networks 12(5) (2016), 1–9.

14.

Iwashita

A.S.

, Papa

J.P.

, An overview on concept drifts learning, IEEE Access 7 (2019), 1532–1547.

15.

S.Y.

, Han

, Shen

M.Y.

, Zhang

C.Y.

, Sun

, Survey of ensemble classification algorithms for data streams with concept drift, Computer Engineering 46(01) (2020), 15–24.

16.

Zyblewski

, Sabourin

, Woźniak

, Preprocessed dynamicclassifie ensemble selection for highly imbalanced drifted datastreams, Information Fusion 66 (2021), 138–154.

17.

Sun

, Shao

, Wang

, Efficient ensemble classification for multi-label data streams with concept drift, Information 10(5) (2019), 2078–2489.

18.

Liu

, Zhang

, Sampling for Big Data Profiling: A Survey, IEEE Access 8 (2020), 72713–72726.

19.

Biggio

, Corona

, Nelson

, et al., Security evaluation of support vector machines in adversarial environments, Support Vector Machines Applications. Berlin: Springer, (2014), 105–153.

20.

Y.Y.

, Boukharouba

, Boonrt

, et al., Application of an incremental SVM algorithm for on-line human recognition from video surveillance using texture and color features,”, Neurocomputing 126(3) (2014), 132–140.

21.

Oza

N.C.

, Online Bagging and Boosting, Proc of 2005 IEEE International Conference on Systems, Man and Cybernetics. Waikoloa: IEEE 3 (2005), 2340–2345.

22.

Wang

, Yu

P.S.

, Han

, Mining Concept-Drifting Data Streams, Data Mining & Knowledge Discovery Handbook, (2003), 789–802.

23.

Elwell

, Polikar

, Incremental learning of concept drift in nonstationary environments, IEEE Transactions on Neural Networks 22(10) (2011), 1517–1531.

24.

Domingos

, Hulten

, Mining high-speed data streams, ACM KDD Conference, (2000), 71–80.

25.

Ortiz Díaz

A.A.

, Frías Blanco

I.I.

Palomino Mariño

L.M.

, et al., An Online Tree-Based Approach for Mining Non-Stationary High-Speed Data Streams, Revista de Informática Teórica e Aplicada 27(1) (2020), 36–47.

26.

Krawczyk

, WozniaM

, Weighted Nave Bayes Classifier with Forgetting for Drifting Data Streams. 2015 IEEE International Conference on Systems, Man, and Cybernetics (SMC) IEEE, (2015), 2147–2152.

27.

Sweetlin Hemalatha

, Pathak

and Vaidehi

, Hybrid decision trees for data streams based on Incremental Flexible Naive Bayes prediction at leaf nodes, Evolutionary Intelligence, 12(4) (2019), 515–526.

28.

Tsang

, Kocsor

, Kwok

, Simpler core vector machines with enclosing balls, Proc of the Twenty-Fourth International Conference, Oregon: ACM, 227 (2007), 911–918.

29.

Rai

, Daumé III

and Venkatasubramanian

, Streamed Learning: One-Pass SVMs, Proc of the 21st International Joint Conference on Artificial Intelligence (2009), 1211–1216.

30.

Law

, Zaniolo

, An Adaptive Nearest Neighbor ClassificationAlgorithm for Data Streams, Berlin, Heidelberg: Springer BerlinHeidelberg 3721 (2005), 108–120.

31.

Martha

A.C.

, Roseberry, Multi-label kNN classifier with self adjusting memory for drifting data streams, Proc of Machine Learning Research 94 (2018), 23–37.

32.

Leite

, Costa

, Gomide

, Evolving granular neural networks from fuzzy data streams, Neural Networks 38 (2013), 1–16.

33.

Torsten Hothorn

B.L.

, Double-bagging: combining classifiers by bootstrap aggregation, Pattern Recognition 36 (2003), 1303–1309.

34.

Ghimire

, Lee

, Extreme learning machine ensemble using bagging for facial expression recognition, Journal of Information Processing Systems 10(3) (2014), 443–458.

35.

Avidan

, Spatial Boost: Adding Spatial Reasoning to AdaBoost, Heidelberg: Springer Berlin Heidelberg, (2006), 386–396.

36.

Tkachenko

, Izonin

, Kryvinska

, et al., An Approach towards Increasing Prediction Accuracy for the Recovery of Missing LoT Data based on the GRNN-SGTM Ensemble, Sensors 20(9) (2020).

37.

Izonin

, Tkacheno

, Verhun

, Zub

, An Approach towards missing data management using improved GRNN-SGTM ensemble method, Engineering Science an Technology 24 (2021), 749–759.

38.

Ortiz-Díaz

A.A.

, Baldo

, Mariño

L.M.P.

, et al., Fast Adaptive Stacking of Ensembles for Supporting Active Learning, Procof the 14th International Conference on Nature Computation, (2018), 732–738.

39.

Ding

W.M.

, Wu

S.L.

, A cross-entropy based stacking method in ensemble learning, Journal Intelligence & Fuzzy System 39(3) (2020), 4677–4688.

40.

Izonin

, Tkachenko

, Vitynskyi

, Tkachenko

, Dronyuk

, Stacking-based GRNN-SGTM Ensemble Model for Prediction Tasks, 2020 International Conference on Decision Aid Sciences and Application (DASA), (2020), 326–330.

41.

Yang

, Zhang

Y.S.

, Ding

W.Z.

, Multiple Heterogeneous P-DCNNs Ensemble With Stacking Algorithm: A Novel Recognition Method of Space Target ISAR Images Under the Condition of Small Sample Set, IEEE Access 8 (2020), 75543–75570.

42.

Cortes

, Mohri

, Syed

, Deep Boosting, Proc of the 31st International Conference on Machine Learning, 32 (2014), 1179–1187.

43.

Bifet

, Frank

, Holmes

, Pfahringer

, Ensembles of Restricted Hoeffding Trees, Trans. Intell. Syst. Technol. 3(2) (2012), 1–20.

44.

Liberati

, et al., The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration, J. Clin. Epidemiol. 62(10) (2009), 1–28.

45.

Polikar

, Upda

S.S.

, et al., Learn++: an incremental learning algorithm for supervised neural networks, IEEE Transactions on Systems, Man and Cybernetics 31(4) (2001), 497–508.

46.

Minku

F.L.

, Inoue

, Yao

, Negative correlation in incremental learning, Natural Computing 8(2) (2009), 289–320.

47.

Zhao

Q.L.

, Jiang

Y.H.

, Xu

, Incremental Learning by Heterogeneous Bagging Ensemble, Proc of Advanced Data Mining & Applications-international Conference, Berlin: Springer, 6441 (2010), 1–12.

48.

Kidera

, Ozawa

, Abe

, An Incremental Learning Algorithm of Ensemble Classifier Systems, Proc of 2006 IEEE International Joint Conference on Neural Network Proceedings, Vancouver, Canada: IEEE (2006), 3421–3427.

49.

Saffari

, Leistner

, Santner

J.M.

, et al., On-line Random Forests, Proc of 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops, Kyoto: IEEE (2009), 1393–1400.

50.

Zhai

J.H.

, Wang

J.G.

, Ensemble online sequential extreme learning machine for large data set classification, Proc of 2014 IEEE International Conference on Systems, Man, and Cybernetics (SMC), San Digeo: IEEE (2014), 2250–2255.

51.

Wang

H.X.

, Wei

, Yu

P.S.

, et al., Mining concept-drifting data streams using ensemble classifiers, Proc of the ninth ACM SIGKDD international conference on Knowledge discovery and data mining, New York: Association for Computing Machinery, (2003), 226–235.

52.

Deckert

, Batch Weighted Ensemble for Mining Data Streams with Concept Drift, Berlin: Springer, (2011), 290–299.

53.

Brzezinski

, Stefanowski

, Reacting to Different Types of Concept Drift: The Accuracy Updated Ensemble Algorithm, IEEE Transactions on Neural Networks and Learning Systems 25(1) (2014), 81–94.

54.

Junior

R.B.J.

, Nicoletti

M.D.C.

, An iterative boosting-based ensemble for streaming data classification, Information Fusion 45 (2019), 66–78.

55.

Abdualrhman

M.A.A.

, Padma

M.C.

, Deterministic concept drift detection in ensemble classifier based data stream classification process, International Journal of Grid and High Performance Computing 11(1) (2019), 29–48.

56.

Minku

L.L.

, Yao

, DDD: A New Ensemble Approach for Dealing withConcept Drift, IEEE Transactions on Knowledge and DataEngineering 24(4) (2012), 619–633.

57.

Sidhu

, Bhatia

M.P.S.

, An online ensembles approach for handling concept drift in data streams: diversified online ensembles detection, International Journal of Machine Learning and Cybernetics 6(6) (2015), 883–909.

58.

Ancy

, Paulraj

, Online learning model for handling different concept drifts using diverse ensemble classifiers on evolving data streams, Cybernetics and Systems 50(7) (2019), 579–608.

59.

Sidhu

, Bhatia

M.P.S.

, A novel online ensemble approach to handle concept drifting data streams: diversified dynamic weighted majority, International Journal of Machine Learning and Cybernetics 9(1) (2018), 37–6.

60.

Finlay

, Pears

and Connor

A.M,

, SMOTE for Predicting Software Build Outcomes, Proc of 26th International Conference on Software Engineering and Knowledge Engineering, Vancouver: Knowledge Systems Institute Graduate School (2014), 546–551.

61.

Wang

, Minku

L.L.

, Yao

, Resampling-based ensemble methods for online class imbalance learning, IEEE Transactions on Knowledge and Data Engineering 27(5) (2015), 1356–1368.

62.

Ancy

, Paulraj

, Handling imbalanced data with concept drift by applying dynamic sampling and ensemble classification model, Computer Communications 153 (2020), 553–560.

63.

Zyblewski

, Sabourin

, Woźniak

, Preprocessed dynamic classifier ensemble selection for highly imbalanced drifted data streams, Information Fusion 66 (2021), 138–154.

64.

Zhang

, Liu

, Wang

, et al., Resample-based ensemble framework for drifting imbalanced data streams, IEEE Access 7 (2019), 65103–65115.

65.

Sun

, Kamel

M.S.

, Wong

A.K.C.

, et al., Cost-sensitive boosting for classification of imbalanced data, Pattern Recognition 40 (2007), 3358–3378.

66.

Tao

, Li

, Guo

, et al., Self-adaptive cost weights-based support vector machine cost-sensitive ensemble for imbalanced data classification, Information Sciences 487 (2019), 31–56.

67.

Wong

M.L.

, Seng

, Wong

P.K.

, Cost-sensitive ensemble of stacked denoising autoencoders for class imbalance problems in business domain, Expert Systems with Applications 141 (2020).

68.

Loezer

, Enembreck

, Barddal

, et al., Cost-sensitive learning for imbalanced data streams, Proc of the 35th Annual ACM Symposium on Applied Computing, New York: ACM 7 (2020), 498–504.

69.

Read

, Pfahringer

, Holmes

, et al., Classifier chains for multi-label classification, Machine Learning 85(3) (2011), 333–359.

70.

Wang

, Shen

, Tian

, Weighted ensemble classification of multi-label data streams, Advances in Knowledge Discovery and Data Mining 10235 (2017), 551–562.

71.

Nguyen

T.T.T.

, Nguyen

T.T.

, et al., An Online Variational Inference and Ensemble Based Multi-label Classifier for Data Streams, The Eleventh International Conference on Advanced Computational Intelligence, Guilin: IEEE (2019), 302–307.

72.

Sun

, Shao

, Wang

, Efficient ensemble classification for multi-label data streams with concept drift, Information 10(5) (2019).

73.

Chu

, Li

P.P.

, Hu

X.G.

, Co-training Based on Semi-Supervised Ensemble Classification Approach, Proc of the 2019 International Conference on Big Knowledge, (2019), 58–65.

74.

Bjerring

, Frank

, Beyond Trees: Adopting MITI to Learn Rules and Ensemble Classifiers for Multi-Instance Data, Berlin: Springer, (2011), 41–50.

75.

Zhang

, Zhou

, Ensembles of Multi-Instance Neural Networks, Boston: Springer, (2005), 471–474.

76.

Babenko

, Yang

, Belongie

, Robust Object Tracking with Online Multiple Instance Learning, IEEE Transactions on Pattern Analysis and Machine Intelligence 33(8) (2011), 1619–1632.

77.

Wang

, Yoon

, Xie

S.J.

, et al., Visual tracking with semi-supervised online weighted multiple instance learning, The Visual Computer 32(3) (2016), 307–320.

78.

Trofimov

, Sokolov

, Shavkunov

, et al., Distributed Classification of Text Streams: Limitations, Challenges, and Solutions, Proceedings of the International Workshop on Real-Time Business Intelligence and Analytics, Los Angeles: ACM (2019), 1–6.

79.

Song

, Ye

, Zhang

M.H.J.

, et al., Dynamic clustering forest: an ensemble framework to efficiently classify textual data stream with concept drift, Information Sciences 357 (2016), 125–143.

80.

Yang

, Zhang

, Li

, Classifying text streams by keywords using classifier ensemble, Data & Knowledge Engineering 70(9) (2011), 775–793.

81.

Pan

S.R.

, Zhang

, Li

, Dynamic classifier ensemble for positive unlabeled text stream classification, Knowledge Information System 33 (2012), 267–287.

82.

Upadhyay

, Nguyen

T.T.

, Massie

, et al., WEC: Weighted Ensemble of Text Classifiers, 2020 IEEE Congress on Evolutionary Computation, Glasgow: IEEE (2020), 1–8.

83.

Samami

, Soure

E.M.

, Binary classification of Lupus scientific articles applying deep ensemble model on text data, Proc of 7th International Conference on Digital Information Processing and Communications, Trabzon: IEEE (2019), 12–17.

84.

Aggarwal

C.C.

, Yao

, Yu

P.S.

, On Supervised Change Detection in Graph Streams, Proc of the 2020 International Conference on Data Mining, Cincinnati: SIAM (2020), 289–297.

85.

Aggarwal

C.C.

, “On Classification of Graph Streams, Proc of the Eleventh International Conference on Data Mining Society for Industrial and Applied Mathematics, Mesa: SIAM (2011), 652–663.

86.

Pan

, Wu

, Zhu

, et al., Graph ensemble boosting for imbalancednoisy graph stream classification, IEEE Trans Cybern 45(5) (2015), 940–954.

87.

Liu

, Song

, Zhao

, et al., Manifold-preserving sparse graph-based ensemble FDA for industrial label-noise fault classification, IEEE Transactions on Instrumentation and Measurement 69(6) (2020), 2621–2634.

88.

, Rousu

, Multilabel classification through random graph ensembles, Machine Learning 99(2) (2015), 231–256.

89.

Lin

, Wu

, Lin

, et al., An ensemble random forest algorithm for insurance big data analysis, IEEE Access 5 (2017), 16568–16575.

90.

Mwangi

P.I.

, Nderu

, Mwigereri

, A Stacked Ensemble Model based on RUSBoost and a Cost-Sensitive Convolutional Neural Network for Class Imbalance in Big Data Analytics, Proc of the 2020 African Conference on Software Engineering, Nairobi: CEUR-WS, 2689 (2020).

91.

Marron

, Ayguade

, Herrero

J.R.

, et al., Low-latency multi-threaded ensemble learning for dynamic big data streams, Proc of International Conference on Big Data, Boston: IEEE (2017), 223–232.

92.

Denham

, Pears

, Naeem

M.A.

, HDSM: A distributed data mining approach to classifying vertically distributed data streams, Knowledge-Based Systems 189 (2020).

93.

Gama

, Medas

, Rocha

, Forest Trees for On-line Data, Proc of the 2004 Symposium on Applied Computing, Nicosia: ACM 632–636 (2004).

94.

Bifet

, Efficient Online Evaluation of Big Data Stream Classifiers,” Proc of the 21th International Conference on Knowledge Discovery and Data Mining, Sydney: ACM, (2015), 59–68.

95.

Gomes

H.M.

, Bifet

, Read

, et al., Adaptive random forests for evolving data stream classification, Machine Learning 106(9) (2017), 1469–149.

96.

Grzenda

, Gomes

H.M.

, Bifet

, Delayed labelling evaluation for data streams, Data Mining and Knowledge Discovery 34(5) (2020), 1237–1266.

97.

Bifet

, Holmes

, Pfahringer

, et al., Fast Perceptron Decision Tree Learning from Evolving Data Streams, Berlin: Springer, (2010), 299–310.

98.

Kubat

, Holte

R.C.

, Matwin

, Machine learning for the detection of oil spills in satellite radar images, Machine Learning 30(2) (1998), 195–215.

99.

Santos

A.M.

, Canuto

A.M.P.

, Neto

A.F.

, Evaluating classification methods applied to multi-label tasks in different domains, Proc of 10th International Conference on Hybrid Intelligent System, Atlanta: IEEE (2010), 61–66.

100.

Shaker

, Hüllermeier

, Recovery analysis for adaptive learning from non-stationary data streams: Experimental design and case study,” Neurocomputing (Amsterdam), 150 (2015), 250–264.

An overview of complex data stream ensemble classification

Abstract

Keywords

1 Overview

2.1 Data streams

3.1 Steady data streams

3.1.1 Method based on block learning

3.1.2 Method based on online learning

3.1.3 Chapter summary

Table 1 Algorithms comparison Algorithm Datasets Accuracy Time GNCL [46] Mushroom 98.87-99.48 74.98 Bagging++ [47] Mushroom 93.90-99.94 9.17 OzaBag [21] CovType 74.71 16.05 LevBag [3] CovType 87.81 6.98

3.2.1 Method based on block learning

3.2.3 Chapter summary

Table 2 Algorithms comparison Algorithm Datasets Accuracy Time AWE [51] SEA 87.73 0.67 AUE2 [53] SEA 89.19 0.61 DWM [5] Hyper 89.94 0.21 OAUE [2] Hyper 90.43 0.22

3.3.1 Data preprocessing method

3.3.2 Cost-sensitive learning method

Table 3 Confusion matrix Category Positive Negative Positive C (+ , +) C (+ , -) Negative C (- , +) C (- , -)

Table 4 Algorithms comparison Algorithm Datasets AUC Accuracy UOB [61] PAKDD 60.7 42.4 OOB [61] PAKDD 63.5 70.7 ARF [68] PAKDD 63.2 80.1 RE-DI [64] PAKDD 65.8 80.3

3.4.1 Multi-label data streams

3.4.2 Multi-instance data streams

Table 5 Algorithms comparison Algorithm Datasets Hamming loss Subset accuracy MLOzaBag [72] Random Tree 0.0934 0.689 ML-SAM-knn [72] Random Tree 0.0541 0.550 MHT [72] Random Tree 0.0614 0.574 MLAW [72] Random Tree 0.0135 0.792

4.1 Text data streams

4.2 Graph data streams

4.3 Big data streams

5 Complex data streams evaluation methods

Table 7 Confusion matrix Category Positive sample Negative sample Positive Sample True Positive (TP) False Negative (FN) Negative Sample False Positive (FP) True Negative (TN)

6 Research directions and prospects

7 Conclusion

Footnotes

Acknowledgments

References

Table 1
Algorithms comparison

Algorithm Datasets Accuracy Time

GNCL [46] Mushroom 98.87-99.48 74.98

Bagging++ [47] Mushroom 93.90-99.94 9.17

OzaBag [21] CovType 74.71 16.05

LevBag [3] CovType 87.81 6.98

Table 2
Algorithms comparison

Algorithm Datasets Accuracy Time

AWE [51] SEA 87.73 0.67

AUE2 [53] SEA 89.19 0.61

DWM [5] Hyper 89.94 0.21

OAUE [2] Hyper 90.43 0.22

Table 3
Confusion matrix

Category Positive Negative

Positive C (+ , +) C (+ , -)

Negative C (- , +) C (- , -)

Table 4
Algorithms comparison

Algorithm Datasets AUC Accuracy

UOB [61] PAKDD 60.7 42.4

OOB [61] PAKDD 63.5 70.7

ARF [68] PAKDD 63.2 80.1

RE-DI [64] PAKDD 65.8 80.3

Table 5
Algorithms comparison

Algorithm Datasets Hamming loss Subset accuracy

MLOzaBag [72] Random Tree 0.0934 0.689

ML-SAM-knn [72] Random Tree 0.0541 0.550

MHT [72] Random Tree 0.0614 0.574

MLAW [72] Random Tree 0.0135 0.792

Table 7
Confusion matrix

Category Positive sample Negative sample

Positive Sample True Positive (TP) False Negative (FN)

Negative Sample False Positive (FP) True Negative (TN)