Safe semi-supervised classification algorithm combined with active learning sampling strategy

Abstract

In order to improve the performance of semi-supervised learning, a kind of safe semi-supervised classification algorithm based active learning sampling strategy is proposed. First, an active learning sampling method based on uncertainty and representativenes is designed. The weighted algorithm combining the uncertainty and representativenesss is used to select the unlabeled samples with rich information and representation, providing for semi-supervised learning. Second, a method of label prediction based on grouping verification is designed. Prelabeling is executed on unlabeled sample selected by active learning. The sample with pseudo-label is added into the labeled sample set to carry out grouping, training and testing. The corresponding errors of various pseudo-labels are calculated and the pseudo-label making the accuracy least is selected as the candidate label of the unlabeled sample. Third, a method of security verification is designed. Only the label making the accuracy lower than before is selected as the final label of the unlabeled sample to expand the number of labeled samples. Iterations are repeatedly executed until a certain precision is met. Finally, the classifier is trained using the final labeled set. The experiments are carried out on semi-supervised datasets and UCI datasets, and the results show that the proposed algorithms are effective.

Keywords

Active learning sample semi-supervised learning safety label prediction grouping verification

1 Introduction

Semi-supervised learning [1, 2] is a kind of new machine learning method between supervised learning and unsupervised learning, whose purpose is to make full use of large unlabeled samples to make up for the lack of labeled samples. Semi-supervised learning is divided into semi-supervised clustering and semi-supervised classification. The main aim of semi-supervised classification is to study how to use a large number of unlabeled samples to help to train supervised learning classifier, when labeled samples are insufficient.

Active learning and semi-supervised learning face the same problems and challenges. They use labeled samples to build highly accurate classifiers, and reduce the workload of manually marking unlabeled samples. The common point between active learning and semi-supervised learning is to mark unlabeled samples and expand the number of labeled samples [3, 4]. The major difference between them is as follow: (1) The semi-supervised learning algorithm realizes the automatic labeling of the unlabeled samples, and active learning is artificial to mark the unlabeled samples by experts; (2) The unlabeled samples are screened by active learning and only the samples with abundant information are marked.

For semi-supervised learning, the calculation of the classification model is mainly dependent on a small number of labeled samples and a large number of unlabeled samples. A small number of labeled samples are mainly used for the initialization of the classification model, and its classification performance will affect the performance of the final classification model. For semi-supervised classification algorithm, it is necessary to extract labeled samples according to the data distribution and mark the unlabeled samples with abundant information as much as possible, which is an effective way to improve the accuracy of the initial classification model [5, 6]. So it requires the combination of active learning and semi-supervised learning, and active learning should select the sample with high information to provide for semi-supervised learning.

Some scholars combined active learning with semi-supervised learning and proposed several improved semi- supervised learning algorithm. By embedding active learning into semi-supervised learning, namely active semi-supervised learning, the generalization performance of classifier can be effectively improved. Zhang [7] applied co-training to select the most reliable instances according to the two criterions of high confidence and nearest neighbor for boosting the classifier, exploit the most informative instances with human annotation for improving the classification performance. He [8] combined active learning and semi-supervised learning to obtain a confident and sufficient labeled training data for multivariate time series classification. Chang [9] proposed a new active learner learning pairwise constraints known as must-link and cannot-link constraints, it combined semi-supervised clustering and active learning. Reitmaier [6] provided a transductive active learning, a completely labeled data pool was provided in each active learning cycle. It used this generative model to label all samples not labeled by an expert so far in order to train the kind of classifier training with the active learning process. Yan [10] introduced active learning in the process of semi-supervised learning. Some samples have been selected by artificial markers, and these new added labeled samples have been used to optimize semi-supervised learning. Hajmohammadi [11] proposed a learning model based on the combination of uncertainty-based active learning and semi-supervised self-training approaches to incorporate unlabeled sentiment documents from the target language in order to improve the performance of cross-lingual methods.

However, the existing semi-supervised classification algorithms combined with active learning still have many problems to be improved, and the main problems are as follows: (1) The sample selected by active learning only has uncertainty or representativeness, and it can‘t provide effective unlabeled samples for semi-supervised learning. (2) They do not make full use of semi-supervised learning to label the unlabeled samples automatically, and the unlabeled samples selected by active learning are all manually labeled by experts. (3) In the process of semi-supervised learning, some noise may be introduced to reduce the performance of data classification.

In order to improve the performance of semi-supervised classifier, a kind of safe semi-supervised classification algorithm based active learning sampling strategy (S3CA-AL for short) was proposed in this paper. On the whole, S3CA-AL was divided into three modules: active learning sampling, label prediction and security verification. (1) In active learning sampling module, an active learning sampling method based on uncertainty and representativeness (AL-ur for short) is designed. The weighted algorithm is used to select the unlabeled samples with information and representation, providing for semi-supervised learning. (2) In label prediction module, a method of label prediction based on grouping verification (LP-gv for short) is designed. The pseudo-label making the classifier error least is selected as the candidate label. (3) In security verification, a method of security verification (SV for short) is designed. The candidate label is verified and only the label making the classification rate lower than before is selected as the final label of the unlabeled sample. The three modules are composed and iterations are repeatedly executed until a certain precision is met. Finally, the experiment is executed to verify the effectiveness of the algorithm.

The remainder of the present study is structured as follows: In Section 2, methods of previous studies on active learning and semi-supervised learning are summarized. In Section 3, our proposed algorithm is presented. In Section 4, the experiment and the discussion are presented. In Section 5, the conclusion is given.

2 Related work

2.1 Active learning

Active learning was first proposed by Professor Angluin [12] of the Yale University in the United States, the main idea is to use the unlabeled samples to help train the classifier. The method is to select and mark some unlabeled samples, and put them into labeled samples set to train the classifier. Then the classifier is used to select unlabeled samples again, and iterations are repeated.

The iterative process of the active learning algorithm is as follows [12]: (1) The classifier is trained in the labeled sample set. (2) The classifier is used to classify the unlabeled samples. (3) According to the classification results, the sampling engine is used to select unlabeled samples to be given to experts for marking and the sample will be marked to a labeled sample set for the next classifier training. (4) The newly added labeled samples are added to the labeled sample set for the next training. The termination condition of the algorithm is that the marking cost or the generalization accuracy of the classifier reaches a certain standard [13 –15]. The algorithm description is shown in Algorithm 1.

Algorithm 1. Active learning algorithm
Input: The labeled training set L;
The unlabeled training set U;
Learning engine (LE);
Sample engine (SE)
Output: Learning engine (LE)
Process:
For i from 1 to N
Train (LE,L); /Training the classifier LE on the labeled dataset L/
T = Test(LE,U); /Using the classifier LE to predict the data in the unlabeled sample set U/
S = Select(SE,U\|T); /Using the sampling engine SE to select unlabeled samples/
Label (S); /Marking the selected unlabeled sample S/
L = L + S; /the newly labeled sample is added into L/
U = U–S; /Remove the labeled sample from the U/
End For

Active learning is different from semi-supervised learning only using existing data, but not using external information. In active learning, the classifier usually picks out some unlabeled samples to obtain its label from experts, then use these newly added labeled samples to learn a classifier again. This process will circulate until a satisfactory classifier is achieved. Research on active learning usually focuses on reducing the number of queries, to obtain better classification results with less labeled data [14, 15].

In active learning, there are two strategies for selecting and querying samples. One of strategies is based on the information content of samples, that is, the sample can reduce the degree of uncertainty of the classification model in statistical learning. The other strategy is based on the representation of samples, that is, whether the sample can represent the whole data set. Most of the active learning techniques are based on one of these two strategies to select samples. Huang [16] has put forward a systematic way to combine these two selection strategies, so as to select informative and representative unlabeled samples to query.

2.2 Semi-supervised learning

The basic idea of semi-supervised learning is to use the model hypothesis in the data distribution to establish a learner to mark unlabeled samples. At present, there are three commonly basic model assumptions in semi- supervised learning: Cluster Assumption [17, 18], Manifold Assumption [19, 20] and Local & Global Consistency Assumption hypothesis [21]. Semi-supervised classification uses a large number of unlabeled data to expand the training set of the classification algorithm to make up for of the lack of labeled data. Semi-supervised classification mainly includes disagreement-based methods, generative methods, discriminative methods and graph-based methods.

The disagreement-based semi-supervised classification realizes the utilization of unlabeled data by using multiple classifiers. In the learning process, the unlabeled data is used as a platform for information interaction between multiple classifiers. The original disagreement-based algorithm was developed by A. Blum and T. Mitchell [22] in 1998, or called the standard collaborative training algorithm. They assumed that the data set had two views of sufficient redundancy, meeting the following conditions: First, each set of attributes was sufficient to describe the problem; second, each attribute set was conditioned to be independent of another set of attributes when it was marked. The collaborative training process iterated until a stop condition was reached. The description of the algorithm is shown in Algorithm 2.

Algorithm 2. Semi-supervised classification algorithm Co-training
Input: The labeled training set L;
The unlabeled training set U;
Supervised classification algorithm S.
Output: Semi-supervised classifier
Process:
Create a pool U‘ of examples by choosing u examples at random from U;
Loop for k iterations:
Use L to train a classifier h1 that considers only the x1 portion of x;
Use L to train a classifier h2 that considers only the x2 portion of x;
Allow h1 to label p positive and n negative examples from U‘;
Allow h2 to label p positive and n negative examples from U‘;
Add these self-labeled examples to L;
Randomly choose 2p + 2n examples from U to replenish U‘.

The generative method [23] assumes that the sample and class labels are generated by a set of probability distributions of a certain or certain structural relationship. Class prior distribution P(y) and class conditional distribution P (x|y) have been known, repeated sampling y∼p (y) and x∼p (x|y). From these distributions, the sample L with the class label and the sample U without the class label are generated. Then, the posteriori distribution P (y|x) is obtained according to the theorem of probability, and the class label which makes the largest P (y|x) is found to mark the x. Graph-based learning [24] is a very active direction of semi-supervised learning in recent years. The essence of the graph based approach is the label propagation. It describes the sample space by graph, and uses neighbors to spread label information in point cloud. The discriminative method [25] uses the maximum interval algorithm to train the learning decision boundary of the labeled sample and unlabeled sample. The purpose of learning is to make the classification hyperplane through the low density data region, and to make the distance maximum between the classification hyperplane and the nearest sample. The discriminant methods include LDA, generalized discriminant analysis, semi-supervised support vector machines, entropy regularization and KNN.

3 Our method

Here, we propose a kind of safe semi-supervised classification algorithm based active learning sampling strategy (S3CA-AL for short). It includes three modules: active learning sampling module, label prediction module and security verification module. In the following content, firstly we introduce the overall framework of S3CA-AL, and then introduce the function and process of each module.

3.1 Algorithm framework

The framework diagram of S3CA-AL is shown in Fig. 1. The first step is the process of using active learning to sample. Active learning strategy is used to select informative and representative sample from the unlabeled sample set, and these selected samples are used as candidate samples to be labeled by semi-supervised learning algorithm. The second step is label prediction. A method of label prediction based on grouping verification is used to predict the candidate samples above as a predicted label for unlabeled samples. The third step is the process of the security verification. For the predicted label of unlabeled sample from the second step, second verification is carried out to confirm the final label for the unlabeled sample, making the classifier error rate lower than before. The fourth step is the process of new sample addition. The newly label and its corresponding sample are added into the training set L to provide the next step training for active learning and semi-supervised learning. Iterations are repeatedly executed until a certain precision is met. Finally, the final classifier is formed by training the supervised classification algorithm using the final labeled set L.

Fig.1

Algorithm framework.

Through active learning, the space of unlabeled samples is reduced, providing semi-supervised learning with abundant informative unlabeled samples, shortening the time of semi-supervised learning, and improving the performance of semi-supervised learning. On the other hand, semi-supervised learning is used to extract information from unlabeled samples and improve the reliability of labeled samples information for active learning. At the same time, the whole process is completed automatically without the need of manual participation.

3.2 Active learning sampling module

In active learning, there are two strategies for selecting and querying samples. One is based on the information content of samples, that is, the sample can reduce the degree of uncertainty of the classification model in statistical learning. The other is based on the representation of samples, that is, whether the sample can represent the whole data set. Most of the active learning techniques are based on one of these two strategies to select samples.

Here a method is used to combine these two selection strategies, and an active learning sampling method based on uncertainty and representativeness (AL-ur for short) is proposed. The weighted algorithm is used to combine uncertainty and representativeness, so that both informative and representative unlabeled samples are selected for marking. The definition of the weighted algorithm is shown in the formula (1): $W (x) = α \times V (x) + (1 - α) \times R (x)$ (1)

Where V (x) represents the voting entropy of the sample x, that is the uncertainty; R (x) represents the representativeness of the sample x, W (x) represents the weighted value of uncertainty and representativeness, α represents the weighing parameter, and α ∈ [0, 1].

The sample density [24] proposed by Zhu is used to measure the representativeness of the samples, its formalization is defined as formula (2). $R (x) = \frac{\sum_{s_{i} \in s (x)} sim (x, s_{i})}{k}$ (2) where sim (x_i, x_j) represents the user similarity that is standardized to [0,1], it is calculated by the standardized Pearson correlation coefficient. s (x) = {s₁, s₂,…, s_k} represents the K sample with the highest similarity with the sample x.

In the process of active learning, the representativeness of samples is considered. If the sample is outlier and the representation is too small, even if the uncertainty is large enough, it will not be selected as a candidate sample. Through active learning sampling, high informative and representative unlabeled samples are selected and marked by predictive module, and the scale of labeled samples is expanded.

3.3 Label prediction module

In this module, a method of label prediction based on grouping verification (LP-gv for short) is proposed. The main idea of the algorithm is as follows. First, prelabeling is carried out for unlabeled sample selected by active learning. The sample with pseudo-label is added into the labeled sample set. Then, grouping, training and testing are executed in the labeled set, and the corresponding errors of various pseudo-labels are calculated. At last, the pseudo-label with the lowest classification rate is selected as the candidate label of the unlabeled sample for next step judgement.

The algorithm description is shown in Algorithm 3. First, an unlabeled sample set U is selected. Different pseudo-labels are respectively classified according to different classification categories and mark the selected sample. Second, the selected sample and its corresponding pseudo-label are added into the labeled set to form a new training set. The training set is grouped into multiple groups. One of group acts as validation set and the other groups are used as training sets. The training set is used to train the classifier and test is carried out on the validation set. Cross validation is performed many times, allowing each group to act as a validation set. Then calculate the average value of the corresponding error rate after each cross validation. Third, the error values of the classifier are calculated based on various pseudo-labels, and the corresponding pseudo-label with the lowest error values are selected as the final predictive value of the sample.

Algorithm 3. Label prediction algorithm
Input: Labeled sample set L;
Unlabeled sample set U;
Supervised classifier
Output: The candidate label e_x of the unlabeled sample w after the prediction
Progress:
For i = 1 to m /* m is the number of classified categories for data*/
Use pseudo-label M_i to mark w in U, record as w_{i ;}
Add w_i into L to form training set;
Group L into k groups, record as L(1),L(2)…,L(k);
For j = 1 to k
Take L(j) as validation set, the other k-1groups as training set
Using a training set to train a supervised classifier C_ij
The supervised classifier C_ij is used to test the validation set
Record the verifying accuracy at this time as acc_i (j)
EndFor
Calculate the average accuracy: acc_i = (acc_i(1) + acc_i (2) +…+ acc_i (k))/k
Calculate error e_i = 1–acc_i
EndFor
Get the minimum value e_x in e₁,e₂,…,e_m
e_x is used as the candidate predictive value of the sample w

3.4 Security verification module

In order to improve the accuracy and safety of new labeled samples for classifier, a method of security verification(SV for short) is proposed to judge the new added label above. Only the candidate label that meets the criteria of judgment can be used as final label to mark the unlabeled w and expand the labeled set L. $e^{t} < e^{t - 1}$ (3)

Where e^t represents the classification error of the classifier in the t iteration, e^t-1 represents the classification error of the classifier in the t-1 iteration.

The judging condition is as shown in formula (3). It assumes that the newly added candidate label and its corresponding sample are added into the labeled set L to form L‘. The training set L is used to train the classifier, which is used to test on the test set, and then calculate the classification e. At the same time, the training set L‘ is used to train the classifier, which is used to test on the test set, and calculate the classification e‘. Judge whether it satisfies the formula (3).

If the formula (3) is satisfied, the candidate sample is used as a new sample of L, otherwise the candidate sample will be eliminated directly. In this way, it can guarantee that the error rate of the current round is reduced compared with that of the previous one when the new sample is added into L. It can ensure that the classifier evolves in the direction of improving the classification rate. It can avoid the increase of unlabeled samples to make the classifier performance worse and ensure that the classification algorithm is safe and reliable.

4 Experiment

4.1 Experimental platform and data set

The proposed approaches are evaluated on data set including three semi-supervised benchmark data sets digit1, g241c, coil, and six UCI data sets. The size of data ranges from 270 to 1500, and the dimensionality ranges from 8 to 241. The data set is shown in Table 1.

Table 1
The data set

Data Sets Dimension # Instance

# positive # negative # total

australian 42 383 307 690

breastw 9 239 444 683

coil 241 750 750 1500

diabetes 8 268 500 768

digit1 241 734 766 1500

g241c 241 750 750 1500

heart 9 120 150 270

isolet 51 300 300 600

optdigits 42 572 571 1143

Data Sets	Dimension	# Instance
australian	42	383	307	690
breastw	9	239	444	683
coil	241	750	750	1500
diabetes	8	268	500	768
digit1	241	734	766	1500
g241c	241	750	750	1500
heart	9	120	150	270
isolet	51	300	300	600
optdigits	42	572	571	1143

4.2 Evaluation metrics and experiment scheme

Here, the classification is a two classification problem, which is divided into normal and abnormal. Four metrics are typically applied to evaluate the results.

True Positive (TP): It is judged to be a positive sample, and in fact it is a positive sample.

True Negative (TN): It is judged to be a negative sample and in fact it is a negative sample.

False Negative (FN): It is judged to be a negative sample, but in fact it is a positive sample.

False Positive (FP): It is judged to be a positive sample, but in fact it is a negative sample.

Because the data set is balanced, the classification accuracy is used as an important evaluation index to directly evaluate the entire prediction performance of the proposed method. Based on these four metrics above, the classification accuracy is defined using the following formula (4): $Accuracy = \frac{TP + TN}{TN + FN + TP + FP} \times 100 %$ (4)

In order to verify the effectiveness of the proposed algorithm, the experiment is divided into the following four categories.

Experiment 1. In order to verify the validity of the label prediction module, LP-gv is used to predict unlabeled samples, and compare with other traditional prediction algorithms to verify its effectiveness.

Experiment 2. In order to verify the validity of the label prediction module and the security verification module, LP-gv and SV are formed into a whole to act as the classifier of self-training semi-supervised learning (SSC-LP-SV for short). In SSC-LP-SV, the unlabeled samples are marked by LP-gv and SV. The newly added label and its corresponding sample are added into labeled set, expanding the number of labeled set. Iterations are repeatedly carried out until the unlabeled sample set is empty. The final labeled set is used to train the supervised classifier and the test set is used to verify the performance of SSC-LP-SV. At last, compared with other traditional semi-supervised classification algorithms, the validity of SSC-LP-SV is verified.

Experiment 3. In order to verify the validity of the semi-supervised classification algorithm S3CA-AL and verify of the effectiveness of the combination of active learning and semi-supervised learning, experiment is carried out to compare S3CA-AL with SSC-LP-SV proposed in experiment 2. The main purpose of the experiment is to determine whether the performance of S3CA-AL and SSC-LP-SV is similar. Because S3CA-AL is trained by a small number of samples, but SSC-LP-SV is trained by a large number of samples.

Experiment 4. In order to verify the effectiveness of active learning sampling in this algorithm, we compare the proposed algorithm AL-ur with other active learning methods to verify whether our active learning algorithm performs better than other active learning algorithms.

4.3 Results and analysis

4.3.1 Results and analysis of experiment 1

In the experiment, LP-gv was used to predict and classify unlabeled samples, and compared with other traditional classification algorithms, such as SVM, decision tree and Logistic Regression. In the experiment, the training set and the test set are assigned according to 1 : 1, and the training set is used to train the classifier to carry out classification on the test set and calculate the classification rate in formula (4).

Experiment result is shown in Table 2. It can be seen that the accuracy of the algorithm LP-gv is better than SVM, decision tree and logistic regression, indicating that the proposed algorithm LP-gv has higher accuracy and better classification performance.

Table 2
Comparison of accuracy (mean-std.)

Data Sets Classification algorithm

SVM Decision tree Logistic regression LP-gv

australian 75.8±8.6 74.3±7.5 74.0±6.8 77.5±5.3

breastw 95.2±2.0 92.5±3.8 91.7±3.7 96.6±2.4

coil 58.5±6.1 56.4±5.4 57.0±6.8 61.8±4.2

diabetes 65.5±5.4 65.0±2.4 65.3±8.7 66.9±3.4

digit1 76.6±5.4 74.5±4.6 78.8±7.5 82.8±5.7

g241c 60.8±2.8 57.4±3.5 62.7±4.7 78.4±3.8

heart 72.1±6.5 70.7±7.2 67.8±4.9 74.8±4.5

isolet 92.9±3.3 90.5±2.1 89.6±1.5 98.5±0.8

optdigits 95.4±2.3 91.3±1.5 95.8±2.1 98.8±0.7

Data Sets	Classification algorithm
australian	75.8±8.6	74.3±7.5	74.0±6.8	77.5±5.3
breastw	95.2±2.0	92.5±3.8	91.7±3.7	96.6±2.4
coil	58.5±6.1	56.4±5.4	57.0±6.8	61.8±4.2
diabetes	65.5±5.4	65.0±2.4	65.3±8.7	66.9±3.4
digit1	76.6±5.4	74.5±4.6	78.8±7.5	82.8±5.7
g241c	60.8±2.8	57.4±3.5	62.7±4.7	78.4±3.8
heart	72.1±6.5	70.7±7.2	67.8±4.9	74.8±4.5
isolet	92.9±3.3	90.5±2.1	89.6±1.5	98.5±0.8
optdigits	95.4±2.3	91.3±1.5	95.8±2.1	98.8±0.7

4.3.2 Results and analysis of experiment 2

In this experiment, a semi-supervised experiment is simulated. SSC-LP-SV proposed in experiment 2 is compared with the traditional semi-supervised classification algorithm, such as SVM, S3VM [26] and S4VM [26].

The selected samples are divided into two parts: the training set and the test set. The proportion of the sample number of the training set and the test set is set to 2 : 1. At the same time, the training set is divided into labeled samples and unlabeled samples. According to the number of labeled samples accounting for the proportion of the number of the training set, three types of semi-supervised classification data set is constructed. In the first category of experiment, labeled samples accounts for 5% of training set samples (λ= 5%). In the second category of experiment, labeled samples accounts for 10% of training set samples (λ= 10%). In the third category of experiment, labeled samples accounts for 15% of training set samples (λ= 15%). According to the proportion, three kinds of experiments are carried out to verify the performance of the proposed algorithm. The condition of the algorithm termination is that the unlabeled sample set is empty. The experiments are repeated 30 times, and the performance of several classification algorithms is compared by calculating the mean value.

The results of the experiment are shown in Tables 3, 4 and 5. Among them, Table 3 shows the semi-supervised classification result when the percent is 5%. Table 4 shows the semi-supervised classification result when the percent is 10%. Table 5 shows the semi-supervised classification result when the percent is 15%.

Table 3
Comparison of accuracy (mean-std.) λ= 5%

Data Sets Classification algorithm

SVM S3VM S4VM SSC-LP-SV

australian 68.1±3.2 72.3±3.5 73.8±2.3 75.0±2.5

breastw 88.2±2.0 90.2±5.3 92.8±5.4 93.1±1.8

coil 54.4±6.1 53.1±3.7 57.4±1.3 60.4±5.6

diabetes 61.9±4.2 60.7±1.4 64.2±2.5 65.9±6.0

digit1 73.6±4.6 78.2±2.7 80.4±4.5 81.8±3.3

g241c 56.8±3.4 70.8±6.1 74.2±3.6 77.3±2.1

heart 67.3±2.1 70.4±2.7 72.1±4.6 71.7±3.2

isolet 87.6±4.3 93.7±5.4 94.4±3.3 96.2±5.4

optdigits 88.4±7.5 93.4±4.5 95.2±0.5 96.8±1.5

Data Sets	Classification algorithm
australian	68.1±3.2	72.3±3.5	73.8±2.3	75.0±2.5
breastw	88.2±2.0	90.2±5.3	92.8±5.4	93.1±1.8
coil	54.4±6.1	53.1±3.7	57.4±1.3	60.4±5.6
diabetes	61.9±4.2	60.7±1.4	64.2±2.5	65.9±6.0
digit1	73.6±4.6	78.2±2.7	80.4±4.5	81.8±3.3
g241c	56.8±3.4	70.8±6.1	74.2±3.6	77.3±2.1
heart	67.3±2.1	70.4±2.7	72.1±4.6	71.7±3.2
isolet	87.6±4.3	93.7±5.4	94.4±3.3	96.2±5.4
optdigits	88.4±7.5	93.4±4.5	95.2±0.5	96.8±1.5

Table 4

Comparison of accuracy (mean-std.) λ= 10%

Data Sets	Classification algorithm
	SVM	S3VM	S4VM	SSC-LP-SV
australian	75.0±3.2	75.3±6.2	76.2±7.1	77.0±2.5
breastw	90.2±2.0	93.1±5.3	94.8±2.6	95.7±1.8
coil	55.4±6.1	59.8±3.7	60.0±1.3	61.4±5.6
diabetes	62.0±4.2	63.7±0.3	65.8±2.5	65.9±6.0
digit1	73.6±4.6	76.8±4.2	81.5±7.2	82.0±3.3
g241c	56.0±3.4	72.7±3.5	75.4±2.0	77.8±2.1
heart	69.3±2.1	71.6±5.5	72.0±5.0	72.7±3.2
isolet	89.6±4.3	94.7±5.4	96.5±2.8	97.2±5.4
optdigits	91.4±7.5	95.4±4.5	96.5±0.5	97.8±1.5

Table 5

Comparison of accuracy (mean-std.) λ= 15%

Data Sets	Classification algorithm
	SVM	S3VM	S4VM	SSC-LP-SV
australian	75.5±4.8	76.0±4.3	76.8±2.3	77.5±4.4
breastw	92.5±1.3	93.2±5.8	94.8±5.4	96.6±7.1
coil	58.4±6.1	60.1±4.6	60.4±6.2	61.8±3.6
diabetes	65.0±3.6	65.7±1.4	65.9±2.5	66.7±2.5
digit1	77.0±4.7	78.2±2.7	82.4±4.5	82.5±3.9
g241c	63.8±3.4	73.8±6.1	76.2±3.6	78.0±2.7
heart	69.3±2.1	71.9±2.7	72.8±4.6	74.8±1.6
isolet	90.8±0.3	94.7±5.4	97.4±3.3	98.5±1.2
optdigits	94.4±1.4	95.8±6.2	98.2±4.3	98.5±2.4

It can also be seen from Tables 3–5: (1) As the number of labeled samples increases, the accuracy of semi-supervised learning is improved. This situation can be reflected in SVM, S3VM, S4VM and our algorithm SSC-LP-SV. It indicates that the more the labeled samples are, the stronger the classification ability of the generated classifier is. (2) SSC-LP-SV uses a cross grouping method to mark and extend unlabeled samples and verify two times. Compared with SVM, S3VM and S4VM, SSC-LP-SV has a higher accuracy. It indicates that our algorithm SSC-LP-SV is an effective semi-supervised classification algorithm, which can extend the number of labeled samples well and improve the accuracy.

It can also be seen in Table 3 that the S3VM classification algorithm has a decline in accuracy as the introduction of noise. In the experiment, the reduction of accuracy has not appeared in SSC-LP-SV. It is because that SSC-LP-SV selects the candidate label with minimum error to mark the sample, and the algorithm evolves towards the direction of classifier performance improvement. So SSC-LP-SV is a safe semi-supervised classification algorithm.

4.3.3 Results and analysis of experiment 3

In this experiment, the selection method of data sets is the same as experiment 2. In the process of the experiment, the semi-supervised classification process is simulated. The condition of the algorithm termination is that the unlabeled sample set is empty. The experiment was done 30 times, and the average value was calculated as the result of the experiment.

In the experiment, algorithm S3CA-AL with active learning sampling is compared with algorithm SSC-LP-SV without active learning sampling, so as to judge whether the combination of active learning and semi- supervised learning can improve the classification performance rather than cause the decline of classification performance.

The results of the experiment are shown in Tables 6, 7 and 8. Among them, Table 6 shows the semi-supervised classification result when the percent of labeled sample is 5%. Table 7 shows the semi-supervised classification result when the percent of labeled sample is 10%. Table 8 shows the semi-supervised classification result when the percent of labeled sample is 15%.

Table 6
Comparison of accuracy (mean-std.) 5%

Data Sets Classification algorithm

SVM SSC-LP-SV S3CA-AL

australian 68.1±3.2 75.0±2.5 75.0±3.1

breastw 88.2±2.0 93.1±1.8 92.9±0.9

coil 54.4±6.1 60.4±5.6 60.0±1.4

diabetes 61.9±4.2 65.9±6.0 65.5±3.5

digit1 73.6±4.6 81.8±3.3 81.0±7.8

g241c 56.8±3.4 77.3±2.1 76.8±2.1

heart 67.3±2.1 71.7±3.2 71.0±4.7

isolet 87.6±4.3 96.2±5.4 96.2±5.4

optdigits 88.4±7.5 96.8±1.5 96.8±1.5

Data Sets	Classification algorithm
australian	68.1±3.2	75.0±2.5	75.0±3.1
breastw	88.2±2.0	93.1±1.8	92.9±0.9
coil	54.4±6.1	60.4±5.6	60.0±1.4
diabetes	61.9±4.2	65.9±6.0	65.5±3.5
digit1	73.6±4.6	81.8±3.3	81.0±7.8
g241c	56.8±3.4	77.3±2.1	76.8±2.1
heart	67.3±2.1	71.7±3.2	71.0±4.7
isolet	87.6±4.3	96.2±5.4	96.2±5.4
optdigits	88.4±7.5	96.8±1.5	96.8±1.5

Table 7

Comparison of accuracy (mean-std.) 10%

Data Sets	Classification algorithm
	SVM	SSC-LP-SV	S3CA-AL
australian	75.0±3.2	77.0±2.5	76.5±3.2
breastw	90.2±2.0	95.7±1.8	95.0±1.6
coil	55.4±6.1	61.4±5.6	62.0±0.9
diabetes	62.0±4.2	65.9±6.0	65.0±3.4
digit1	73.6±4.6	82.0±3.3	81.6±7.8
g241c	56.0±3.4	77.8±2.1	77.2±3.5
heart	69.3±2.1	72.7±3.2	72.5±2.3
isolet	89.6±4.3	97.2±5.4	97.8±3.2
optdigits	91.4±7.5	97.8±1.5	98.0±0.7

Table 8

Comparison of accuracy (mean-std.) 15%

Data Sets	Classification algorithm
	SVM	SSC-LP-SV	S3CA-AL
australian	75.5±4.8	77.5±4.4	77.0±2.0
breastw	92.5±1.3	96.6±7.1	96.2±4.1
coil	58.4±6.1	61.8±3.6	61.2±8.7
diabetes	65.0±3.6	66.7±2.5	66.2±1.8
digit1	77.0±4.7	82.5±3.9	83.6±4.2
g241c	63.8±3.4	78.0±2.7	78.2±5.2
heart	69.3±2.1	74.8±1.6	74.8±1.6
isolet	90.8±0.3	98.5±1.2	98.5±1.2
optdigits	94.4±1.4	98.5±2.4	98.6±2.4

It can be seen from the experimental results: Although only a few part of the labeled sample is used for training classifier, the accuracy of S3CA-AL is very close to SSC-LP-SV, and even higher. For example, the accuracy of data set isolet and optdigits in Table 7 have been improved, and the accuracy of data set digit1, g241c and optdigits in Table 8 have also been improved. It is because that S3CA-AL uses active learning to select high informative and representative samples to mark, and it can improve classification performance to a certain extent. On the premise of reducing the scale of semi-supervised learning, it can maintain the accuracy of the algorithm and even improve the accuracy to some extent.

4.3.4 Results and analysis of experiment 4

In the experiment, the proposed active learning algorithm AL-ur is compared with exiting algorithm QBC [13] (the query by committee) and exiting algorithm SUD [27] (sampling by uncertainty and density) to estimate its performance. Three kinds of active learning AL-ur, QBC and SUD are combined respectively with semi-supervised algorithm SSC-LP-SV proposed in experiment 2, to form three semi-supervised classification algorithms combined with active learning: S3CA-AL, QBC_SSC-LP-SV, QBC_SSC-LP-SUD. Three kinds of active learning are used to sampling separately, and SSC-LP-SV is used to judge the sampling performance of the three active learning algorithms [28 –30].

In the experiment, labeled samples are constantly added into the training set. With the increasing number of labeled samples, the accuracy of the three semi-supervised classification algorithm is calculated respectively and the change trend of accuracy is judged.

Compared with the increase rate and stability value of the accuracy among three algorithms, we can find that S3CA-AL proposed in this paper has faster and better effect than other active learning methods on classification performance. The result on Australian, breastw, diabetes, heart and isolet dataset shows that the performance of S3CA-AL is stable when the labeled set reaches 10% of the total sample. The result on Coil, digit1, g241c and optdigits dataset shows that the performance of S3CA-AL is stable when the labeled set reaches 6% of the total sample, and only a small fluctuation occurs. Meanwhile, we can know that the active learning AL-ur proposed in this paper has better performance, and it also shows that the application of active learning algorithm to semi-supervised learning can solve the problem of label shortage [31 –34].

5 Conclusion

In this paper, the active learning algorithm and semi-supervised learning are combined to realize data classification. Active learning is used to select unlabeled samples with abundant and representative information. Semi-supervised learning is used to expand the number of labeled samples. The whole process is the automatic completion of the algorithm, and no manual intervention is needed. In the semi-supervised learning process, the security of the classifier is emphasized.

Several simulation experiments are carried out on the proposed algorithm and several modules. The experiment shows that the proposed algorithm can effectively reduce the size of the sample and it is easier to mark the unlabeled samples. The result also shows that the proposed algorithm has good classification performance.

Footnotes

Acknowledgments

The Project was supported by Natural Science Basic Research Plan in Shaanxi Province of China (No. 2015JM6347), the Science and Technology Research Project of Shangluo University (No. 14SKY026), Scientific Research Project of Education Department in Shaanxi Province of China (No. 09JK424).

References

Zhou

Z.H.

and Li

, Semi-supervised learning by disagreement, Knowledge & Information Systems24(3) (2010), 415–439.

Zhu

and Goldberg

, Introduction to semi-supervised learning, Morgan & Claypool37(1) (2014), 3036–3036.

Settles

, Active learning, Synthesis Lectures on Artificial Intelligence and Machine Learning6(1) (2012), 1–114.

Criminisi

, Shotton

and Konukoglu

, Decision forests: A unified framework for classification, regression, density estimation, manifold learning and semi-supervised learning, Foundations and Trends^® in Computer Graphics and Vision7(2–3) (2012), 81–227.

Balcan

, Berlind

, Ehrlich

et al., Efficient semi-supervised and active learning of disjunctions, Proceedings of the 30th International Conference on Machine Learning2013, pp. 633–641.

Reitmaier

, Calma

and Sick

, Transductive active learning – A new semi-supervised learning approach based on iteratively refined generative models to capture structure in data, Information Sciences293(293) (2015), 275–298.

Zhang

, Wen

, Wang

et al., Semi-supervised learning combining co-training with active learning, Expert Systems with Applications41(5) (2014), 2372–2378.

, Li

and Zhao

, An uncertainty and density based active semi-supervised learning scheme for positive unlabeled multivariate time series classification, Knowledge-Based Systems124 (2017), 80–92.

Chang

C.C.

and Lin

P.Y.

, Active learning for semi-supervised clustering based on locally linear propagation reconstruction, Neural Networks63 (2015), 170–184.

10.

Leng

, Xu

and Qi

, Combining active learning and semi-supervised learning to construct SVM classier, Knowledge-Based Systems44 (2013), 121–131.

11.

Hajmohammadi

M.S.

, Ibrahim

, Selamat

et al., Combination of active learning and self-training for cross-lingual sentiment classification with density analysis of unlabelled samples, Information Sciences317(C) (2015), 67–77.

12.

Angluin

, Queries and concept learning, Machine Learning2(4) (1988), 319–342.

13.

Settles

, Active learning literature survey, University of Wisconsin, Madison52 (2010), 55–66.

14.

Tuia

, Volpi

, Copa

et al., A survey of active learning algorithms for supervised remote sensing image classification, Selected Topics in Signal Processing, IEEE Journal of5(3) (2011), 606–617.

15.

Balcan

M.F.

, Hanneke

and Vaughan

J.W.

, The true sample complexity of active learning, Machine Learning80(2-3) (2010), 111–139.

16.

Huang

S.-J.

, Jin

and Zhou

Z.-H.

, Active learning by querying informative and representative examples, IEEE Transactions on Pattern Analysis and Machine Intelligence36(10) (2014), 1936–1949.

17.

Wang

, Meng

, Fu

et al., Towards Safe semi-supervised Classification: Adjusted Cluster Assumption via Clustering, Neural Processing Letters46(3) (2017), 1–12.

18.

C.J.

and Yang

Y.P.

, A batch-mode active learning SVM method based on semi-supervised clustering, Intelligent Data Analysis19(2) (2015), 345–358.

19.

Chen

and Wang

, Semi-supervised learning via regularized boosting working on multiple semi-supervised assumptions, IEEE Transactions on Pattern Analysis & Machine Intelligence33(1) (2010), 129–143.

20.

Niyogi

, Manifold regularization and semi-supervised learning: Some theoretical analyses, Journal of Machine Learning Research14(1) (2013), 1229–1250.

21.

Bao

B.K.

, Min

, Li

et al., Joint local and global consistency on interdocument and interword relationships for co-clustering, IEEE Transactions on Cybernetics45(1) (2014), 15–28.

22.

Blum

and Mitchell

, Combining labeled and unlabeled data with co-training, Conference on Computational Learning Theory1998, pp. 92–100.

23.

Jiang

, Zhang

and Zeng

, A hybrid generative/discriminative method for semi-supervised classification, Knowledge-Based Systems37(2) (2013), 137–145.

24.

Zhuang

, Zhou

, Gao

et al., Label information guided graph construction for semi-supervised learning, IEEE Transactions on Image Processing26(9) (2017), 4182–4192.

25.

Appice

, Guccione

and Malerba

, Transductive hyperspectral image classification: Toward integrating spectral and relational features via an iterative ensemble system, Machine Learning103(3) (2016), 343–375.

26.

Ding

, Zhu

and Zhang

, An overview on semi-supervised support vector machine, Neural Computing & Applications28 (2017), 1–10.

27.

Zhu

, Wang

, Ma

et al., Active learning with sampling by uncertainty and density for data annotations, IEEE Transactions on Audio Speech & Language Processing18(6) (2010), 1323–1331.

28.

Shen

, Mi

and Zhang

, A positioning lockholes of container corner castings method based on image recognition, Polish Maritime Research24(SI) (2017), 95–101.

29.

, Liu

, Furuta

and Peng

, Characteristics of activated carbon remove sulfur particles against smog, Saudi Journal of Biological Sciences24(6) (2017), 1370–1374.

30.

Yilmaz

A.E.

and Aktas

, Ridit and exponential type scores for estimating the kappa statistic, Kuwait Journal of Science45(1) (2018), 89–99.

31.

Chen

, Mean square exponential stability of uncertain singular stochastic systems with discrete and distributed delays, Journal of Interdisciplinary Mathematics20(1) (2017), 13–26.

32.

Feng

, Shu-Rong

, Hui

, Hong-Ping

and Jian-Zhong

, An empirical research on the influence of Chinese rural financial reform on cultivation of new agricultural business entities, Journal of Discrete Mathematical Sciences & Cryptography20(1) (2017), 389–405.

33.

Sun

, Varankina

V.I.

and Sadovaya

V.V.

, Didactic aspects of the academic discipline “history and methodology of mathematics”, Eurasia Journal of Mathematics Science and Technology Education13(7) (2017), 2923–2940.

34.

Plata

S.A.

and Sáez

S.B.

, After notes on Chebyshev’s iterative method, Applied Mathematics and Nonlinear Sciences2 (2017), 1–12.

Data Sets	Dimension	# Instance
		# positive	# negative	# total
australian	42	383	307	690
breastw	9	239	444	683
coil	241	750	750	1500
diabetes	8	268	500	768
digit1	241	734	766	1500
g241c	241	750	750	1500
heart	9	120	150	270
isolet	51	300	300	600
optdigits	42	572	571	1143

Data Sets	Classification algorithm
	SVM	Decision tree	Logistic regression	LP-gv
australian	75.8±8.6	74.3±7.5	74.0±6.8	77.5±5.3
breastw	95.2±2.0	92.5±3.8	91.7±3.7	96.6±2.4
coil	58.5±6.1	56.4±5.4	57.0±6.8	61.8±4.2
diabetes	65.5±5.4	65.0±2.4	65.3±8.7	66.9±3.4
digit1	76.6±5.4	74.5±4.6	78.8±7.5	82.8±5.7
g241c	60.8±2.8	57.4±3.5	62.7±4.7	78.4±3.8
heart	72.1±6.5	70.7±7.2	67.8±4.9	74.8±4.5
isolet	92.9±3.3	90.5±2.1	89.6±1.5	98.5±0.8
optdigits	95.4±2.3	91.3±1.5	95.8±2.1	98.8±0.7