Multi-view learning based on maximum margin of twin spheres support vector machine

Abstract

Multi-view learning utilizes information from multiple representations to advance the performance of categorization. Most of the multi-view learning algorithms based on support vector machines seek the separating hyperplanes in different feature spaces, which may be unreasonable in practical application. Besides, most of them are designed to balanced data, which may lead to poor performance. In this work, a novel multi-view learning algorithm based on maximum margin of twin spheres support vector machine (MvMMTSSVM) is introduced. The proposed method follows both maximum margin principle and consensus principle. By following the maximum margin principle, it constructs two homocentric spheres and tries to maximize the margin between the two spheres for each view separately. To realize the consensus principle, the consistency constraints of two views are introduced in the constraint conditions. Therefore, it not only deals with multi-view class-imbalanced data effectively, but also has fast calculation efficiency. To verify the validity and rationlity of our MvMMTSSVM, we do the experiments on 24 binary datasets. Furthermore, we use Friedman test to verify the effectiveness of MvMMTSSVM.

Keywords

Multi-view learning twin spheres SVM maximum margin consensus principle

1 Introduction

With the advent of the big data era, the types of data are becoming more and more diverse. For the same object, the data collected by different measurements or different media is called multi-view (MV) data [1]. Multiple representations or views can afford complementary information for the algorithms, thus increasing the classification performance. Multi-view learning (MVL) is a new emerging and fast developing direction in the machine learning community. The corresponding algorithms are of great interest for applications in medical diagnosis [2], face recognition [3], human emotion recognition [4], action recognition [5], document classification [6], and so on.

In the early research, there are two ways to deal with multi-view data: 1) regardless of the differences between data from different views, the features are combined together and sing-view learning method is used to train a model; 2) the classifier is trained for each view separately, and then the corresponding results are integrated and optimized by utilizing the relevance of different views [7, 8]. Each of them has pros and cons. Later, many researchers have found that constructing MVL algorithms using multi-views jointly can get better results than the aforementioned two methods [9, 10].

Most MVL methods fall into the following three types [1]: co-training style algorithms [11 –13], co-regularization style algorithms [14 –17] and margin-consistency style algorithms [18, 19]. In order to solve different machine learning tasks, some algorithms have been put forward, such as MV clustering [20 –22], MV transfer learning [23, 24], MV feature selection [25, 26], MV graph embedding [27], MV ensemble learning [28, 29], etc. Besides, some scholars have done thorough analysis on the generalization error bounds based on the theory of machine learning, such as Rademacher complexity [30], PAC-Bayes bounds [31], and so forth.

The support vector machine (SVM) [32] tries to seek a separating hyperplane to classify the samples of two classes. It has already outperformed other machine learning methods due to its good properties. To improve its computational speed or generalization performance, different kinds of SVM-based algorithms have emerged. One of the types is the sphere-based algorithm which looks for the hypersphere rather than the hyperplane to classify. In 2005, the core vector machine (CVM) [33] was firstly proposed by introducing the core set. CVM uses the minimal enclosing ball approximation algorithm, which can massively reduce the computational time. Inspired by the thought of twin SVM [34], twin hypersphere SVM (THSVM) [35] seeks two independent balls to describe the samples. THSVM is especially suitable for the samples of two class belonging to different Gaussian distributions. Different from THSVM, the small sphere and large margin approach (SSLM) [36] seeks two homocentric spheres, and it has unique advantages of dealing with class-imbalance classification problem. However, SSLM needs to solve a large quadratic programming problem (QPP) to gain the optimal solutions, which makes it have expensive calculation cost. Therefore, a maximum margin of twin spheres SVM (MMTSSVM) [37] is raised to further improve this issue.

Till now, many SVM-based MVL algorithms have been studied. Most of them belong to co-regularization style algorithms, such as SVM-2K [14], MV-SVM-2C [38], MV privileged SVM [30, 39], MV nonparallel SVM [7], MV generalized SVM [40], etc. Nevertheless, most of the aforementioned SVM-based MVL algorithms seek the hyperplanes in different feature spaces, which may be unreasonable for real applications. In addition, when the data is unbalanced, their performance may become worse. To overcome the above defects, we propose a novel MVL algorithm based on MMTSSVM (MvMMTSSVM) in this paper, and its key characteristics are listed as follows:

The proposed MvMMTSSVM can make full use of the coherence and the diversity of multiple representations by following the principle of margin maximization and the consensus principle.

Observing the principle of margin maximization, the proposed MvMMTSSVM constructs two homocentric spheres for each view separately, and tries to maximize the distance of the two homocentric spheres in each view respectively.

The absolute value constraints combined the two representations can make sure the implementation of the consensus principle.

The optimal solutions of MvMMTSSVM are obtained just by solving a smaller-scale QPP and a linear programming problem (LPP).

The proposed method not only inherits the fast training speed of MMTSSVM for data imbalance classification problem, but also deals with the MVL problem effectively and efficiently. Therefore, it is more suitable for solving practical problems.

The remainder of the paper is arranged as follows. In Section 2, we outline the basics of MMTSSVM. In Section 3, we give the modeling process, solving process, and algorithm flow of the proposed MvMMTSSVM. In Section 4, we perform the experiments to testify the performance of MvMMTSSVM. In the end, the conclusion is made in Section 5.

2 Related works

For the sake of simplicity, we use the following notations. Let the training set be ${(x_{i}, y_{i})}_{i = 1}^{l} \subseteq X \times Y$ , where $x_{i} \in ℝ^{n}$ is the sample and $X$ denotes the feature space, y_i ∈ {+1, - 1} is corresponding label and $Y$ is the label space, l stands for the number of samples. Let l₁ and l₂ denote the number of positive and negative samples, respectively, and l₁ + l₂ = l.

From the perspective of MVL, the MMTSSVM can be seen as a single-view algorithm. It has achieved good results in handling imbalanced classification problem. The primal formulations of MMTSSVM are as follows, $\begin{matrix} min_{R^{2}, C, ξ_{i}} & R^{2} - \frac{ν}{l_{2}} \sum_{j \in I_{2}} ∥ ϕ (x_{j}) - C ∥^{2} + \frac{1}{ν_{1} l_{1}} \sum_{i \in I_{1}} ξ_{i} \\ s . t . & ∥ ϕ (x_{i}) - C ∥^{2} \leq R^{2} + ξ_{i}, \\ ξ_{i} \geq 0, i \in I_{1} \end{matrix}$ (1) and $\begin{matrix} min_{ρ^{2}, η_{j}} & R^{2} - ρ^{2} + \frac{1}{ν_{2} l_{2}} \sum_{j \in I_{2}} η_{j} \\ s . t . & ∥ ϕ (x_{j}) - C ∥^{2} \geq R^{2} + ρ^{2} - η_{j}, \\ η_{j} \geq 0, j \in I_{2} \end{matrix}$ (2) where ϕ (x_i) maps sample x_i into a higher-dimensional feature space. ν, ν₁ and ν₂ are given parameters. C is the center of the two homocentric spheres. R and $\sqrt{R^{2} + ρ^{2}}$ are the radiuses of the small sphere and large sphere, respectively. I₁ and I₂ stand for the positive set and negative set, respectively.

One can assign the label of new testing point x by the following equation, $f (x) = {\begin{matrix} 1, & if ∥ ϕ (x) - C ∥ < \frac{R + \sqrt{R^{2} + ρ^{2}}}{2} \\ - 1, & else \end{matrix}$ (3) which means if the distance of x from the center C is less than the average radiuses of two homocentric spheres, the label of x will be assigned to class +1, otherwise, it will be assigned to class -1.

3 The proposed MvMMTSSVM

As MMTSSVM has several overwhelming advantages over other algorithms, we will give the detailed process of MVL algorithm based on MMTSSVM in the following.

Suppose ${(x_{i}^{A}, x_{i}^{B}, y_{i})}_{i = 1}^{l} \subseteq X_{A} \times X_{B} \times Y$ is a two-view dataset, where $X_{A}$ and $X_{B}$ are two-view feature spaces, the superscripts A and B of x_i are used to distinguish the two views of sample x_i. Let $ϕ (x_{i}^{A})$ and $ϕ (x_{i}^{B})$ denote the hi-dimensional nonlinear feature mappings. Constructing two homocentric spheres on views A and B respectively and making use of the relevance of two representations by the consistency constraints, the proposed MvMMTSSVM can be built as follows:

$\begin{matrix} min & R_{A}^{2} - \frac{ν_{A}}{l_{2}} \sum_{j \in I_{2}} ∥ ϕ (x_{j}^{A}) - C_{A} ∥^{2} \\ + R_{B}^{2} - \frac{ν_{B}}{l_{2}} \sum_{j \in I_{2}} ∥ ϕ (x_{j}^{B}) - C_{B} ∥^{2} \\ + \frac{1}{ν_{1}^{A} l_{1}} \sum_{i \in I_{1}} ξ_{i}^{A} + \frac{1}{ν_{1}^{B} l_{1}} \sum_{i \in I_{1}} ξ_{i}^{B} + D_{1} \sum_{i \in I_{1}} η_{i} \\ s . t . & | ∥ ϕ (x_{i}^{A}) - C_{A} ∥^{2} - ∥ ϕ (x_{i}^{B}) - C_{B} ∥^{2} \\ - R_{A}^{2} + R_{B}^{2} | \leq ε + η_{i}, \\ | | ϕ (x_{i}^{A}) - C_{A} | |^{2} \leq R_{A}^{2} + ξ_{i}^{A}, \\ | | ϕ (x_{i}^{B}) - C_{B} | |^{2} \leq R_{B}^{2} + ξ_{i}^{B}, \\ η_{i} \geq 0, ξ_{i}^{A} \geq 0, ξ_{i}^{B} \geq 0, i \in I_{1}, \end{matrix}$ (4) and

$\begin{matrix} min & R_{A}^{2} + R_{B}^{2} - ρ_{A}^{2} - ρ_{B}^{2} + \frac{1}{ν_{2}^{A} l_{2}} \sum_{j \in I_{2}} ξ_{j}^{A} \\ + \frac{1}{ν_{2}^{B} l_{2}} \sum_{j \in I_{2}} ξ_{j}^{B} + D_{2} \sum_{j \in I_{2}} μ_{j} \\ s . t . & | ∥ ϕ (x_{j}^{A}) - C_{A} ∥^{2} - ∥ ϕ (x_{j}^{B}) - C_{B} ∥^{2} \\ - R_{A}^{2} - ρ_{A}^{2} + R_{B}^{2} + ρ_{B}^{2} | \leq ε + μ_{j}, \\ ∥ ϕ (x_{j}^{A}) - C_{A} ∥^{2} \geq R_{A}^{2} + ρ_{A}^{2} - ξ_{j}^{A}, \\ ∥ ϕ (x_{j}^{B}) - C_{B} ∥^{2} \geq R_{B}^{2} + ρ_{B}^{2} - ξ_{j}^{B}, \\ μ_{j} \geq 0, ξ_{j}^{A} \geq 0, ξ_{j}^{B} \geq 0, j \in I_{2}, \end{matrix}$ (5) where $ν_{A}, ν_{B}, ν_{1}^{A}, ν_{1}^{B}, ν_{2}^{A}, ν_{2}^{B}$ , D₁ and D₂ are the parameters given in advance. C_A and C_B are the centers of spheres of view A and B, respectively. R_A and $\sqrt{R_{A}^{2} + ρ_{A}^{2}}$ are the radiuses of small sphere and large sphere on view A. R_B and $\sqrt{R_{B}^{2} + ρ_{B}^{2}}$ are the radiuses of small sphere and large sphere on view B. $ξ_{i}^{A}, ξ_{i}^{B}, i \in I_{1}$ and $ξ_{j}^{A}, ξ_{j}^{B}, j \in I_{2}$ are slack variables. In order to understand the model construction principle of MvMMTSSVM, we draw Fig. 1 to illustrate it.

Fig. 1

Illustration of MvMMTSSVM.

Under the assumption that the two views are equally important, the MvMMTSSVM aims to learn two decision functions for the two views, and its detailed explanations are as follows:

(1) The variables to be solved in problem (4) are $R_{A}^{2}, R_{B}^{2}, C_{A}, C_{B}, ξ_{i}^{A}, ξ_{i}^{B}, η_{i}, i \in I_{1}$ . The center and radius of small sphere in view A (resp. view B) are C_A (resp. C_B) and R_A (resp. R_B) respectively. The small sphere of view A (resp. view B) should contain as many positive points as possible. At the same time, the negative points are asked to stay as far away from the center C_A (resp. C_B) as possible. Namely, we need to minimize $R_{A}^{2} - \frac{ν_{A}}{l_{2}} \sum_{j \in I_{2}} ∥ ϕ (x_{j}^{A}) - C_{A} ∥^{2}$ (resp. $R_{B}^{2} - \frac{ν_{B}}{l_{2}} \sum_{j \in I_{2}} ∥ ϕ (x_{j}^{B}) - C_{B} ∥^{2}$ ). The second (resp. third) inequality constraint of (4) implies that the positive points of view A (resp. view B) should be contained in the small sphere, and the errors $ξ_{i}^{A}, i \in I_{1}$ (resp. $ξ_{i}^{B}, i \in I_{1}$ ) adopting the soft margin loss function are used to punish the positive points outside the small spheres. Minimizing $\frac{1}{ν_{1}^{A} l_{1}} \sum_{i \in I_{1}} ξ_{i}^{A}$ and $\frac{1}{ν_{1}^{B} l_{1}} \sum_{i \in I_{1}} ξ_{i}^{B}$ means the sum of error variables in representations A and B should be as small as possible.

(2) In problem (5), we need to obtain the variables $ρ_{A}^{2}, ρ_{B}^{2}, ξ_{j}^{A}, ξ_{j}^{B}, μ_{j}, j \in I_{2}$ . The center and radius of large sphere in view A (resp. view B) are C_A (resp. C_B) and $\sqrt{R_{A}^{2} + ρ_{A}^{2}}$ (resp. $\sqrt{R_{B}^{2} + ρ_{B}^{2}}$ ), respectively. Based on the principle of large margin mechanism, minimizing $- ρ_{A}^{2}$ and $- ρ_{B}^{2}$ in the objective function is equivalent to maximize the margin of two homocentric spheres in view A and B, respectively. The second and third constrains in (5) indicate that most negative points should locate outside the large sphere as far as possible in views A and B, respectively, and the errors $ξ_{j}^{A}, i \in I_{2}$ (resp. $ξ_{j}^{B}, i \in I_{2}$ ) judged by the soft margin loss function are used to punish the dissatisfying negative points. Minimizing $\frac{1}{ν_{2}^{A} l_{2}} \sum_{j \in I_{2}} ξ_{j}^{A}$ and $\frac{1}{ν_{2}^{B} l_{2}} \sum_{j \in I_{2}} ξ_{j}^{B}$ indicates minimizing the sum of errors from corresponding negative samples located inside the large sphere in views A and B, respectively.

(3) The first constraint of (4) (resp. (5)) makes the positive (negative) classifier of two views be consistent, which is also the embodiment of distance minimization of kernel canonical correlation analysis (KCCA). The existence of ε allows some points to violate the constraints appropriately. Therefore, the gap between the two views is shrunk into ε and the proposed model follows the consensus principle. The variables η_i, i ∈ I₁ (resp. μ_j, j ∈ I₂) are used to punish the positive (resp. negative) samples who fail to meet ε-similarity of the positive (resp. negative) classifiers.

(4) The MvMMTSSVM regards the positive class as majority class. In (4), minimizing variables $R_{A}^{2}$ and $R_{B}^{2}$ instead of R_A and R_B makes (4) be a convex QPP. Similarly, minimizing variables $ρ_{A}^{2}$ and $ρ_{B}^{2}$ instead of ρ_A and ρ_B can guarantee that (5) is a convex LPP.

In order to solve the problem (4), we can construct its Lagrangian function as follows, $\begin{matrix} L_{1} = R_{A}^{2} + R_{B}^{2} + \frac{1}{ν_{1}^{A} l_{1}} \sum_{i \in I_{1}} ξ_{i}^{A} + \frac{1}{ν_{1}^{B} l_{1}} \sum_{i \in I_{1}} ξ_{i}^{B} \\ - \frac{ν_{A}}{l_{2}} \sum_{j \in I_{2}} ∥ ϕ (x_{j}^{A}) - C_{A} ∥^{2} - \frac{ν_{B}}{l_{2}} \sum_{j \in I_{2}} ∥ ϕ (x_{j}^{B}) - C_{B} ∥^{2} \\ + \sum_{i \in I_{1}} γ_{i}^{+} (∥ ϕ (x_{i}^{A}) - C_{A} ∥^{2} - R_{A}^{2} - ∥ ϕ (x_{i}^{B}) - C_{B} ∥^{2} \\ + R_{B}^{2} - η_{i}) \\ - \sum_{i \in I_{1}} γ_{i}^{-} (∥ ϕ (x_{i}^{A}) - C_{A} ∥^{2} - R_{A}^{2} - ∥ ϕ (x_{i}^{B}) - C_{B} ∥^{2} \\ + R_{B}^{2} + η_{i}) \\ + \sum_{i \in I_{1}} α_{i}^{A} (| | ϕ (x_{i}^{A}) - C_{A} | |^{2} - R_{A}^{2} - ξ_{i}^{A}) \\ + \sum_{i \in I_{1}} α_{i}^{B} (| | ϕ (x_{i}^{B}) - C_{B} | |^{2} - R_{B}^{2} - ξ_{i}^{B}) \\ - \sum_{i \in I_{1}} β_{i}^{A} ξ_{i}^{A} - \sum_{i \in I_{1}} β_{i}^{B} ξ_{i}^{B} - \sum_{i \in I_{1}} τ_{i} η_{i} + D_{1} \sum_{i \in I_{1}} η_{i} \end{matrix}$ (6) where $γ_{i}^{+}, γ_{i}^{-}$ , $α_{i}^{A}, α_{i}^{B},$ $β_{i}^{A}, β_{i}^{B}, τ_{i} \geq 0$ are Lagrange multipliers.

Differentiating the Lagrangian function L₁ with respect to variables $R_{A}^{2}, R_{B}^{2}$ , $ξ_{i}^{A}, ξ_{i}^{B}$ , η_i, C_A, C_B, we can get the following Karush-Kuhn-Tucker (KKT) conditions: $\frac{\partial L_{1}}{\partial R_{A}^{2}} = 0 \Rightarrow \sum_{i \in I_{1}} (α_{i}^{A} + γ_{i}^{+} - γ_{i}^{-}) = 1,$ (7) $\frac{\partial L_{1}}{\partial R_{B}^{2}} = 0 \Rightarrow \sum_{i \in I_{1}} (α_{i}^{B} - γ_{i}^{+} + γ_{i}^{-}) = 1,$ (8) $\frac{\partial L_{1}}{\partial ξ_{i}^{A}} = 0 \Rightarrow α_{i}^{A} + β_{i}^{A} = \frac{1}{ν_{1}^{A} l_{1}},$ (9) $\frac{\partial L_{1}}{\partial ξ_{i}^{B}} = 0 \Rightarrow α_{i}^{B} + β_{i}^{B} = \frac{1}{ν_{1}^{B} l_{1}},$ (10) $\frac{\partial L_{1}}{\partial η_{i}} = 0 \Rightarrow γ_{i}^{+} + γ_{i}^{-} + τ_{i} = D_{1},$ (11) $\begin{matrix} \frac{\partial L_{1}}{\partial C_{A}} = 0 & \Rightarrow \frac{ν_{A}}{l_{2}} \sum_{j \in I_{2}} (ϕ (x_{j}^{A}) - C_{A}) \\ + \sum_{i \in I_{1}} δ_{i}^{A} (ϕ (x_{i}^{A}) - C_{A}) = 0, \end{matrix}$ (12) $\begin{matrix} \frac{\partial L_{1}}{\partial C_{B}} = 0 & \Rightarrow \frac{ν_{B}}{l_{2}} \sum_{j \in I_{2}} (ϕ (x_{j}^{B}) - C_{B}) \\ + \sum_{i \in I_{1}} δ_{i}^{B} (ϕ (x_{i}^{A}) - C_{B}) = 0 . \end{matrix}$ (13) where $δ_{i}^{A} = α_{i}^{A} + γ_{i}^{+} - γ_{i}^{-}, i \in I_{1},$ $δ_{i}^{B} = α_{i}^{B} - γ_{i}^{+} + γ_{i}^{-}, i \in I_{1}$ . Eqs.(7) and (8) are also equivalent to

$\sum_{i \in I_{1}} δ_{i}^{A} = \sum_{i \in I_{1}} δ_{i}^{B} = 1 .$ (14)

Because $β_{i}^{A} \geq 0$ , $β_{i}^{B} \geq 0, i \in I_{1}$ , from Eqs.(9) and (10), we can obtain

$0 \leq α_{i}^{A} \leq \frac{1}{ν_{1}^{A} l_{1}}, 0 \leq α_{i}^{B} \leq \frac{1}{ν_{1}^{B} l_{1}} .$ (15)

Since τ_i ≥ 0, i ∈ I₁, from Eq.(11), we can get

$0 \leq γ_{i}^{+}, γ_{i}^{-}, γ_{i}^{+} + γ_{i}^{-} \leq D_{1} .$ (16)

From Eqs. (12) and (13), we can derive the centers of two homocentric spheres of view A and view B, respectively, where $\begin{matrix} C_{A} = & \frac{1}{1 - ν_{A}} \cdot (- \frac{ν_{A}}{l_{2}} \sum_{j \in I_{2}} ϕ (x_{j}^{A}) + \sum_{i \in I_{1}} δ_{i}^{A} ϕ (x_{i}^{A})), \\ C_{B} = & \frac{1}{1 - ν_{B}} \cdot (- \frac{ν_{B}}{l_{2}} \sum_{j \in I_{2}} ϕ (x_{j}^{B}) + \sum_{i \in I_{1}} δ_{i}^{B} ϕ (x_{i}^{B})), \end{matrix}$ then the values 〈C_A· C_A 〉 and 〈C_B· C_B 〉 can be obtained, namely

$\begin{matrix} 〈 C_{A} \cdot C_{A} 〉 \\ = & \frac{1}{(1 - ν_{A})^{2}} \cdot (\frac{ν_{A}^{2}}{l_{2}^{2}} \sum_{j \in I_{2}} \sum_{j \in I_{2}} K (x_{j}^{A}, x_{j}^{A}) \\ + \sum_{i \in I_{1}} \sum_{i \in I_{1}} δ_{i}^{A} δ_{i}^{A} K (x_{i}^{A}, x_{i}^{A}) \\ - \frac{2 ν_{A}}{l_{1}} \sum_{i \in I_{1}} \sum_{j \in I_{2}} δ_{i}^{A} K (x_{i}^{A}, x_{j}^{A})), \end{matrix}$ (17) and

$\begin{matrix} 〈 C_{B} \cdot C_{B} 〉 \\ = & \frac{1}{(1 - ν_{B})^{2}} \cdot (\frac{ν_{B}^{2}}{l_{2}^{2}} \sum_{j \in I_{2}} \sum_{j \in I_{2}} K (x_{j}^{B}, x_{j}^{B}) \\ + \sum_{i \in I_{1}} \sum_{i \in I_{1}} δ_{i}^{B} δ_{i}^{B} K (x_{i}^{B}, x_{i}^{B}) \\ - \frac{2 ν_{B}}{l_{1}} \sum_{i \in I_{1}} \sum_{j \in I_{2}} δ_{i}^{B} K (x_{i}^{B}, x_{j}^{B})) . \end{matrix}$ (18)

Taking the values of C_A and C_B into Eq. (6) and using Eqs.(7), (8), (14), (15) and (16), omitting the terms irrelevant to $δ_{i}^{A}$ and $δ_{i}^{B}$ , we can derive the dual formulation of QPP (4) as follows, $\begin{matrix} max_{α_{i}^{A / B}, γ_{i}^{+ / -}} & \frac{1}{1 - ν_{A}} (- \sum_{i_{1} \in I_{1}} \sum_{i_{2} \in I_{1}} δ_{i_{1}}^{A} δ_{i_{2}}^{A} K (x_{i_{1}}^{A}, x_{i_{1}}^{A}) \\ + \frac{2 ν_{A}}{l_{2}} \sum_{i \in I_{1}} δ_{i}^{A} \sum_{j \in I_{2}} K (x_{j}^{A}, x_{i}^{A})) \\ + \frac{1}{1 - ν_{B}} (- \sum_{i_{1} \in I_{1}} \sum_{i_{2} \in I_{1}} δ_{i_{1}}^{B} δ_{i_{2}}^{B} K (x_{i_{1}}^{B}, x_{i_{1}}^{B}) \\ + \frac{2 ν_{B}}{l_{2}} \sum_{i \in I_{1}} δ_{i}^{B} \sum_{j \in I_{2}} K (x_{j}^{B}, x_{i}^{B})) \\ + \sum_{i \in I_{1}} δ_{i}^{A} K (x_{i}^{A}, x_{i}^{A}) + \sum_{i \in I_{1}} δ_{i}^{B} K (x_{i}^{B}, x_{i}^{B}) \\ s . t . & \sum_{i \in I_{1}} δ_{i}^{A} = \sum_{i \in I_{1}} δ_{i}^{B} = 1, \\ 0 \leq α_{i}^{A} \leq \frac{1}{ν_{1}^{A} l_{1}}, 0 \leq α_{i}^{B} \leq \frac{1}{ν_{1}^{B} l_{1}}, \\ 0 \leq γ_{i}^{+}, γ_{i}^{-}, γ_{i}^{+} + γ_{i}^{-} \leq D_{1}, i \in I_{1} . \end{matrix}$ (19)

Because the square of the radiuses of the small spheres of view A and view B are $R_{A}^{2} = ∥ ϕ (x_{i}^{A}) - C_{A} ∥^{2}$ and $R_{B}^{2} = ∥ ϕ (x_{i}^{B}) - C_{B} ∥^{2}$ , respectively. By taking into the values C_A and C_B, we can get

$\begin{matrix} R_{A}^{2} & = K (x_{i}^{A}, x_{i}^{A}) + C_{A}^{2} - \frac{2}{1 - ν_{A}} \\ \cdot (\sum_{i \in I_{1}} δ_{j}^{A} K (x_{i}^{A}, x_{i}^{A}) - \frac{ν}{l_{2}} \sum_{j \in I_{2}} K (x_{i}^{A}, x_{j}^{A})), \end{matrix}$ (20) and

$\begin{matrix} R_{B}^{2} & = K (x_{i}^{B}, x_{i}^{B}) + C_{B}^{2} - \frac{2}{1 - ν_{B}} \\ \cdot (\sum_{i \in I_{1}} δ_{j}^{A} K (x_{i}^{A}, x_{i}^{A}) - \frac{ν}{l_{2}} \sum_{j \in I_{2}} K (x_{i}^{B}, x_{j}^{B})), \end{matrix}$ (21) where K (x_i, x_j) = (ϕ (x_i) · ϕ (x_j)) is a kernel function which is chosen in advance.

To obtain the optimal solutions of the second optimization problem (5), we still need to derive its dual problem. By introducing the Lagrangian multipliers, the Lagrangian function of (5) can be written as: $\begin{matrix} L_{2} = R_{A}^{2} + R_{B}^{2} - ρ_{A}^{2} - ρ_{B}^{2} + \frac{1}{ν_{2}^{A} l_{2}} \sum_{j \in I_{2}} ξ_{j}^{A} \\ + \frac{1}{ν_{2}^{B} l_{2}} \sum_{j \in I_{2}} ξ_{j}^{B} + D_{2} \sum_{j \in I_{2}} μ_{j} \\ + \sum_{j \in I_{2}} γ_{j}^{+} (∥ ϕ (x_{j}^{A}) - C_{A} ∥^{2} - R_{A}^{2} - ρ_{A}^{2} \\ - ∥ ϕ (x_{j}^{B}) - C_{B} ∥^{2} + R_{B}^{2} + ρ_{B}^{2} - μ_{j}) \\ - \sum_{j \in I_{2}} γ_{j}^{-} (∥ ϕ (x_{j}^{A}) - C_{A} ∥^{2} - R_{A}^{2} - ρ_{A}^{2} \\ - ∥ ϕ (x_{j}^{B}) - C_{B} ∥^{2} + R_{B}^{2} + ρ_{B}^{2} + μ_{j}) \\ - \sum_{j \in I_{2}} α_{j}^{A} (∥ ϕ (x_{j}^{A}) - C_{A} ∥^{2} - R_{A}^{2} - ρ_{A}^{2} + ξ_{j}^{A}) \\ - \sum_{j \in I_{2}} α_{j}^{B} (∥ ϕ (x_{j}^{B}) - C_{B} ∥^{2} - R_{B}^{2} - ρ_{B}^{2} + ξ_{j}^{B}) \\ - \sum_{j \in I_{2}} β_{j}^{A} ξ_{j}^{A} - \sum_{j \in I_{2}} β_{j}^{B} ξ_{j}^{B} - \sum_{j \in I_{2}} τ_{j} μ_{j}, \end{matrix}$ (22) where $γ_{j}^{+ / -} \geq 0$ , $α_{j}^{A / B} \geq 0$ and $β_{j}^{A / B} \geq 0, j \in I_{2}$ are Lagrangian multipliers. Differentiating L₂ with respect to variables $ρ_{A}^{2}, ρ_{B}^{2}$ , $ξ_{j}^{A}, ξ_{j}^{B}$ and μ_j can yield the following KKT conditions: $\frac{\partial L_{2}}{\partial ρ_{A}^{2}} = 0 \Rightarrow \sum_{j \in I_{2}} (α_{j}^{A} - γ_{j}^{+} + γ_{j}^{-}) = 1,$ (23) $\frac{\partial L_{2}}{\partial R_{B}^{2}} = 0 \Rightarrow \sum_{j \in I_{2}} (α_{j}^{B} + γ_{j}^{+} - γ_{j}^{-}) = 1,$ (24) $\frac{\partial L_{2}}{\partial ξ_{j}^{A}} = 0 \Rightarrow α_{j}^{A} + β_{j}^{A} = \frac{1}{ν_{2}^{A} l_{2}},$ (25) $\frac{\partial L_{2}}{\partial ξ_{j}^{B}} = 0 \Rightarrow α_{j}^{B} + β_{j}^{B} = \frac{1}{ν_{2}^{B} l_{1}},$ (26) $\frac{\partial L_{2}}{\partial μ_{j}} = 0 \Rightarrow γ_{j}^{+} + γ_{j}^{-} + τ_{j} = D_{2} .$ (27)

Similarly, we can derive the dual formulation of (5) as follows:

$\begin{matrix} max_{{\bar{δ}}_{j}^{A / B}, γ_{j}^{+ / -}} & - \sum_{j \in I_{2}} {\bar{δ}}_{j}^{A} (∥ ϕ (x_{j}^{A}) - C_{A} ∥^{2}) \\ - \sum_{j \in I_{2}} {\bar{δ}}_{j}^{B} (∥ ϕ (x_{j}^{B}) - C_{B} ∥^{2}) \\ s . t . & \sum_{j \in I_{2}} {\bar{δ}}_{j}^{A} = \sum_{j \in I_{2}} {\bar{δ}}_{j}^{B} = 1, \\ 0 \leq α_{j}^{A} \leq \frac{1}{ν_{2}^{A} l_{2}}, 0 \leq α_{j}^{B} \leq \frac{1}{ν_{2}^{B} l_{2}}, \\ 0 \leq γ_{j}^{+}, γ_{j}^{-}, γ_{j}^{+} + γ_{j}^{-} \leq D_{2}, j \in I_{2} . \end{matrix}$ (28) where ${\bar{δ}}_{j}^{A} = α_{j}^{A} - γ_{j}^{+} + γ_{j}^{-}$ and ${\bar{δ}}_{j}^{B} = α_{j}^{B} + γ_{j}^{+} - γ_{j}^{-}, j \in I_{2}$ .

In order to obtain $ρ_{A}^{2}$ and $ρ_{B}^{2}$ , we use the negative points $x_{j}^{A}$ and $x_{j}^{B}$ whose Lagrangian multipliers $α_{j}^{A}$ and $α_{j}^{B}$ satisfy $0 < α_{j}^{A} < \frac{1}{ν_{2}^{A} l_{2}}$ and $0 < α_{j}^{B} < \frac{1}{ν_{2}^{B} l_{2}}$ , and suppose the number is $n_{n}^{A}$ and $n_{n}^{B}$ , respectively. According to KKT complementary relaxation conditions:

$\begin{matrix} α_{j}^{A} (∥ ϕ (x_{j}^{A}) - C_{A} ∥^{2} - R_{A}^{2} - ρ_{A}^{2} + ξ_{j}^{A}) = 0, \\ α_{j}^{B} (∥ ϕ (x_{j}^{B}) - C_{B} ∥^{2} - R_{B}^{2} - ρ_{B}^{2} + ξ_{j}^{B}) = 0, \end{matrix}$ we can get $ρ_{A}^{2} = \frac{1}{n_{n}^{A}} \sum_{j = 1}^{n_{n}^{A}} (∥ ϕ (x_{j}^{A}) - C_{A} ∥^{2} - R_{A}^{2}),$ (29) $ρ_{B}^{2} = \frac{1}{n_{n}^{B}} \sum_{j = 1}^{n_{n}^{B}} (∥ ϕ (x_{j}^{B}) - C_{B} ∥^{2} - R_{B}^{2}) .$ (30)

Obviously, the dual problem (19) is a QPP. Problem (28) is a LPP with respect to the variables ${\bar{δ}}^{A / B}$ rather than a QPP, which can greatly shorten the elapsed time. Therefore, the proposed MvMMTSSVM inherits the elegant formulation of MMTSSVM. Besides, the QPP (19) and LPP (28) do not involve the operation of matrix inversion, which can also improve the calculation speed to a certain extent.

After acquiring the optimal solutions $(C_{A}^{*}, R_{A}^{*}, ρ_{A}^{*})$ and $(C_{B}^{*}, R_{B}^{*}, ρ_{B}^{*})$ , we can use the following equations to calculate the label of a new point (x^A, x^B) from representations A and B, respectively, $f (x^{A}) = {\begin{matrix} 1, & if ∥ ϕ (x^{A}) - C_{A} ∥ < \frac{R_{A} + \sqrt{R_{A}^{2} + ρ_{A}^{2}}}{2} \\ - 1, & else \end{matrix}$ (31) and $f (x^{B}) = {\begin{matrix} 1, & if ∥ ϕ (x^{B}) - C_{B} ∥ < \frac{R_{B} + \sqrt{R_{B}^{2} + ρ_{B}^{2}}}{2} \\ - 1, & else \end{matrix}$ (32)

Because the two representations can complement each other, we can also make the prediction by using two views collectively, where

$f (x^{A}, x^{B}) = {\begin{matrix} 1, & if ∥ ϕ (x^{A}) - C_{A} ∥ + ∥ ϕ (x^{B}) - C_{B} ∥ \\ < \frac{R_{A} + \sqrt{R_{A}^{2} + ρ_{A}^{2}} + R_{B} + \sqrt{R_{B}^{2} + ρ_{B}^{2}}}{2} \\ - 1, & else \end{matrix}$ (33)

In above all, the flowchart of proposed MvMMTSSVM is summarized as Algorithm 1.

Algorithm 1: The flowchart of MvMMTSSVM.

Input: Training dataset

{(x_{i}^{A}, x_{i}^{B}, y_{i})}_{i = 1}^{l}

, where y_i ∈ {+1, - 1}, and a testing point x.

Output: The label of testing point.

1. Initialize the parameters

ν_{i}^{A}, ν_{i}^{B}, ν \in (0, 1)

, D_i > 0, i = 1, 2, kernel parameter r.

2. Solve QPP (19), get

α_{i}^{A / B}, γ_{i}^{+ / -}

, get

(R_{A}^{2}, R_{B}^{2})

by Eq. (20) and Eq. (21), respectively.

3. Solve LPP (28), get

{\bar{δ}}_{j}^{A / B}, γ_{j}^{+ / -}

, get

(ρ_{A}^{2}, ρ_{B}^{2})

by Eq.(29) and Eq. (30), respectively.

4. For a texting sample x with single view x^A or x^B, we can predict its label by Eq. (31) and Eq.(32),

or predict the label of testing point (x^A, x^B) by Eq. (33).

4 Experiments

In this section, we compare our MvMMTSSVM with other basic algorithms to verify its effectiveness on imbalanced multi-view datasets. All the experiments are operated in Matlab R2014a. The system configuration of the personal computer is Inter Core i3-4160, CPU3.60GHz with 8.00GB RAM.

4.1 Datasets

1) Synthetic dataset 1: The distribution is shown in Fig. 2, where the positive and negative points are represented by red “+” and blue “o”, respectively. The positive and negative points come from Gaussian distributions $(x_{i}^{A / B}, x_{i}^{A / B}), i \in I_{1} \sim N (μ_{+}^{A / B}, Σ_{+}^{A / B})$ , and $(x_{j}^{A / B}, x_{j}^{A / B}), j \in I_{2} \sim N (μ_{-}^{A / B}, Σ_{-}^{A / B})$ , where $μ_{+}^{A} = (- 2 - 2)$ and $μ_{-}^{A} = (2 2)$ for view A, $μ_{+}^{B} = (2 2)$ and $μ_{-}^{B} = (- 2 - 2)$ for view B, and the covariance for the two views are as follows $Σ_{+}^{A} = (\begin{matrix} 1 & 0.5 \\ 0.5 & 1.5 \end{matrix}), Σ_{-}^{A} = (\begin{matrix} 1 & 0.5 \\ 0.5 & 1.5 \end{matrix}),$ and $Σ_{+}^{B} = (\begin{matrix} 0.6 & 0.1 \\ 0.1 & 0.5 \end{matrix}), Σ_{-}^{B} = (\begin{matrix} 1.2 & 0.2 \\ 0.1 & 1 \end{matrix}) .$ We generate 200 positive samples and 100 negative samples in total.

Fig. 2

Illustration of Synthetic dataset 1.

2) Synthetic dataset 2: Likewise the work of Belkin et al. [41], we generate the dataset with two views, where view A is two-moon data and view B is two-line data. We generate 1000 positive and 500 negative samples. Fig. 3 is the illustration of sample distribution, where the red “+” represents positive sample and the blue “o” stands for the negative sample.

Fig. 3

Illustration of Synthetic dataset 2.

3) Ionosphere: It comes from UCI database 1 with 225 positive and 126 negative samples. It has 34 features. However, it is not a multi-view dataset. We regard the original 34 data as view A. As for view B, we adopt the principal component analysis (PCA) to extract its 99% information, and the extracted data has 25 features.

4) Advertisement: It is made up of 3279 samples with 459 of them being ads and the rest being non-ads. It also comes from UCI database 2 . The attributes are divided into two views: view A contains 587 features, which describes the information of image itself; view B contains 971 features, which includes textual features and so on.

5) Handwritten digit: It also comes from UCI database 3 , which includes features of ten handwritten digits (0-9). It has 2000 samples with 10 classes (200 samples per class) with 6 different views (mfeatfac-216, mfeatfou-76, mfeatkar-64, mfeatmor-6, mfeatpix-240, mfeatzer-47). In the experiment, we consider the mfeatfac with 216 features as view A, and the mfeatfou with 76 features as view B. Since it is a multi-classification dataset, we adopt the “one-verse-rest” strategy for each category separately. Therefore, we need to train 10 binary classifiers in total.

6) Corel: It comes from Corel database 4 , which consists of 1000 color pictures with 10 different categories (People /Beach /Architechture /Bus /Dinasor /Elephant /Flower /Horse /Mountain /Food) [42]. The illustration of Corel dataset is shown in Fig. 4. Each category has 100 pictures. We extract its 72 dimension HSV histogram as view A. Based on the gray level co-occurrence matrix of RGHV, we extract 32 texture features as view B. Similarly, we adopt the “one-verse-rest” scenario to transform the multi-class situation to ten binary classification problems. We consider 100 images coming from the required class as negative samples, and images remains as positive.

Fig. 4

Illustration of Corel dataset.

The information of these datasets is summarized in Table 1. These multi-view datasets are all imbalanced data.

Table 1

The information of 6 multi-view datasets

Datasets	# Samples	# Positive	# Negative	# View A	# View B
Synthetic1	300	200	100	2	2
Synthetic2	1500	1000	500	2	2
Ionosphere	351	225	126	34	25
Advertisement	3279	2820	459	587	971
Handwritten	2000	1800	200	216	76
Corel	1000	900	100	72	32

4.2 Benchmark methods

We compare the proposed MvMMTSSVM with the following benchmark methods:

SVM-2K [14]: It is a two-view learning algorithm, which firstly combines the SVM and KCCA.

MvTSVM [16]: It is a multi-view leaning algorithm, which implements the consensus principle and empirical risk minimization principle simultaneously. It needs to train the TSVMs in different views, then combines them by introducing the constraint of similarity. Nevertheless, it needs matrix inversion which will affect the solving speed.

MMTSSVM-A/B: MMTSSVM [37] is a single view learning algorithm which is trained separately on view A or view B.

joint-MMTSSVM: It can be regarded as a kind of multi-view learning algorithm, since it adopts the concatenating strategy to combine the two views into one single view.

4.3 Experimental settings

We adopt the averaged G-means to evaluate the classification performance. The definition of G-means is: $G - means = \sqrt{Sn \cdot Sp}$ , where Sn is the abbreviation of sensitivity and $Sn = \frac{TP}{TP + FN} \cdot 100$ ; Sp is the abbreviation of specificity and $Sp = \frac{TN}{TN + FP} \cdot 100$ ; TP, TN, FP, FN stand for the number of true positive, true negative, false positive, and false negative, respectively. All experiments are repeated five times. In the following results, “G-means” and “Time” stand for the averaged G-means and time of five experimental results, respectively.

The Gaussian RBF kernel K (x_i, x_j) = exp (- ||x_i - x_j||²/r²) is chosen for nonlinear case, where r is the parameter chosen in advance. We use grid search to find the best parameter combination. In SVM-2K, we set c^A = c^B. In MvTSVM, we let $c_{1}^{A} = c_{2}^{A} = c_{1}^{B} = c_{2}^{B}$ ; In MMTSSVM-A, MMTSSVM-B and joint-MMTSSVM, we set ν₁ = ν₂; In MvMMTSSVM, we set ν_A = ν_B = ν, $ν_{1}^{A} = ν_{1}^{B} = ν_{2}^{A} = ν_{2}^{B}$ . In a nutshell, we need to choose the parameters (c^A, D, r) in SVM-2K; (c₁, D, r) in MvTSVM; parameters (ν, ν₁, r) in MMTSSVM-A, MMTSSVM-B and joint-MMTSSVM; parameters (ν, ν₁, c₂, D, r) in the proposed MvMMTSSVM. The parameters $c^{A}, c_{1}^{A}, D$ are tuned over {2ⁱ|i = -2, . . . , 2}. Parameters ν, ν₁ are selected over {0.2, . . . , 0.7}. The kernel parameter r is searched in {2ⁱ|i = -2, . . . , 2}.

4.4 Experimental results

Table 2 shows the average G-means and elapsed time of six compared algorithms on 24 imbalanced datasets, where the bold values are the best G-means. The results in Table 2 indicate that the proposed MvMMTSSVM obtains satisfactory performance because it obtains the highest G-means on 14 datasets out of 24 comparisons.

Table 2
Performance comparison of six algorithms on 24 datasets

SVM-2K MvTSVM MMTSSVM-A MMTSSVM-B j-MMTSSVM MvMMTSSVM

Datasets G-means (%) G-means (%) G-means (%) G-means (%) G-means (%) G-means (%)

Time (s) Time (s) Time (s) Time (s) Time (s) Time (s)

Synthetic1 100.00±0.00 100.00±0.00 97.06±2.23 100.00±0.00 100.00±0.00 100.00±0.00

0.1973 0.2488 0.1176 0.1018 0.1243 0.6315

Synthetic2 99.49±0.09 99.41±0.16 98.07±2.93 99.06±0.36 99.66±0.64 99.75±0.18

7.3936 17.5177 1.8046 1.9567 1.9294 8.6505

Ionosphere 94.68±2.77 94.25±3.36 91.41±4.28 95.85±3.55 92.06±4.32 96.73±1.66

1.1635 0.3913 0.1290 0.1188 0.1542 0.4930

Advertisement 96.00±1.40 93.91±2.03 94.00±3.32 77.88±22.63 93.57±1.97 95.39±0.97

46.0971 147.6619 69.6662 135.5940 268.1945 262.9648

Handwritten0 99.22±1.75 99.39±0.53 98.49±1.18 99.21±0.72 98.01±0.43 98.48±1.68

8.2721 19.2753 6.8117 7.5326 12.7761 13.3653

Handwritten1 83.16±11.09 96.30±0.31 85.85±27.03 91.99±5.13 96.31±1.03 98.48±1.68

9.0075 21.0357 8.2519 6.2470 8.0202 9.9516

Handwritten2 97.63±2.18 98.22±2.99 97.98±1.60 97.09±1.79 98.59±0.84 98.89±1.01

9.2359 57.4173 9.5299 4.8785 7.7319 9.7439

Handwritten3 96.91±3.43 97.96±1.61 98.32±1.49 90.63±7.75 95.47±0.51 97.69±0.94

9.3050 50.5275 6.9937 7.6889 7.8752 10.0001

Handwritten4 91.20±7.55 83.64±29.06 97.98±1.52 78.39±22.99 95.60±8.20 97.86±1.45

8.6501 31.3239 8.2057 8.8426 9.6947 11.3006

Handwritten5 97.02±3.15 83.05±28.74 84.69±28.18 83.23±23.27 77.20±28.47 97.33±0.99

8.4595 29.2433 10.0445 9.4194 8.9053 13.4246

Handwritten6 86.08±14.09 95.59±0.48 96.09±6.32 73.68±27.49 96.03±0.60 97.86±1.45

8.8374 59.9516 9.1890 7.2970 7.0931 12.8982

Handwritten7 99.24±1.02 95.56±0.27 95.59±0.36 94.94±1.88 97.38±1.42 96.47±1.39

6.7013 60.5804 7.8007 4.9881 7.0619 14.7134

Handwritten8 97.00±6.55 98.50±0.61 93.54±10.29 98.31±1.02 98.04±0.94 97.89±1.09

6.5776 28.7636 7.2551 4.9953 6.9813 31.0360

Handwritten9 99.13±1.63 84.53±29.56 97.54±2.26 78.89±25.06 96.67±1.39 98.23±0.84

6.9376 23.1985 8.1770 5.4749 9.3912 30.4821

Corel-people 88.64±9.52 94.56±3.58 92.47±4.76 69.06±20.73 85.64±14.32 92.72±6.70

21.0945 11.3500 1.1882 1.1311 1.4036 5.5042

Corel-beach 92.98±5.73 91.97±9.76 91.63±8.19 72.93±15.49 87.79±12.66 95.65±0.36

19.345 10.9983 1.5175 1.7794 1.9568 5.6331

Corel-architecture 95.79±0.53 91.56±4.38 95.35±0.30 71.42±14.16 78.71±11.71 95.79±0.63

23.5402 11.6874 3.0190 1.2832 1.5693 6.2334

Corel-bus 97.09±1.13 96.63±0.63 96.13±0.32 93.75±6.15 90.54±9.95 97.25±1.29

19.9241 14.9733 2.6787 1.1405 1.4192 6.4334

Corel-dinosaur 99.78±0.12 98.43±0.71 98.52±1.79 98.55±1.02 97.64±3.74 98.74±1.94

21.4152 13.2160 3.7283 1.0618 1.6533 6.6104

Corel-elephant 68.02±25.94 71.28±33.91 85.55±13.55 73.80±21.78 74.11±19.94 82.66±28.37

19.7977 10.5517 1.1888 1.2063 1.4448 5.9228

Corel-flower 96.40±2.50 96.63±0.74 82.13±24.11 94.71±3.96 95.74±6.94 93.75±8.60

20.1361 13.9237 3.6142 1.1220 1.4996 7.5425

Corel-horse 97.86±0.79 96.90±1.41 97.02±2.16 89.39±7.67 96.08±0.72 98.17±0.73

11.1617 3.7029 1.1944 1.4724 1.3239 5.6272

Corel-mountain 89.77±12.60 79.56±27.90 68.68±40.06 56.60±15.81 80.09±15.82 95.20±0.13

16.6317 10.6606 2.7116 1.1705 1.5794 5.8860

Corel-food 85.50±9.25 91.92±11.83 85.41±15.87 67.14±16.74 91.00±12.54 90.45±12.71

21.7454 11.6095 3.3015 1.1871 3.2872 4.6090

Win/draw/loss 16/2/6 17/1/6 21/0/3 20/1/3 19/1/4

	SVM-2K	MvTSVM	MMTSSVM-A	MMTSSVM-B	j-MMTSSVM	MvMMTSSVM
Synthetic1	100.00±0.00	100.00±0.00	97.06±2.23	100.00±0.00	100.00±0.00	100.00±0.00
	0.1973	0.2488	0.1176	0.1018	0.1243	0.6315
Synthetic2	99.49±0.09	99.41±0.16	98.07±2.93	99.06±0.36	99.66±0.64	99.75±0.18
	7.3936	17.5177	1.8046	1.9567	1.9294	8.6505
Ionosphere	94.68±2.77	94.25±3.36	91.41±4.28	95.85±3.55	92.06±4.32	96.73±1.66
	1.1635	0.3913	0.1290	0.1188	0.1542	0.4930
Advertisement	96.00±1.40	93.91±2.03	94.00±3.32	77.88±22.63	93.57±1.97	95.39±0.97
	46.0971	147.6619	69.6662	135.5940	268.1945	262.9648
Handwritten0	99.22±1.75	99.39±0.53	98.49±1.18	99.21±0.72	98.01±0.43	98.48±1.68
	8.2721	19.2753	6.8117	7.5326	12.7761	13.3653
Handwritten1	83.16±11.09	96.30±0.31	85.85±27.03	91.99±5.13	96.31±1.03	98.48±1.68
	9.0075	21.0357	8.2519	6.2470	8.0202	9.9516
Handwritten2	97.63±2.18	98.22±2.99	97.98±1.60	97.09±1.79	98.59±0.84	98.89±1.01
	9.2359	57.4173	9.5299	4.8785	7.7319	9.7439
Handwritten3	96.91±3.43	97.96±1.61	98.32±1.49	90.63±7.75	95.47±0.51	97.69±0.94
	9.3050	50.5275	6.9937	7.6889	7.8752	10.0001
Handwritten4	91.20±7.55	83.64±29.06	97.98±1.52	78.39±22.99	95.60±8.20	97.86±1.45
	8.6501	31.3239	8.2057	8.8426	9.6947	11.3006
Handwritten5	97.02±3.15	83.05±28.74	84.69±28.18	83.23±23.27	77.20±28.47	97.33±0.99
	8.4595	29.2433	10.0445	9.4194	8.9053	13.4246
Handwritten6	86.08±14.09	95.59±0.48	96.09±6.32	73.68±27.49	96.03±0.60	97.86±1.45
	8.8374	59.9516	9.1890	7.2970	7.0931	12.8982
Handwritten7	99.24±1.02	95.56±0.27	95.59±0.36	94.94±1.88	97.38±1.42	96.47±1.39
	6.7013	60.5804	7.8007	4.9881	7.0619	14.7134
Handwritten8	97.00±6.55	98.50±0.61	93.54±10.29	98.31±1.02	98.04±0.94	97.89±1.09
	6.5776	28.7636	7.2551	4.9953	6.9813	31.0360
Handwritten9	99.13±1.63	84.53±29.56	97.54±2.26	78.89±25.06	96.67±1.39	98.23±0.84
	6.9376	23.1985	8.1770	5.4749	9.3912	30.4821
Corel-people	88.64±9.52	94.56±3.58	92.47±4.76	69.06±20.73	85.64±14.32	92.72±6.70
	21.0945	11.3500	1.1882	1.1311	1.4036	5.5042
Corel-beach	92.98±5.73	91.97±9.76	91.63±8.19	72.93±15.49	87.79±12.66	95.65±0.36
	19.345	10.9983	1.5175	1.7794	1.9568	5.6331
Corel-architecture	95.79±0.53	91.56±4.38	95.35±0.30	71.42±14.16	78.71±11.71	95.79±0.63
	23.5402	11.6874	3.0190	1.2832	1.5693	6.2334
Corel-bus	97.09±1.13	96.63±0.63	96.13±0.32	93.75±6.15	90.54±9.95	97.25±1.29
	19.9241	14.9733	2.6787	1.1405	1.4192	6.4334
Corel-dinosaur	99.78±0.12	98.43±0.71	98.52±1.79	98.55±1.02	97.64±3.74	98.74±1.94
	21.4152	13.2160	3.7283	1.0618	1.6533	6.6104
Corel-elephant	68.02±25.94	71.28±33.91	85.55±13.55	73.80±21.78	74.11±19.94	82.66±28.37
	19.7977	10.5517	1.1888	1.2063	1.4448	5.9228
Corel-flower	96.40±2.50	96.63±0.74	82.13±24.11	94.71±3.96	95.74±6.94	93.75±8.60
	20.1361	13.9237	3.6142	1.1220	1.4996	7.5425
Corel-horse	97.86±0.79	96.90±1.41	97.02±2.16	89.39±7.67	96.08±0.72	98.17±0.73
	11.1617	3.7029	1.1944	1.4724	1.3239	5.6272
Corel-mountain	89.77±12.60	79.56±27.90	68.68±40.06	56.60±15.81	80.09±15.82	95.20±0.13
	16.6317	10.6606	2.7116	1.1705	1.5794	5.8860
Corel-food	85.50±9.25	91.92±11.83	85.41±15.87	67.14±16.74	91.00±12.54	90.45±12.71
	21.7454	11.6095	3.3015	1.1871	3.2872	4.6090
Win/draw/loss	16/2/6	17/1/6	21/0/3	20/1/3	19/1/4

To further analyze the experimental results, we adopt the method of Win/Draw/Loss (WDL). The last line of Table 2 displays the number of datasets for which our MvMMTSSVM obtains higher, equal or lower G-means than the other five methods. The proposed MvMMTSSVM obtains higher G-means than SVM-2K on 16 comparisons. Compared with MvTSVM, our MvMMTSSVM has better ability of dealing with class imbalanced problem, because it is superior to MvTSVM in 17 out of 24 datasets. The proposed MvMMTSSVM yields better performance than MMTSSVM-A and MMTSSVM-B, because its G-means is better or equal to that of MMTSSVM-A and MMTSSVM-B in 21 out of 24 comparions. In datasets Synthetic1 and Synthetic2, MMTSSVM-B performs better than MMTSSVM-A, which can also be seen in Fig. 2 and Fig. 3 obviously. Our MvMMTSSVM obtains higher G-means than j-MMTSSVM on 19 datasets. Besides, SVM-2K and MvTSVM perform better than MMTSSVM-A, MMTSSVM-B and j-MMTSSVM.

From the perspective of elapsed time, we find that the proposed method spends more running time than MMTSSVM-A, MMTSSVM-B and j-MMTSSVM. The reason is that the number of Lagrange multipliers is four times as that of MMTSSVM-A, MMTSSVM-B and j-MMTSSVM. The proposed MvMMTSSVM spends less running time than MvTSVM in most datasets. That’s because our algorithm doesn’t involve matrix inverse operation and the optimal solutions can be got just by solving a QPP and a LPP, which can greatly reduce the computational cost.

In addition, we do experiments on four datasets, i.e. Synthetic1, Synthetic2, Ionosphere and Advertisement by using K-nearest neighbor (KNN) [43] and multi-view nonparallel SVM (MvNPSVM) [7]. Because KNN is a traditional machine learning algorithm for single-view learning, it cannot deal with multi-view data directly. We adopt similar strategy as MMTSSVM. For the sake of fairness, the optimal solutions of MvNPSVM are solved by the function “quadprog” in Matlab toolbox. The results are shown in Table 3. Compared with KNN-A, KNN-B and KNN, the proposed MvMMTSSVM can get higher G-means on datasets Synthetic2, Ionosphere and Advertisement. However, it takes longer time than KNN. The performance of MvNPSVM is better than our MvMMTSSVM on dataset Ionosphere. However, the computational efficiency of MvNPSVM is too low. Especially, MvNPSVM runs out of memory on dataset Advertisement. The main reason is that the number of Lagrange multipliers need to be solved in MvNPSVM is eight times the sample size, which will take a long time to get the optimal solution.

Table 3

Results of KNN and MvNPSVM on four datasets

	Synthetic1		Synthetic2		Ionosphere		Advertisement
Algorithms	G-means (%)	Time (s)	G-means (%)	Time (s)	G-means (%)	Time (s)	G-means (%)	Time (s)
KNN-A	97.81±2.50	0.1005	99.60±0.45	0.0096	87.24±6.09	0.0208	84.65±4.23	1.0588
KNN-B	100.00±0.00	0.0071	99.01±0.43	0.0084	94.36±2.20	0.0258	93.11±4.64	1.3253
KNN	100.00±0.00	0.0081	99.61±0.48	0.0091	87.24±6.09	0.0261	87.23±4.69	1.5242
MvNPSVM	99.74±0.59	4.0660	99.45±0.47	357.2901	97.68±2.56	3.7762	-	-

4.5 Friedman test

The averaged G-means of our MvMMTSSVM in Table 2 are not always higher than that of other five methods. Therefore, we use Friedman test [44] to do further analysis. For each dataset, the highest G-means is ranked as 1, the next is ranked as 2, and so on. The average ranks of the six compared algorithms are shown in Table 4. Significantly, the last line of Table 4 shows that the average rank of our MvMMTSSVM is 2.02, which is the lowest among the six algorithms. The results reflect that the proposed MvMMTSSVM performs best among the six compared algorithms.

Table 4
Average rank on G-means of six algorithms

Datasets SVM-2K MvTSVM MMTSSVM-A MMTSSVM-B j-MMTSSVM MvMMTSSVM

Synthetic1 3 3 6 3 3 3

Synthetic2 3 4 6 5 2 1

Ionosphere 3 4 6 2 5 1

Advertisement 1 4 3 6 5 2

Handwritten0 2 1 4 3 6 5

Handwritten1 6 3 5 4 2 1

Handwritten2 5 3 4 6 2 1

Handwritten3 4 2 1 6 5 3

Handwritten4 4 5 2 6 3 1

Handwritten5 2 5 3 4 6 1

Handwritten6 5 4 2 6 3 1

Handwritten7 1 5 4 6 2 3

Handwritten8 5 1 6 2 3 4

Handwritten9 1 5 3 6 4 2

Corel-people 4 1 3 6 5 2

Corel-beach 2 3 4 6 5 1

Corel-architecture 1.5 4 3 6 5 1.5

Corel-bus 2 3 4 5 6 1

Corel-dinosaur 1 5 4 3 6 2

Corel-elephant 6 5 1 4 3 2

Corel-flower 2 1 6 4 3 5

Corel-horse 2 4 3 6 5 1

Corel-mountain 2 4 5 6 3 1

Corel-food 4 1 5 6 2 3

Average rank 2.98 3.33 3.88 4.88 3.92 2.02

Datasets	SVM-2K	MvTSVM	MMTSSVM-A	MMTSSVM-B	j-MMTSSVM	MvMMTSSVM
Synthetic1	3	3	6	3	3	3
Synthetic2	3	4	6	5	2	1
Ionosphere	3	4	6	2	5	1
Advertisement	1	4	3	6	5	2
Handwritten0	2	1	4	3	6	5
Handwritten1	6	3	5	4	2	1
Handwritten2	5	3	4	6	2	1
Handwritten3	4	2	1	6	5	3
Handwritten4	4	5	2	6	3	1
Handwritten5	2	5	3	4	6	1
Handwritten6	5	4	2	6	3	1
Handwritten7	1	5	4	6	2	3
Handwritten8	5	1	6	2	3	4
Handwritten9	1	5	3	6	4	2
Corel-people	4	1	3	6	5	2
Corel-beach	2	3	4	6	5	1
Corel-architecture	1.5	4	3	6	5	1.5
Corel-bus	2	3	4	5	6	1
Corel-dinosaur	1	5	4	3	6	2
Corel-elephant	6	5	1	4	3	2
Corel-flower	2	1	6	4	3	5
Corel-horse	2	4	3	6	5	1
Corel-mountain	2	4	5	6	3	1
Corel-food	4	1	5	6	2	3
Average rank	2.98	3.33	3.88	4.88	3.92	2.02

The null-hypothesis is that the six algorithms are equivalent. The Friedman statistic can be calculated by

$χ_{F}^{2} = \frac{12 N}{k (k + 1)} [\sum_{j} R_{j}^{2} - \frac{k {(k + 1)}^{2}}{4}],$ (34) where $R_{j} = \frac{1}{N} \sum_{i} r_{i}^{j}$ , $r_{i}^{j}$ is the jth of k algorithms on the ith of N datasets. Friedman’s $χ_{F}^{2}$ can be computed by

$F_{F} = \frac{(N - 1) χ_{F}^{2}}{N (k - 1) - χ_{F}^{2}} .$ (35)

From Eq.(34), we can get $χ_{F}^{2} = 32.8107$ . Bring it into Eq.(35), we can obtain F_F = 8.6553, where F_F is distributed according to F-distribution with (5, 115) degrees of freedom. If α = 0.05, then the critical value of F (5, 115) is 2.29, which is much lesser than F_F (8.6553). That means it rejects the null-hypothesis. Namely, there are significant differences between the six methods.

Because of rejecting the null hypothesis, we can further proceed with the Nemenyi test [45]. Fig. 5 displays the results, where the the hollow circles stand for the average ranks of the six algorithms and the straight lines centered on the “∘” are the critical difference CD, where

$CD = q_{α} \sqrt{\frac{k (k + 1)}{6 N}} .$ (36) Here q_α = 2.850 and CD = 1.3982 for the level of significance α = 0.05. The results in Fig. 5 indicate that our MvMMTSSVM performs significantly better than j-MMTSSVM, MMTSSVM-A, MMTSSVM-B at the 95% confidence level.

Fig. 5

Friedman test.

5 Conclusions

In this work, we propose a new MVL termed as MvMMTSSVM which follows the the maximum principle and the consensus principle at the same time. By following the maximum principle, it seeks two homocentric spheres and tries to maximize the margin of the two spheres on each view. Besides, the absolute value constraints of two views can make sure the implementation of the consensus principle. The proposed method not only inherits the elegant formulation of MMTSSVM, but also deals with the multi-view imbalance problem effectively. Therefore, it is more suitable for practical application. Experimental results on 24 binary datasets confirm that our MvMMTSSVM is feasible and valid.

Footnotes

Acknowledgments

This work was supported in part by the Fundamental Research Funds for the Central Universities (No. BLX201928) and National Natural Science Foundation of China (No. 12071475).

References

Zhao

, Xie

, Xu

and Sun

, Multi-view learning overview: Recent progress and new challenges, Information Fusion 38 (2017), 43–54.

El-Regaily

S.A.

, Salem

M.A.M.

, Aziz

M.H.A.

and Roushdy

M.I.

, Multi-view Convolutional Neural Network for lung nodule false positive reduction, Expert Systems with Applications 162 (2020), 113017.

Jiang

, Zhou

, Zhang

and Wang

, A novel multi-view SVM based on consistent hidden density distributions between views for face recognition, Journal of Intelligent & Fuzzy Systems 36(6) (2019), 5245–5259.

Guo

, Feng

, Li

, Wang

, Liu

and Qiao

, Multi-view laplacian least squares for human emotion recognition, Neurocomputing 370 (2019), 78–87.

Ullah

, Muhammad

, Hussain

and Baik

S.W.

, Conflux LSTMs Network: A novel approach for multi-view action recognition, Neurocomputing (2020).

Mirończuk

M.M.

, Protasiewicz

and Pedrycz

, Empirical evaluation of feature projection algorithms for multi-view text classification, Expert Systems with Applications 130 (2019), 97–112.

Tang

, Li

, Tian

and Liu

, Multi-view learning based on nonparallel support vector machine, Knowledge-Based Systems 158 (2018), 94–108.

Tang

, Tian

, Liu

and Kou

, Coupling privileged kernel method for multi-view learning, Information Sciences 481 (2019), 110–127.

Gao

, Multiple rank multi-linear kernel support vector machine formatrix data classification, International Journal of Machine Learning and Cybernetics 9(2) (2015).

10.

Wang

, Wang

E.K.

, Li

, Ye

, Lau

R.Y.K.

and Du

, Multi-view learning via multiple graph regularized generative model, Knowledge-Based Systems 121 (2017), 153–162.

11.

Wei

and Zhou

Z.H.

, A New Analysis of Co-Training, Proceedings of the 27th International Conference on Machine Learning (ICML-10), June 21–24, 2010, Haifa, Israel (2010).

12.

, Allinson

, Tao

and Li

, Multitraining support vector mchine for image retrieval, IEEE Transactions on Image Processing A Publication of the IEEE Signal Processing Society 15(11) (2006), 3597–3601.

13.

Sheikh Hassani

and Green

J.R.

, Multi-view co-training for microRNA prediction, Scientific Reports 9 (2019).

14.

Farquhar

, Hardoon

, Meng

, Shawe-taylor

and Szedmak

, Two view learning: Svm-2k, theory and practice, Proceedings of the Annual Conference on Neural Information Processing Systems (2005), 355–362.

15.

Houthuys

, Langone

and Suykens

J.A.K.

, Multi-view least squares support vector machines classification, Neurocomputing 282 (2018), 78–88.

16.

Xie

and Sun

, Multi-view twin support vector machines, Intelligent Data Analysis 19(4) (2015), 701–712.

17.

Xie

, Regularized multi-view least squares twin support vector machines, Applied Intelligence 48(9) (2018), 3108–3115.

18.

Chao

and Sun

, Consensus and complementarity based maximum entropy discrimination for multi-view classification,368}, Information Sciences 367{– (2016), 296–310.

19.

Xie

and Sun

, General multi-view learning with maximum entropy discrimination, Neurocomputing 332 (2019), 184–192.

20.

Jing

, Su

, Li

and Nie

, Learning robust affinity graph representation for multi-view clustering, Information Sciences 544 (2021), 155–167.

21.

Araujo

A.F.R.

, Antonino

V.O.

and Ponce-Guevara

K.L.

, Selforganizing subspace clustering for high-dimensional and multiview data, Neural Networks 130 (2020), 253–268.

22.

Houthuys

, Langone

and Suykens

J.A.K.

, Multi-view jernel spectral clustering, Information Fusion 44 (2018), 46–56.

23.

Tang

, He

, Tian

, Liu

, Kou

and Alsaadi

F.E.

, Coupling loss and self-used privileged information guided multiview transfer learning, Information Sciences (2020).

24.

, Tian

and Liu

, Multi-view transfer learning with privileged learning framework, Neurocomputing 335 (2019), 131–142.

25.

, Han

, Nie

and Li

, Multi-view scaling support vector machines for classification and feature selection, IEEE Transactions on Knowledge and Data Engineering 32 (2020), 1419–1430.

26.

Niu

, Shang

and Tian

, Multi-view svm classification with feature selection, Procedia Computer Science 162 (2019), 405–412.

27.

Salim

, Shiju

S.S.

and Sumitra

, Design of multi-view graph embedding using multiple kernel learning, Engineering Applications of Artificial Intelligence 90 (2020), 103534.

28.

Song

, Wang

, Ye

, Wang

, Yin

and Wang

, Multi-view ensemble learning based on distance-to-model and adaptive clustering for imbalanced credit risk assessment in P2P lending, Information Sciences 525 (2020), 182–204.

29.

, Dai

, Dong

and Wang

, Multi-view ensemble learning method for microblog sentiment classification, Expert Systems with Applications 166 (2021), 113987.

30.

Tang

, Tian

, Liu

, Li

, Lv

and Kou

, Improved multi-view privileged support vector machine, Neural Networks 106 (2018), 96–109.

31.

Sun

, Shawe-Taylor

and Mao

, PAC-Bayes analysis of multi-view learning, Information Fusion 35 (2017), 117–131.

32.

Vapnik

V.N.

, The nature of statistical learning theory, Springer, Berlin, (1995).

33.

Tsang

I.W.

, Kwok

J.T.

and Cheung

P.M.

, Core vector machines: fast SVM training on very large data sets, Journal of Machine Learning Research 6(2) (2005), 363–392.

34.

Jayadeva

R.K.

and Chandra

, Twin support vector machines for pattern classification, IEEE Transactions on Pattern Analysis and Machine Intelligence 29(5) (2007), 905–910.

35.

Peng

and Xu

, A twin-hypersphere support vector machine classifier and the fast learning algorithm, Information Sciences 221 (2013), 12–27.

36.

and Ye

, A small sphere and large margin approach for novelty detection using training data with outliers, IEEE Transations on Pattern Analysis and Machine Learning 31(11) (2009), 2088–2092.

37.

, Maximum margin of twin spheres support vector machine for imbalanced data classification, IEEE Transactions on Cybernetics 47(16) (2016), 1540–1550.

38.

Xie

and Sun

, Multi-view support vector machines with the consensus and complementarity information, IEEE Transactions on Knowledge and Data Engineering 32 (2020), 2401–2413.

39.

Tang

, Tian

, Zhang

and Liu

, Multiview privileged support vector machines, IEEE Transactions on Neural Networks and Learning Systems 29(8) (2018), 3463–3477.

40.

Cheng

, Fu

, Luo

, Ye

, Liu

and Zhu

, Multi-view generalized support vector machine via mining the inherent relationship between views with applications to face and fire smoke recognition, Knowledge-Based Systems 210 (2020), 106488.

41.

Belkin

, Niyogi

, Sindhwani

and Bartlett

, Manifold regularization: a geometric framework for learning from examples, Journal of Machine Learning Research 7(1) (2006), 2399–2434.

42.

and Wang

, Automatic linguistic indexing of pictures by a statistical modeling approach, IEEE Transactions on Pattern Analysis and Machine Intelligence 25(9) (2003), 1075–1088.

43.

Cover

T.M.

and Hart

P.E.

, Nearest neighbor pattern classification, IEEE Transactions on Information Theory 13 (1967), 21–27.

44.

Dems̃ar

, Statistical comparisons of classifiers over multiple data sets, Journal of Machine Learning Research 7 (2006), 1–30.

45.

Nemenyi

P.B.

, Distribution-free multiple comparisons. PhD thesis, Princeton University, (1963).