Deep support vector neural networks

Abstract

Kernel based Support Vector Machines, SVM, one of the most popular machine learning models, usually achieve top performances in two-class classification and regression problems. However, their training cost is at least quadratic on sample size, making them thus unsuitable for large sample problems. However, Deep Neural Networks (DNNs), with a cost linear on sample size, are able to solve big data problems relatively easily. In this work we propose to combine the advanced representations that DNNs can achieve in their last hidden layers with the hinge and $\epsilon$ insensitive losses that are used in two-class SVM classification and regression. We can thus have much better scalability while achieving performances comparable to those of SVMs. Moreover, we will also show that the resulting Deep SVM models are competitive with standard DNNs in two-class classification problems but have an edge in regression ones.

Keywords

Support vector machines deep learning

1. Introduction

Kernel Support Vector Machines (SVM; [1]) were very popular in the 1990s and early 2000’s because of the powerful models they provided. However they are somewhat out of fashion in the current big data era. The main reason for this is their cost, at least quadratic (and often worse) with respect to sample size, in a sharp contrast with the linear cost in sample size of Deep Neural Networks (DNNs) that currently dominate big data machine learning (ML).

This cost is hard to avoid in the usual kernel formulation of SVMs. In fact, while at the end, SVMs are linear models on extended representations $\Phi(x)$ of the original patterns $x$ , when working with effective kernels such as the Gaussian one, the representation lies in a Reproducible Kernel Hilbert Space (RKHS) and can only be handled implicitly through the dot products $\Phi(x)\cdot\Phi(x^{\prime})=k_{\Phi}(x,x^{\prime};\Theta)$ where $k_{\Phi}$ is a suitable explicit kernel and $\Theta$ denotes possible kernel parameters. This implies that the SVM dual problem must be solved. But its dimension is $N$ , the sample size, one must work with an $N\times N$ kernel matrix and have to apply the SMO algorithm with a cost that very often will be at least quadratic in $N$ (see Section 2 below). All this means that standard SVM training and, even more so, hyperparametrization will simply be too costly in a big data setting.

Many proposals have appeared in the literature to apply SVMs on large datasets, usually focused on Support Vector classification (SVC) problems. Among them we can mention incremental learning [2], ensemble learning [3] or the cutting planes method [4]. Nevertheless, it can be said that, unless substantial hardware resources are committed, current kernel SVM training methods are not time competitive for datasets with more than about 100,000 patterns. Recently, GPU compliant implementations of SVM training have been proposed [5] but, while greatly accelerating SVM training, the memory limitations of large sample SVMs still remain.

Larger datasets can be handled by working with linear SVMs using the LIBLINEAR library [6] over either the primal or dual problems, often using coordinate descent [7] to solve the minimization problems. However, efficient linear SVM models can only be expected when the original features have very large dimensions so that feature projections are no longer needed. When sample dimension is just moderate, linear SVM models are usually much less powerful that their Gaussian counterparts. Pegasos [8] can be used on a kernel setting but, as with LIBLINEAR, it must work with homogeneous models $w\cdot x$ or $w\cdot\Phi(x)$ without a bias term; this restricts the models to be homogeneous and while sometimes classification performance is comparable with that of full fledged SVMs [9], in other cases falls behind. Given the above difficulties in large problems, the possibility of bringing SVM training into a neural network setting has already been considered in [10], where it is compared with standard softmax DNNs on classification problems, and by [11] in speech recognition.

However, and in a different vein from these works, we first observe that the non differentiability of the SVM primals is rather mild; in fact, Pegasos exploits subgradient descent essentially by skipping the single non differentiability at 0 of the hinge loss, or at $\pm$ $\epsilon$ of the $\epsilon$ -insensitive one. This is precisely the same approach followed on DNNs when having to handle the non differentiability of the ReLU activations while computing gradients by backpropagation. As a matter of fact, the hinge loss is already available in Keras [12], and the $\epsilon$ -insensitive loss can be written using, say, TensorFlow primitives; in turn, this makes possible to perform backpropagation on that loss.

This opens the way to try to solve the primal problem directly working with an explicit kernel transformation $\Phi(x;{\cal W})$ parameterized by a certain weight set ${\cal W}$ which, in turn, can be obtained through some training process. Of course, an appropriate $\Phi$ has to be found and a natural choice is the transformation performed by a DNN between its inputs and its last hidden layer; we will call the resulting models Deep SVMs and our proposal in this work is to build Deep SVM models in much the same way that DNNs are now routinely trained.

Notice that, at the end, this point of view also leads to a linear model acting on a hidden representation. However, and although seemingly close, the final models will be quite different. Following the terminology in [13], DNNs belong to the category of iterative summarization methods; on the other hand, Gaussian SVMs, the most popular way to derive non linear SVMs, are placed in [13] within the case-based reasoning methods, jointly for instance with $k$ -NN methods. In fact, Gaussian SVMs can be seen as covering the sample space with generalized balls centered on the learned support vectors and, similarly to $k$ -NN methods, they cannot handle very large samples. However, and as pointed out in [13] and often observed in the literature (see for instance [14]), when properly hyperparameterized, different ML methods have similar performances in problems with features that have clear predictive meanings. This suggests to compare first the performance of Gaussian and Deep SVMs, which we did in our previous work [15] and which we summarize below for the reader’s convenience.

In any case, Gaussian SVM comparisons are only possible for datasets up to a given size. In fact, as of today, they have substantial train and, more so, hyperparameterization costs with sample sizes above $10^{5}$ and, thus, they are no match for Deep SVMs on problems of such sizes. But this is far from implying that Deep SVMs are then to be the preferred model in that setting for then another obvious adversary arises, namely standard DNNs with cross entropy loss for classification and squared error loss for regression. First, though, it must be observed that, contrary with what happened with Gaussian SVMs, Deep SVMs and standard DNNs work at first sight in a very similar fashion. In fact, two learning processes take place simultaneously. First, a non linear representation $z=\Phi(x,{\cal W}^{*})$ between the original inputs and their last hidden layer (LHL) representation is learned. On the other hand, the network also learns a linear model $y=w^{*}\cdot z+w^{*}_{0}$ between these LHL representations and the network outputs $y$ .

The difference lies, however, in their respective training goals. On standard DNNs the goal is the estimation of posterior class distributions in classification or to mean squared error minimization in regression. On the other hand, hinge loss minimization in our proposed Deep SVM classifier training means that a maximum margin model is learned on the last hidden layer; similarly, the $\epsilon$ insensitive loss achieved in the training of our proposed Deep SVM regressors implies that an optimal $\epsilon$ tube is fitted around the model built in the last hidden layer. Those are quite different training objectives with possible advantages of their own, as large margins should result in good generalization and the $\epsilon$ parameter gives extra flexibility to the deep support vector regressors.

We will focus here our attention in the two-class classification and regression performance of fully non linear DNNs versus that of Deep SVMs. We will first work with feed-forward fully connected architectures for classification and regression but we will also consider image classification datasets where we will use problem specific architectures already proposed in the literature and whose results are close to the state of the art. In summary, our main contributions are:

•
The proposal of Deep SVMs with the same architectures and training procedures of either standard or advanced DNNs but using the hinge and $\epsilon$ -insensitive losses typical of SVMs for two-class classification and regression respectively.
•
A statistical comparison of the performance of DNNs and deep SVMs with standard, fully connected feedforward architectures in mid size classification and regression problems.
•
A statistical comparison of the performance of DNNs and Deep SVMs over large image datasets, using problem specific advanced deep architectures that are representative of the current state of the art.

In other to streamline the presentation, we will use the acronyms DNNC and DNNR (Deep Neural Network Classification and/or Regression) to refer to models with a DNN architecture and the cross entropy or mean squared error losses in classification and regression respectively. Similarly, we will use the acronyms DSVC and DSVR (Deep Support Vector Classification and/or Regression) to refer to models with a DNN architecture but using now the hinge loss for two-class classification and the $\epsilon$ -insensitive loss for regression. Finally we will refer to Gaussian SVM models for two class classification as GSVC and to Gaussian SVM models for regression as GSVR.

Putting together our results here and those in [15], a first conclusion is that GSVCs, DNNCs and our proposed DSVCs have similar performances in two-class classification problems. However, the situation for regression problems is different as we will numerically show that DSVRs have an hedge over DNNRs, not only when mean absolute error (MAE) is the cross validation (CV) score used for model hyperparametrization but also when using mean squared error (MSE) as the CV score.

The rest of the paper is organized as follows. We give a short unified exposition to standard SVMs in Subsection 2, where we also address training and test complexity and model sparsity. We present our deep SVM proposal in Section 3, where we also briefly discuss training complexity. For the sake of the reader, we briefly recover our results in [15] and Sections 5 and 6 contain our comparison of standard DNNs versus Deep SVMs. More precisely, in Section 5 we will compare, using the same datasets of Section 4, standard fully connected DNNCs and DSVCs first and then DNNRs and DSVRs. In both cases we will also address the statistical significance of our results. On the other hand, in Section 6 we shall work with three large image datasets, the MNIST, MNIST Fashion and CIFAR 10 problems. For the first two we will use the well known LeNet-5 architecture [16] to compare the cross entropy and hinge losses (i.e., the corresponding DNNC and DSVC models) on the ten 2-class problems of distinguishing on MNIST each of the 0 to 9 digits from the others, and similarly for MNIST Fashion. We will follow the same approach with the ten class CIFAR 10 problems but here the cross entropy and hinge losses will act on top of a ResNet32 V1 model [17], a substantially more complex deep residual network. The comparisons will also include LIBLINEAR models and we will address their statistical significance. Finally, the paper ends with a discussion, conclusions and pointers to further work.
2. A SVM review

2.1 Primal and dual SVC and SVR

We will briefly review here primal and dual SVC and SVR in a unified view following [18]. Assume a sample given as a set of triplets ${\cal S}=\{(x_{i},y_{i},p_{i}):i=1,\ldots,N\}$ with $y_{i}=\pm$ 1 and $p_{i}$ some scalar values, and consider the minimization problem:

$\displaystyle\min_{w,b,\xi}{\cal P}(w,b,\xi)=\frac{1}{2}\|{w}\|^{2}+C\sum_{i}% \xi_{i}$

(1) $\displaystyle s.t.\left\{\begin{array}[]{ll}y_{i}(w\cdot x_{i}+b)\geqslant p_{% i}-\xi_{i}&\forall i,\\ \xi_{i}\geqslant 0&\forall i.\\ \end{array}\right.$

Now if we take $p_{i}=1$ for $i=1,\ldots,N$ , Eq. (2.1) reduces to the standard two-class Support Vector Classification (SVC) problem

$\displaystyle\min_{w,b,\xi}{\cal P}_{C}(w,b,\xi)=\frac{1}{2}\|w\|^{2}+C\sum_{i% =1}^{N}\xi_{i}$ (2) $\displaystyle s.t.\left\{\begin{array}[]{ll}y_{i}(w\cdot x_{i}+b)\geqslant 1-% \xi_{i},&\forall i,\\ \xi_{i}\geqslant 0,&\forall i.\\ \end{array}\right.$

On the other hand, given an initial sample ${\cal R}=\{(x_{i},\linebreak t_{i}):i=1,\ldots,N\}$ and enlarging it as ${\cal S}=\{(x_{i},y_{i},\linebreak p_{i}):i=1,\ldots,2N\}$ by taking, for $1\leqslant i\leqslant N$ , $y_{i}=1$ , $p_{i}=t_{i}-\epsilon$ and $y_{N+i}=-1$ , $p_{N+i}=-t_{i}-\epsilon$ , and $x_{N+i}=x_{i}$ , Eq. (2.1) reduces now to

$\displaystyle\min_{w,b,\xi,\xi^{\prime}}{\cal P}_{R}(w,b,\xi)=\frac{1}{2}\|w\|% ^{2}$ (3) $\displaystyle+C\sum_{i=1}^{N}(\xi_{i}+\xi_{i}^{\prime})$ (4) $\displaystyle s.t.\left\{\begin{array}[]{ll}w\cdot x_{i}+b-t_{i}\leqslant% \epsilon+\xi_{i},&\forall i,\\ t_{i}-w\cdot x_{i}-b\leqslant\epsilon+\xi_{i}^{\prime}&\forall i,\\ \xi_{i}\geqslant 0&\forall i,\\ \xi_{i}^{\prime}\geqslant 0&\forall i,\\ \end{array}\right.$

which is just the standard formulation of the $\epsilon$ -insensitive SV regression (SVR).

While its loss is not differentiable, Eq. (2.1) it is, nevertheless, a convex minimization problem. Because of this, one does not attempt to solve it directly; instead, standard Lagrangian optimization is applied to reduce Eq. (2.1) to the equivalent problem of minimizing the general dual function

$\displaystyle\min_{\alpha}{\cal D}(\alpha)=\frac{1}{2}\sum_{i}\sum_{j}\alpha_{% i}\alpha_{j}Q_{ij}$ $\displaystyle-\sum_{i}\alpha_{i}p_{i}$ (5) $\displaystyle s.t.\left\{\begin{array}[]{ll}0\leqslant\alpha_{i}\leqslant C&% \forall i,\\ \sum_{i}\alpha_{i}y_{i}=0,\\ \end{array}\right.$

where $Q$ is the matrix with entries $Q_{ij}=y_{i}y_{j}x_{i}\cdot x_{j}$ .

In any case, a clear drawback of trying to solve Eq. (2.1) directly is that it would result in plain linear models, possibly not powerful enough to deal with the problems at hand. The dual formulation overcomes this as it can be stated in a kernel setting by replacing the inner products $x_{i}\cdot x_{j}$ in $Q$ with a kernel function $k(x_{i},x_{j})$ . More precisely, for a general kernel, such as the usual Gaussian one, it can be shown that there is a map $\Phi$ projecting the initial patterns into an abstract reproducible kernel Hilbert space (RKHS) [19] such that we have $k(x_{i},x_{j})=\Phi(x_{i})\cdot\Phi(x_{j})$ . The final models will be still linear but no longer in the original features but on their kernel defined extensions $\Phi(x)$ . However, the exact knowledge of $\Phi$ is not needed and can be implicitly handled.
2.2 Complexity and sparsity of kernel SVMs

Notice that the dual problem has to be solved in a space with dimension $N$ , i.e., sample size. It involves having to handle the kernel matrix, with size $N\times N$ , which is bound to be problematic for large samples. Moreover, the standard algorithm to solve Eq. (2.1) is Standard Sequential Minimization, SMO, which proceeds iteratively, with each iteration having an $O(N)$ cost. In turn, the number of iterations is $\Omega(N_{SV})$ , with $N_{SV}$ the number of support vectors, which in most problems is $\Theta(N)$ . Thus, SMO training will require at least $\Theta(N)$ iterations and has, hence, a minimum cost $\Omega(N^{2})$ which is quite higher in practice, sometimes being estimated to be cubic in $N$ . It is thus clear that for large samples (such as those with more than $10^{5}$ samples), SVM training may prove too costly.

Moreover, an often cited property of SVMs is that their solution is sparse, i.e., the number of nonzero optimal $\alpha^{*}_{i}$ values (and of the corresponding support vectors) is less than $N$ ; however, it is also often the case that the number of support vectors (i.e., patterns $x_{i}$ for which $\alpha^{*}_{i}>0$ ) is still $\Theta(N)$ which hinders, for instance, the use of SVMs in near on line settings, as its application of a kernel SVM model on new patterns has then a $\Theta(N)$ cost, as all support vectors enter the model.

Figure 1.

SVC (red) and Logistic Regression (blue) classifiers for two class linearly separable problems. The SVC hyperplane has a maximum margin while the Logistic Regresion one is closer to the minority class.

3. Deep SVC and SVR

It is well known that large margins usually imply better generalization. For linearly separable problems and a $C$ large enough, it is also well known that linear SVC models achieve a maximum margin. But this is not necessarily so when the cross entropy loss of Logistic Regression is used; an example can be seen in Fig. 1. It shows the SVC (red) and Logistic Regression (blue) separating hyperplanes built over linearly separable samples given by two Gaussians centered at (0, 1) (top) and (0, $-$ 1) (bottom); class sizes are 100 for the top class and 500 for the bottom one. Gaussian standard deviation is 0.25 in the left figure and 0.33 in the right one. As it can be seen, the SVC hyperplane (red) achieves a maximum margin while the cross entropy hyperplane (blue) is much closer to the minority top class. (The points of each class closest to the SVC hyperplane are shown as red dots.)

Of course, the usefulness of linear classifiers is rather small and, as we have just discussed, the standard way to introduce non linearity into SVMs is to use the kernel trick. However, the price to pay for this is the need to solve the dual problem, with a cost usually at least quadratic with respect to sample size. But, in any case, we point out that the loss in Eq. (2.1) can be written as

$\displaystyle\max\{0,p-y(w\cdot x+b)\}.$

For classification this is just the hinge loss

$\displaystyle h(y,f(x))=\max\{0,1-yf(x)\}$

with $f(x)=w\cdot x+b$ , while for regression we have the $\epsilon$ -insensitive loss

$\displaystyle\ell_{\epsilon}(y,f(x))=\max\{0,|y-f(x)|-\epsilon\}.$

Observe that in both cases the non differentiability of the primal loss is rather mild and in fact very similar to that of the ReLU activations routinely handled in Deep Neural Network (DNN) packages. Moreover, if ${\cal W}_{h}$ is the set of weights between the input and the last hidden layer in a DNN, the input to last hidden layer transformation $\Phi(x)=F(x,{\cal W}_{h})$ that such a DNN performs usually results in much enhanced pattern representations over which linear models may yield powerful classifiers and regressors. In particular, using a flexible enough DNN would results in a linearly separable problem in the last layer and, as illustrated in Fig. 1, a maximum margin classifier when the hinge loss is used, something that would not happen with the cross entropy loss usually applied in two class DNN classification. All this suggests to exploit this explicit kernel to avoid dealing with the SVM dual problem.

More precisely, consider a DNN with linear outputs, a weight set ${\cal W}_{h}$ between the input and the last hidden layer, and linear weights and bias $(w,b)$ from the last layer to the network outputs. We denote the input-output transformation of such a network as

$\displaystyle f(x,w,b,{\cal W}_{h})=w\cdot F(x,{\cal W}_{h})+b=w\cdot\Phi(x)+b$

where $F(x,{\cal W}_{h})=\Phi(x)$ is the last hidden layer representation of $x$ . The network’s cost function will be

$\displaystyle\quad J(w,b,{\cal W}_{h})$ $\displaystyle{}=\frac{1}{N}\sum_{i=1}^{N}\ell\left(y_{i},w\cdot F(x_{i},{\cal W% }_{h})+b\right)$ (6) $\displaystyle{}+\alpha_{S}\|w\|^{2}+\alpha_{H}{\cal R}({\cal W}_{h}),$

where $\ell(y,\hat{y})$ is the loss that we want to use, and we split weight regularization in two parts, the squared norm $\|w\|^{2}$ of the linear output weight vectors and the sum ${\cal R}({\cal W}_{h})$ of the Frobenius norms for the weight matrices between hidden layers. We shall consider the hinge loss for two-class classification problems and the $\epsilon$ -insensitive loss for regression ones. Now, the weight gradients of such a network can be computed by standard backpropagation and then the network trained by plain gradient descent, as in Algorithm 1, or, as we will do later on, using the Adam solver [20].

Algorithm 1: Deep SVM backpropagation
Input: Random initial weights $W=(W_{\ell})$ , $1\leqslant\ell\leqslant L$ , dataset $(X,Y)$ , learning rate sequence $\rho_{t}$
1 For epoch $t$ in $1\dots T$ :
2 While $(X,Y)$ is not empty:
3 Select a batch $(X^{\prime},Y^{\prime})$ from $(X,Y)$ without
3 replacement
4 # Forward Pass
5 $Z_{0}=X^{\prime}$
6 For layer $\ell$ in $1,\dots,L$ :
7 $Z_{\ell}=f_{\ell}(Z_{\ell-1}W_{l})$ # $f_{\ell}$ : activation at layer $\ell$
8 # Backward Pass
9 $E_{L}={\cal L}(Y^{\prime},Z_{L})$ # ${\cal L}$ either hinge or $\epsilon$ -insensitive loss
10 For layer $\ell$ in $L-1,\dots,1$ :
11 $S_{\ell}=Z_{\ell-1}W_{\ell}$
12 $E_{\ell}=\frac{\partial E_{\ell+1}}{\partial S_{\ell}}$
13 $\nabla_{W_{\ell}}{\cal L}=E_{\ell}\odot Z_{\ell-1}$
14 $W_{\ell}=W_{\ell}-\rho_{t}\nabla_{W_{\ell}}{\cal L}$
Output: Fitted weights $W=(W_{\ell})$

Assume that such a network has been trained, let $(w^{*},b^{*},{\cal W}^{*})$ be the optimal weights and denote as $z^{*}=F(x,{\cal W}^{*})$ the network’s last hidden layer outputs. Then $(w^{*},b^{*})$ solve the problem of minimizing

$\displaystyle\frac{1}{N}\sum_{i=1}^{N}\ell(y_{i},w\cdot(z^{*})_{i}+b)+\alpha_{% S}\|w\|^{2}$ (7)

over $w$ , $b$ : if not and $w^{\prime}$ , $b^{\prime}$ were a better weight-bias pair, the overall weight set $(w^{\prime},b^{\prime},{\cal W}^{*})$ would improve on $(w^{*},b^{*},{\cal W}^{*})$ . We can rewrite Eq. (7) as

$\displaystyle\quad\text{argmin}_{w,b}\frac{1}{2}\|w\|^{2}$ (8) $\displaystyle{}+\frac{1}{2\alpha_{S}N}\sum_{i=1}^{N}\ell(y_{i},w\cdot(z^{*})_{% i}),$

which is just the problem solved in SVC when $\ell$ is the hinge loss and the problem solved in SVR when $\ell$ is the $\epsilon$ -insensitive loss. Thus, after a DNN has been trained to solve Eq. (7), the optimal output weights $w^{*}$ and bias $b^{*}$ solve either the SVC or SVR problems over the network outputs at the last hidden layer. In summary, we can couple DNN architectures with linear outputs and either the hinge or $\epsilon$ -insensitive losses to bring SVC or SVR into a DNN set up; we will call DSVC or DSVR the resulting deep SVM models.

The cost of training them is just that of training a standard DNN, which depends on three inputs. The first is the architecture or, more precisely, the number $n_{W}$ of network weights, the second is the number $n_{I}$ of iterations to be performed by the solver and the third is sample size $N$ . Obviously, $n_{W}$ is independent of $N$ and, contrary to what happens in SMO, so is $n_{I}$ , which basically depends on the precision to which the DNN cost function is to be minimized. Putting all this together, DSVC or DSVR training has a

$\displaystyle O(n_{W}\times n_{I}\times N)$

cost, which now grows linearly with $N$ but not quadratically, as is the case with SVMs. Moreover, the cost of applying the model on a single pattern is now $O(n_{W})$ , independent of sample size and, hence, much faster in principle than that of a SVM model.

In any case, the classification and regression performance of such networks, which we will call Deep SVMs, has to be compared with that of standard Gaussian SVC and SVR. This was done in [15] and we will briefly recall it in Section 4. However, and as mentioned before, even if training Gaussian SVMs on large datasets is, as of today, too costly and deep SVM models can be an alternative, it is clear that in the large sample setting there is an obvious counterpart to Deep SVMs, precisely standard cross entropy or mean squared error DNNs. We will compare them with Deep SVMs in Sections 5 and 6.

4. Gaussian versus deep SVMs

In this section we will briefly summarize for the convenience of the reader the comparisons between Gaussian two class classification (GSVC) and regression (GSVR) models and their deep counterparts DSVC and DSVR given in [15], where we used for comparison the classification datasets a4a, a8a, australian, cod-rna, diabetes, german.numer, ijcnn1, w7a and w8a, and the regression datasets abalone, bodyfat, cadata, cpusmall, housing, mg, mpg and space_ga. All are taken from the LibSVM repository [21] (which includes references to papers where they have been used) and their training and, when available, test sample sizes and dimensions are given in Table 1.

Table 1
Number of train and (when available) test patterns and dimensions in the two-class (top) and regression (bottom) problems

	n. patterns train	n. patterns test	dimension
a4a	4781	27780	123
a8a	22696	9865	123
australian	690		14
cod-rna	59535	271617	8
diabetes	768		8
german.numer	1000		24
ijcnn1	49990	91701	22
w7a	24692	25057	300
w8a	49749	14951	300
abalone	4177		8
bodyfat	252		14
cadata	20640		8
cpusmall	8192		12
housing	506		13
mg	1385		6
mpg	392		7
space_ga	3107		6

We compared GSVC and GSVR models against DSVC and DSVR networks with 1, 3 and 5 hidden layers to be trained using the adam optimizer over 200 pattern minibatches; more details are given in [15]. For GSVCs we must choose optimal values of two hyperparameters, the regularization constant $C$ and the Gaussian kernel width $\gamma$ ; we must add an extra hyperparameter $\epsilon$ for GSVR. In both cases we considered 5 logarithmically equi-spaced $C$ values in the interval [10 ${}^{-3}$ , 10 ${}^{6}$ ] and $\gamma$ values of the form $\frac{2^{k}}{{d}}$ , with $d$ pattern dimension and $-3\leqslant k\leqslant 6$ . Before this we scaled inputs feature-wise to a [0, 1] range. For $\epsilon$ , we considered for GSVR 5 equi-distributed log values in the interval [2 ${}^{-10}$ $\times$ std (y), 2 ${}^{-1}$ $\times$ std (y)]. For DSVC and DSVR models we chose optimal values of the hyperparameters $\alpha_{S}$ and $\alpha_{H}$ by exploring now 5 evenly log spaced values in the interval [2 ${}^{-30}$ , 2 ${}^{10}$ ]; the $\epsilon$ DSVR parameter was chosen again within 5 equi-distributed log values in the interval [2 ${}^{-10}$ $\times$ std (y), 2 ${}^{-1}$ $\times$ std (y)].

In the classification problems having a separate test set available, GSVC and DSVC model hyperparameters were found by 4-fold stratified cross validation (CV) over the train set using accuracy as the CV score; we then computed test accuracies over the test set. When no test set is available (i.e., in three classification and all regression datasets), model performance was measured by a nested, two loop, 4-fold CV approach, in which each one of the four outer folds is kept for testing and the other three folds are passed to the inner loop, where best hyperparameters are obtained again by 4-fold CV and tested on the outer test folds. The regression score was the mean absolute error (MAE).

We report in Table 2 the GSVC accuracies and those of DSVC models with with 1 (DSVC1), 3 (DSVC3) and 5 (DSVC5) layers. Similarly, we report now in Table 3 the GSVR mean absolute errors (MAEs) and those of DSVR models with 1 (DSVR1), 3 (DSVR3) and 5 (DSVR5) layers. For a better reading, in both tables we give in the last column the accuracy or MAE of the best performing DSVC or DSVR model. As it can be seen, GSVCs give the largest accuracies on the four datasets, a4a, cod-rna, german.numer and w7a while for the other five datasets the highest accuracy is achieved by a DSVC model. In regression, GSVRs give the smallest MAE on the four datasets, bodyfat, cpusmall, housing and mpg and a DSVR model gave the smallest MAE for the other four datasets. As concluded in [15], the performance of GSVC and GSVR models, on the one hand, and DSVC and DSVR models is fairly balanced.

Table 2

Accuracies in the two-class problems (taken from [15])

	SVC	DSVC1	DSVC3	DSVC5	DSVC
a4a	84.32	84.19	84.19	84.27	84.27
a8a	84.92	85.18	85.14	85.14	85.18
australian	85.50	85.51	86.82	85.21	86.82
cod-rna	96.58	96.66	96.61	96.45	96.66
diabetes	77.60	76.95	76.43	77.21	77.47
german.numer	76.10	75.80	75.10	75.80	75.80
ijcnn1	97.93	98.88	99.07	98.99	99.07
w7a	98.87	98.82	98.83	98.83	98.83
w8a	99.04	98.99	99.16	99.03	99.16

Table 3

MAEs in the regression problems (taken from [15])

	SVR	DSVR1	DSVR3	DSVR5	DSVR
abalone	1.48	1.49	1.50	1.51	1.49
bodyfat ( $\times$ 100)	0.05	0.42	0.51	0.28	0.28
cadata ( $/$ 10000)	4.96	4.81	3.26	3.26	3.26
cpusmall	2.13	2.21	2.19	2.40	2.19
housing	2.28	2.58	2.34	2.57	2.34
mg ( $\times$ 100)	9.26	9.68	9.13	9.58	9.13
mpg	1.91	2.39	2.49	2.36	2.36
space_ga ( $\times$ 100)	9.67	9.14	8.63	8.86	8.63

Table 4

Train and test times in minutes for the larger regression (top) and classification (bottom) datasets

	tr_svm	tr_dnn	tr_mlp	ts_svm	ts_dnn	ts_mlp
abalone	0.0016	0.2117	0.0380	0.0006	0.0016	0.0001
cadata	0.2840	0.5840	0.2481	0.0206	0.0051	0.0007
cpusmall	0.0462	0.3995	0.0599	0.0051	0.0023	0.0003
space_ga	0.0094	0.1652	0.0126	0.0005	0.0014	0.0001
a4a	0.0241	0.3059	0.1053	0.1035	0.0142	0.0024
a8a	0.5986	0.4780	0.4452	0.1819	0.0055	0.0007
w7a	0.2687	0.7161	0.1755	0.1759	0.0134	0.0020
w8a	11.9484	2.4366	0.3211	0.7897	0.0081	0.0013
ijcnn1	0.2278	2.4964	0.5845	0.2615	0.0387	0.0076
cod-rna	0.4152	3.2763	0.7782	0.6527	0.1082	0.0194

To finish this Section we will briefly discuss time comparisons between Gaussian and Deep SVM models. We have performed execution time experiments for both, but we observe first that measuring times is not as straightforward as it may look, for we are using two quite different implementations. For standard SVMs we use the scikit-learn implementation that has at its core the extremely efficient C $++$ implementation of the LIBSVM library. On the other hand, for deep SVMs we rely on the Keras framework with the Adam solver [20], which, in turn, relies on TensorFlow library. While quite efficient, notice that TensorFlow is a general purpose deep network library and, thus, not specialized in any particular kind of deep architecture. In fact, the loss gradients are derived through automatic differentiation procedures which work with very general architectures and losses. However, while this is very convenient, it must also necessarily hamper execution times; to all this, Keras adds its own overhead. To have an intermediate yardstick we have also measured train and test times of our fully connected models when using the MLPClassifier and MLPRegressor classes in the scikit-learn library. They are implemented in Python and Cython and while being thus less efficient than the compiled LBSVM library, they are more compactly programmed than the Keras models (although much less general), which results in shorter training times.

Table 5

Accuracy comparisons of one, three and five hidden layer models when hyperparameterized with hinge (DSVC) and binary cross-entropy (DNNC) scores. Larger accuracies in bold face

	1 hid. layer		3 hid. layers		5 hid. layers
	DSVC	DNNC	DSVC	DNNC	DSVC	DNNC
a4a	84.46	84.59	83.95	84.48	84.39	84.42
a8a	85.08	85.38	85.43	84.96	84.81	80.38
australian	85.64	87.96	85.36	85.94	85.50	86.66
cod-rna	96.56	96.30	96.56	96.70	96.23	94.59
diabetes	77.34	76.30	77.47	76.82	77.73	76.17
german.numer	75.10	74.80	73.10	75.90	74.50	75.80
ijcnn1	98.59	98.61	98.53	99.03	98.70	96.07
w7a	98.90	98.83	98.89	98.68	98.73	98.84
w8a	99.35	99.30	99.24	99.38	99.20	99.33

With all those caveats, we give in Table 4 for the largest regression (top) and classification (bottom) datasets, timings for train (tr_svm/dnn/mlp columns) and test (ts_svm/dnn/mlp columns), which correspond to LIBSVM, Keras and scikit-learn MLPs, respectively. Model hyperparameters for each dataset are those corresponding to the best cross validation estimators just described. As it can be seen, SVM training times are of the order of magnitude of those of MLPClassifier or MLPRegressor. The exceptions are a4a (a rather small dataset) and w8a (where SVMs take too long). As expected, Keras times are longest but they catch up as sample size increases. But notice also that for all datasets but the relatively small abalone and space_ga, LIBSVM test times are about 10 times longer than Keras’ test times, and near or above 50 times those of MLPClassifier or MLPRegressor. Therefore, and technology and implementation issues aside, these performances essentially confirm what is to be expected on theoretical grounds, as discussed at the end of Sections 2 and 3.

5. Standard DNNs versus deep SVMs

Recall that for large datasets the counterpart to our DSVC and DSVR proposals are standard deep networks for either two-class classification with binary cross entropy as the training loss and sigmoid outputs (i.e., DNNC in our notation) or for regression with the squared loss for training and linear outputs (i.e., DNNR in our notation). In both cases we will use Keras implementations of networks with one, three and five hidden layers, and linear outputs. As in [15], these have 100 units on the intermediate ones and 0.1 $\times$ $|S|$ units in the last hidden layer, with $|S|$ denoting sample size and a lower bound in layer size of 100 and an upper bound of 1000. In other words,the size of the last hidden layer is $\min\{1000,\max\{100,0.1\times|S|\}\}$ . Similarly, the DSVC and DSVR losses will be hinge and $\epsilon$ -insensitive, respectively.

Also, and as before, Tikhonov regularization will be applied in all layers, with two different parameters, $\alpha_{S}$ for the output weights and $\alpha_{R}$ in all others; for DSVR models we will also hyperparametrize the $\epsilon$ loss parameter. We will change slightly the $\alpha_{S},\alpha_{R}$ and $\epsilon$ ranges of the previous section. The $\alpha_{S}$ and $\alpha_{R}$ values will be of the form $10^{k}$ , $-5\leqslant k\leqslant 2$ , while the $\epsilon$ values will be $2^{-k}\times\sqrt{0.5}$ , $-4\leqslant k\leqslant 0$ . Inputs will be scaled feature-wise to zero mean and one standard deviation. For regression problems we will also use a scikit-learn pipeline that incorporates a TransformedTargetRegressor component so that targets are normalized to zero mean and one standard deviation before training and de-normalized afterwards.

Table 6
Mean absolute errors and $p$ values for DSVR and DNNR models with one, three and five hidden layers. Smaller values in bold face when absolute error distributions are statistically different

	1 hid. layer			3 hid. layers			5 hid. layers
	DSVR	DNNR	$p$	DSVR	DNNR	$p$	DSVR	DNNR	$p$
abalone	1.46	1.47	0.04	1.46	1.45	0.82	1.46	1.47	0.00
bodyfat ( $\times$ 10 ${}^{2}$ )	0.13	0.16	0.00	0.17	0.14	0.09	0.13	0.13	0.58
cadata ( $/$ 10 ${}^{4}$ )	3.54	3.76	0.00	3.33	3.48	0.00	3.29	3.55	0.00
cpusmall	2.13	2.16	0.24	2.06	2.08	0.36	2.04	2.20	0.00
housing	2.25	2.66	0.00	2.04	2.18	0.02	2.07	2.10	0.44
mg ( $\times$ 10 ${}^{2}$ )	9.08	9.25	0.09	9.60	9.18	0.00	9.23	9.38	0.16
mpg	1.90	1.95	0.04	2.01	2.04	0.66	2.04	2.04	0.93
space_ga ( $\times$ 10 ${}^{2}$ )	0.08	0.08	0.64	0.07	0.07	0.31	0.07	0.07	0.32

Table 7

Mean squared errors and $p$ values for DSVR and DNNR models with one, three and five hidden layers. Smaller values in bold face when squared error distributions are statistically different

	1 hid. layer			3 hid. layers			5 hid. layers
	DSVR	DNNR	$p$	DSVR	DNNR	$p$	DSVR	DNNR	$p$
abalone	4.51	4.39	0.35	4.53	4.35	0.88	4.57	4.46	0.00
bodyfat ( $\times$ 10 ${}^{4}$ )	0.11	0.13	0.00	0.13	0.13	0.14	0.13	0.14	0.54
cadata ( $/$ 10 ${}^{8}$ )	28.88	30.14	0.00	27.49	27.24	0.00	26.37	26.95	0.00
cpusmall	8.79	8.86	0.32	8.12	8.18	0.52	8.24	8.62	0.00
housing	13.35	17.05	0.00	10.50	11.41	0.03	10.14	10.27	0.40
mg ( $\times$ 10 ${}^{4}$ )	147.58	144.26	0.15	164.54	144.89	0.00	158.91	146.88	0.90
mpg	7.19	7.41	0.01	7.96	8.08	0.64	8.35	8.11	0.91
space_ga ( $\times$ 10 ${}^{4}$ )	107.82	105.24	0.82	103.49	101.61	0.53	101.57	93.32	0.62

We will work with the same CV based hyperparametrization scheme of the previous section but for classification we will use here the $f1$ score for hyperparameter selection, as it involves both recall and precision and gives a more balanced performance measure. We report accuracy values in Table 5, where for easier reading we put in boldface the largest value for each architecture. This has only a descriptive value and a more relevant comparison could be made applying statistical significance results. To try to obtain such a slightly more precise comparison this way, we will pair for each problem and DNN architecture the DSVC and DSVR test accuracies. Since there are 9 dataset and 3 DNN architectures, this gives in principle a paired sample with 27 values; after this grouping, the accuracy averages are 88.75 $\pm$ 9.02 for DNNCs and 88.86 $\pm$ 9.13 for DSVCs, very similar. We can apply a Wilcoxon signed rank test on these 27 test accuracies, but with the clear caveat that these accuracies are probably not independent, as the three models for each problem share the same test set. Thus the following results must be seen in this light. It turns out that the resulting value is $p=$ 0.99; if the test accuracies were independent, this would imply that we cannot reject the null hypothesis of the two paired samples come from the same distribution. Therefore, we could conclude that the performance of DNNC and DSVC models is very similar. However, as said before, accuracy independence is not guaranteed, and the preceding can only be taken as a hint of similar performance.

The situation in regression will be slightly different. Notice that if we retain MAE as the validation score, as done in [15], this would help the DSVR models in detriment of DNNR models that aim to minimize the squared error. And inversely, if mean squared error (MSE) is used as the validation score, DNNRs would have then the advantage over DSVRs. Because of this we will compare DNNRs and DSVRs hyperparametrized using both MAE and MSE for CV scoring. Of course, this will result in possibly different hyperparameters but also on a fairer model comparison.

MAE scores are given in Table 6 and MSE ones in Table 7. Both tables also give $p$ values which have been derived as follows. While, in classification, network outputs are only indirectly related with, say, accuracies, in regression the MAE and MSE values are just the averages of the absolute and square error per pattern. This suggest to perform model comparison by testing the distributions of the absolute and squared errors for each dataset and model. The $p$ values of both tables correspond to the aplication of Wilcoxon signed ranked test on the error distributions of model outputs over the test sub-folds; recall that we apply here the previously described nested 4-fold CV scheme.

Both tables show in bold face the smallest MAE or MSE values when the null hypothesis of the error distributions being the same can be rejected at the $p=$ 0.05 level. As it can be seen in Table 6, DSVRs with 1 hidden layer give a smaller MAE in 5 datasets, in 2 datasets when they have 3 hidden layers and in 3 datasets when having 5 layers; on the other hand, DNNRs give a smaller MAE in just one problem using three hidden layers. We also put in italics the smallest values over the three architectures considered when it is significant at the 0.05 level; this is achieved in 5 problems by a DSVR models and in none by a DNNR one. Thus, DSVR have a small advantage here.

The situation is more balanced with respect to MSE values. As it can be seen in Table 7, DSVRs with 1 hidden layer give a smaller MSE in 4 datasets, in 1 dataset when they have 3 hidden layers and in 2 datasets when having 5 layers; in contrast, DNNRs gives a smaller MSE in two problems using three hidden layers and in one using five. We put also here in italics the smallest values over the three architectures considered when it is significant at the 0.05 level; this is achieved by a DSVR model in 3 problems and in one by a DNNR one. While DSVR also seem to the advantage here, is clearly smaller than in the case of MAEs.

Summing things up, we can say that the classification performance of DNNC and DSVC models is fairly similar but, on the other hand, DSVR models seem to perform clearly better for regression, when MAE is the preferred metric and also slightly so when MSE is considered. As a possible reason for this is that DSVR models can tune an extra parameter, the $\epsilon$ insensitivity used in their losses, having thus more flexibility.

6. Advanced architectures

In this Section we will compare the hinge and cross entropy losses on three large image datasets using more advanced DNN architectures. More precisely, we will first apply two class versions of the well known LeNet-5 convolutional neural network [16] on both the MNIST and MNIST Fashion problems. While quite simple for today’s deep proposals, LeNet-5 has still a fairly complex structure, depicted for instance in Fig. 2 of [16]. Next, we will consider a more advanced Residual Network architecture, ResNet32 V1, on the CIFAR 10 image classification problem. The ResNet structure is still more complicated, as it can be seen, for instance, in Fig. 3 of [17].

Basic dataset information is given in Table 8. GSVC models are very costly to hyperparametrize now, but for illustration purposes we will also give results for LIBLINEAR, the ad-hoc implementation for linear SVMs. While in general not competitive with GSVCs, this is less so for high dimensional problems, as the ones under consideration here. While all of them are 10 class problems, we will consider for each datasets ten 2-class problems where each single class has to be distinguished from the remaining nine classes.

Table 8
Number of train and test patterns and dimension in one versus rest MNIST, MNIST fashion and CIFAR 10 problems

	n. patterns train	n. patterns test	Dimension
MNIST	60,000	10,000	1,024
MNIST fashion	60,000	10,000	1,024
CIFAR 10	50,000	10,000	3,072

6.1 The MNIST problem

MNIST is by now a classical character recognition problem that has been widely used as a DNN benchmark, with many architectures being proposed over the years. According to Rodrigo Benenson’s web page [22], the state of the art networks as of 2016 achieve a test error of 0.21% [23]. The error rate when we apply the LeNet-5 implementation on the full MNIST 10 class problem is of 1.11%; a 0.95% error rate is reported in Y. LeCun web page [24] for LeNet-5. (Notice that here and the following experiments different executions may result in slight random changes in accuracy values). In our experiments we will use the MNIST dataset available in [24], which has 60,000 training and 10,000 test patterns given as 28 $\times$ 28 gray scale images. We will pad them on their boundaries with zeros to work with 32 $\times$ 32 images for better working with convolutional layers.

Our LeNet-5 implementation is fairly standard, with two convolutional layers with 3 $\times$ 3 filters and 6 and 16 output channels respectively, each followed by two dimensional averaging pooling. These are followed by a first dense layer with 120 units and a second one with 84 units, both with ReLU activations. For our two class problems we will replace the 10 unit, softmax outputs of LeNet-5 for the standard 10 class MNIST problem with single unit sigmoidal and linear outputs for the binary cross entropy and hinge losses, respectively. We will use Tikhonov regularization in the dense and output layers, with different penalties $\alpha_{S}$ and $\alpha_{R}$ for the output and dense weights respectively. The corresponding values are $\{0,10^{-8},10^{-6},10^{-4},10^{-3}\}$ for both the $\alpha_{R}$ and $\alpha_{S}$ .

Table 9
Accuracy comparisons of DNNC, DSVC and LibLinear for LeNet5 nets on MNIST (left: train vs test, right: test vs train; larger accuracies in bold face)

	Train vs test			Test vs train
	DNNC	DSVC	LIBLIN	DNNC	DSVC	LIBLIN
0	99.88	99.88	99.09	99.76	99.70	98.59
1	99.91	99.87	99.44	99.67	99.70	98.37
2	99.82	99.79	98.12	99.50	99.40	97.27
3	99.87	99.79	97.91	99.48	99.57	96.78
4	99.87	99.77	98.22	99.52	99.55	97.52
5	99.77	99.71	97.93	99.55	99.48	96.60
6	99.72	99.82	98.69	99.67	99.51	98.14
7	99.62	99.63	98.28	99.45	99.45	97.81
8	99.82	99.77	96.23	99.16	99.38	95.14
9	99.63	99.56	96.57	99.47	99.29	95.47
Ave	99.79	99.76	98.05	99.52	99.50	97.17

We report test accuracies under two different scenarios, a first one were we hyperparametrize and train the networks on the training set and evaluate them on the test subset, and a more difficult one, where we switch the datasets, hyperparametrizing and training now on the 10,000 pattern test set and testing the resulting models on the 50,000 patterns of the train datasets. We also use here $f1$ score for CV hyperparametrization; notice that now the 2-class problem are fairly unbalanced, with the positive classes having 9 times less patterns than the negative ones. Test results are given in Table 9 for the first scenario (left) and for the second one (right); the table also shows mean accuracies. For easier reading we put in bold face largest accuracies for each problem. This has only an illustrative value but clearly show that LIBLINEAR models fall behind. To have a more meaningful comparison we have performed a Wilcoxon test over the DNNC and DSVC accuracies along the lines detailed for the classification problems in Section 5; sample size is now 20 (10 train vs test and 10 test vs train problems). The resulting $p$ value is 0.14, slightly above the rejection value for the same distribution hypothesis at the 0.1 level and, hence, we cannot guarantee that the DNNC and DSVC accuracy distributions are different.

6.2 The MNIST fashion problem

We will use the same LeNet-5 network on the MNIST Fashion dataset [25]. It also contains 60,000 training and 10,000 test patterns in 10 classes of clothing products (trousers, pullovers, …) given as 28 $\times$ 28 gray scale images, which we pad again to size 32 $\times$ 32. We have downloaded the dataset from the Zalando Research github page [26]. According to that page, the state of the art accuracy is 96.7%, achieved using wide residual networks; the test accuracy for the entire 10 class problem of our LeNet-5 network without any hyperparameterization is 90.92%.

We have performed over the MNIST Fashion dataset the same two-class experiments done for standard MNIST, hyperparameterizing again separately the Tikhonov regularization penalties of the dense hidden layer weights and of the output weights; we have used the same $\alpha_{S}$ and $\alpha_{R}$ than for MNIST. The 2-class problems considered are again fairly imbalanced and, thus, we also use the $f1$ score for validation. Accuracy test results are given now in Table 10 for the train vs test problem (left) and for the test vs train one (right); the table also shows mean accuracies. Again, for easier reading we put in bold face largest accuracies for each problem; clearly the table shows that LIBLINEAR models also fall behind. For a more meaningful comparison we have performed a Wilcoxon test along the previous lines over the DNNC and DSVC accuracies; sample size is again 20. The resulting $p$ value is 0.48, which implies we cannot reject the same accuracy distribution hypothesis.

Table 10
Accuracy comparisons of DNNC, DSVC and LibLinear for LeNet5 nets on MNIST fashion (left: train vs test, right: test vs train; larger accuracies in bold face)

	Train vs test			Test vs train
	DNNC	DSVC	LIBLIN	DNNC	DSVC	LIBLIN
0	97.14	97.04	95.89	96.02	95.99	95.58
1	99.69	99.73	99.36	99.60	99.51	99.39
2	96.86	97.26	94.22	96.09	95.71	94.21
3	98.31	98.19	96.47	97.43	97.62	96.66
4	97.34	97.18	94.80	95.84	95.93	94.61
5	99.74	99.72	98.46	99.31	99.24	97.44
6	94.62	95.25	92.63	93.92	93.50	92.51
7	99.33	99.31	98.15	98.72	98.92	97.59
8	99.75	99.69	98.59	99.52	99.33	98.38
9	99.44	99.38	98.67	99.12	99.12	98.23
Ave	98.22	98.28	96.72	97.56	97.49	96.46

6.3 The CIFAR 10 problem

Our last experiment with advanced deep architectures will deal with the CIFAR-10 dataset. It contains 60,000 32 $\times$ 32 RGB color images distributed in 10 classes (airplane, automobile, bird, …), of which 50,000 are for training and the remaining 10,000 for test; each class has 6000 images per class, While the training subset is further divided into five training batches, we won’t use them.

We shall use the ResNet32 V1 implementation in [27] of the deep residual networks proposed in Subsection 4.2 of [17]. The network has three stacks of five residual blocks, each with two convolution layers plus a residual connection. The convolutional layers at each stack result in 16, 32 and 64 output channels respectively. Originally batch normalization is applied but no regularization takes place; moreover, the outputs of the last convolutional layer are flattened and fed into ten softmax model outputs. This results in a total of about 468,000 trainable weights. In our case we just feed these flattened outputs into either a sigmoid output for the DNNC model or a linear one for the DSVC one. We point out that training here is quite costly, with a single epoch taking about 3 minutes. Because of this we have performed a limited hyperparametrization in our CIFAR 10 experiments, considering only $\alpha_{S}$ values in $\{0,10^{-12},10^{-10},10^{-8}\}$ .

The test accuracy reported in [17] is 92.49% using data augmentation; the best accuracy in [22] is 96.53%. With our implementation we have achieved a 85.33% accuracy after 200 epochs without data augmentation and 92.04% with it. Test results are given now in Table 11 for the train vs test problem (left) and for the test vs train one (right); the table also shows mean accuracies. Again, for easier reading we put in bold face largest accuracies for each problem. LIBLINEAR results are quite bad, as it seems to assign all patterns to the majority class. Applying a Wilcoxon signed rank test along the previous lines yields a $p$ value of 0.41; again, this implies we cannot reject the same distribution hypothesis for accuracies.

Table 11
Accuracy comparisons of DNNC, DSVC and LibLinear for ResNet32 V1 nets on CIFAR 10 (left: train vs test, right: test vs train; larger accuracies in bold face)

	Train vs test			Test vs train
	DNNC	DSVC	LIBLIN	DNNC	DSVC	LIBLIN
0	96.46	96.00	90.11	93.52	93.41	90.50
1	98.42	98.43	90.92	96.70	96.76	90.68
2	94.36	94.04	90.00	90.49	91.15	90.00
3	92.48	90.00	90.00	90.10	90.10	90.00
4	96.03	96.10	90.00	89.93	91.23	90.00
5	94.19	94.94	90.00	90.78	92.19	90.00
6	96.66	96.48	90.01	93.92	93.61	90.00
7	97.20	96.92	90.64	93.71	94.34	90.58
8	97.75	97.53	90.56	95.76	96.08	90.00
9	97.55	97.76	90.37	95.12	95.73	90.00
Ave	96.11	95.82	90.26	93.00	93.46	90.18

Finally, and as a summary of the previous results, we give in Table 12 the sample sizes, the values of the Wilcoxon test statistics (i.e., the values returned by the Wilcoxon test algorithm) and the $p$ values corresponding to these statistics, for the MNIST, MNIST Fashion and CIFAR 10 problems; we also give these values when considering together the DNNC and DSVC accuracies for the 60 problems considered in this Section. Contrary to what was discussed for classification problems in Section 5, now all the train and test datasets are different. As it can be seen, all the $p$ values are above the 0.05 level, which implies that in no case we can reject the null hypothesis of the accuracy distributions being the same; this justifies the conclusion of DNNC and DSVC models perform similarly in these two-class classification problems.

Table 12

Summary of Wilcoxon’s signed ranked test results for the accuracies in the MNIST, MNIST Fashion and CIFAR 10 problems, as well as for all these problems

	Mnist	Fashion	Cifar	All
Sample size	20	20	20	60
Wilcox. test statistic	51.50	77.50	74.50	735.00
$p$ _value	0.14	0.48	0.41	0.61

7. Conclusions and further work

We have proposed deep DSVC and DSVR models which jointly learn an optimal last hidden layer representation and a maximum margin hyperplane acting upon it. Our first motivation is that Gaussian SVMs are not time competitive over large datasets. A second motivation is that the mild non differentiability of the hinge and $\epsilon$ -insensitive losses used in SVM models can be automatically handled by modern DNN packages.

After reviewing the results in [15], we have compared our proposals against standard DNN cross entropy classifiers or squared error regressors. We do so because the counterparts in large problems of our deep SVM proposals are precisely standard DNNs. Our experiments tentatively show that DNNC and DSVC models perform similarly in classification problems, as we cannot statistically reject the null hypothesis of equal DNNC and DSVC accuracies over two-class problems (with the caveats mentioned in Section 5). The situation is different, however, in regression problems. In principle, results should be now influenced by the scoring function used to hyperparametrize regularization penalties. Indeed, when using the mean absolute error (MAE) for this, we can reject in favor of our DSVR models the null hypothesis of the DSVR and DNNR error distribution being the same. On the other hand, one would expect a better DNNR performance when using mean squared error (MSE) CV scoring; however, this turns out not to be so and DSVR models have also a light advantage here. We point to the extra $\epsilon$ DSVR hyperparameter as a possible reason for this.

As lines for further work,we first mention that large margins for DNNs have been studied in [28, 29, 30, 31], in some cases using sigmoid/softmax outputs and binary or categorical cross entropies. On the other hand, and as mentioned, Deep SVMs naturally maximize the margins achieved in the last hidden layer. It is thus natural to compare both margins for classification problems. A second research line has to do with the kernels induced by Deep SVMs on the last hidden layer, as there has recently been a flurry of activity on possible kernel structures on the last hidden layer of highly overparametrized DNNs [32, 33, 34, 35]). It is thus natural to study the possible relationship between these kernels and those induced by Deep SVMs on the rather wide last hidden layers considered here. Finally, one of the main advantages of the current DNN paradigms is the great flexibility they allow to define neural architectures (see for instance [36, 37, 38, 39, 40, 41]), which results in their wide application [42, 43, 44, 45, 46, 47]; thus, a natural question is how to take advantage of losses such as the hinge or $\epsilon$ -insensitive used here, to enhance the performance of other neural models. We are currently considering these and other related questions.

Footnotes

Acknowledgments

With partial support from Spain’s grants TIN2016-76406-P. Work supported also by the UAM-ADIC Chair for Data Science and Machine Learning. We gratefully acknowledge the use of the facilities of Centro de Computación Científica (CCC) at UAM.

References

Schölkopf

Smola

. Learning with Kernels: support vector machines, regularization, optimization, and beyond. Adaptive computation and machine learning series. MIT Press; 2002.

Bordes

Ertekin

Weston

Bottou

. Fast kernel classifiers with online and active learning. Journal of Machine Learning Research. 2005; 6: 1579–1619.

Claesen

Smet

Suykens

JAK

Moor

. EnsembleSVM: a library for ensemble learning using support vector machines. Journal of Machine Learning Research. 2014; 15(1): 141–145.

Joachims

. Sparse kernel SVMs via cutting-plane training. Machine Learning. 2009; 76(2–3): 179–193.

Wen

Shi

Chen

. ThunderSVM: a fast SVM library on GPUs and CPUs. Journal of Machine Learning Research. 2018; 19(21): 1–5.

Hsieh

Chang

Lin

. Large linear classification when data cannot fit in memory. TKDD. 2012; 5(4): 23:1–23:23.

Hsieh

Chang

Lin

Keerthi

Sundararajan

. A dual coordinate descent method for large-scale linear SVM. In: Machine Learning, Proceedings of the Twenty-Fifth International Conference (ICML 2008), Helsinki, Finland, June 5–9, 2008; 2008. pp. 408–415.

Shalev-Shwartz

Singer

Srebro

Cotter

. Pegasos: primal estimated sub-gradient solver for SVM. Math Program. 2011; 127(1): 3–30.

Steinwart

Hush

Scovel

. Training SVMs without offset. J Mach Learn Res. 2011; 12: 141–202.

10.

Tang

. Deep Learning using Support Vector Machines. CoRR. 2013; abs/1306.0239. Available from: http://arxiv.org/abs/1306.0239.

11.

Zhang

Liu

Yao

Gong

. Deep neural support vector machines for speech recognition. In: 2015 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2015, South Brisbane, Queensland, Australia, April 19–24, 2015; 2015. pp. 4275–4279.

12.

Chollet

. Keras: Deep Learning library for Theano and TensorFlow; 2015. Available from: https://github.com/fchollet/keras.

13.

Rudin

Carlson

. The Secrets of Machine Learning: Ten Things You Wish You Had Known Earlier to be More Effective at Data Analysis. CoRR. 2019; abs/1906.01998. Available from: http://arxiv.org/abs/1906.01998.

14.

Angelino

Larus-Stone

Alabi

Seltzer

Rudin

. Learning certifiably optimal rule lists for categorical data. J Mach Learn Res. 2017; 18: 234:1–234:78.

15.

Díaz-Vico

Prada

Omari

Dorronsoro

. Deep Support Vector Classification and Regression. In: From Bioinspired Systems and Biomedical Applications to Machine Learning – 8th International Work-Conference on the Interplay Between Natural and Artificial Computation, IWINAC 2019, Almería, Spain, June 3–7, 2019, Proceedings, Part II; 2019. pp. 33–43.

16.

LeCun

Bottou

Bengio

Haffner

. Gradient-based learning applied to document recognition. Proceedings of the IEEE. 1998; 86: 2278–2323.

17.

Zhang

Ren

Sun

. Deep Residual Learning for Image Recognition. In: IEEE Conference on Computer Vision and Pattern Recognition; 2016. pp. 770–778.

18.

Lázaro

Dorronsoro

. Simple proof of convergence of the SMO algorithm for different SVM variants. IEEE Trans Neural Netw Learning Syst. 2012; 23(7): 1142–1147.

19.

Manton

Amblard

. A primer on reproducing kernel hilbert spaces. Foundations and Trends in Signal Processing. 2015; 8(1–2): 1–126.

20.

Kingma

. Adam: A Method for Stochastic Optimization. In: 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7–9, 2015, Conference Track Proceedings; 2015. Available from: http://arxiv.org/abs/1412.6980.

21.

Chang

Lin

. LIBSVM: a library for support vector machines. ACM Transactions on Intelligent Systems and Technology. 2011; 2(3): 27:1–27:27. Software available at http://www.csie.ntu.edu.tw/c̃jlin/libsvm.

22.

Benenson

. What is the Class of This Image? Available from: http://rodrigob.github.io/are_we_there_yet/build/classification_datasets_results.html.

23.

Wan

Zeiler

Zhang

LeCun

Fergus

. Regularization of Neural Networks Using Dropconnect. In: Proceedings of the 30th International Conference on International Conference on Machine Learning, Vol. 28; 2013. pp. III–1058–III–1066.

24.

LeCun

. The MNIST Database of Handwritten Digits. Available from: http://yann.lecun.com/exdb/mnist/.

25.

Xiao

Rasul

Vollgraf

. Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms. CoRR. 2017; abs/1708.07747.

26.

Fashion-MNIST

. Available from: https://github.com/zalandoresearch/fashion-mnist.

27.

Keras Examples: CIFAR-10 ResNet. Available from: https://keras.io/examples/cifar10_resnet/.

28.

Sun

Chen

Wang

Liu

. Large Margin Deep Neural Networks: Theory and Algorithms. ArXiv. 2015; abs/1506. 05232.

29.

Liu

Wen

Yang

. Large-Margin Softmax Loss for Convolutional Neural Networks. In: Proceedings of The 33rd International Conference on Machine Learning, Vol. 48 of Proceedings of Machine Learning Research; 2016. pp. 507–516.

30.

Liang

Wang

Lei

Liao

. Soft-Margin Softmax for Deep Classification. In: Liu

Xie

Zhao

El-Alfy

ESM

, eds. ICONIP. Neural Information Processing. Springer International Publishing; 2017. pp. 413–421.

31.

Elsayed

Krishnan

Mobahi

Regan

Bengio

. Large Margin Deep Networks for Classification. In: Bengio

Wallach

Larochelle

Grauman

Cesa-Bianchi

Garnett

, eds. Advances in Neural Information Processing Systems 31; 2018. pp. 842–852.

32.

Jacot

Gabriel

Hongler

. Neural Tangent Kernel: Convergence and Generalization in Neural Networks. In: Bengio

Wallach

Larochelle

Grauman

Cesa-Bianchi

Garnett

, eds. Advances in Neural Information Processing Systems 31; 2018. pp. 8571–8580.

33.

Arora

Salakhutdinov

Wang

. On Exact Computation with an Infinitely Wide Neural Net. CoRR. 2019; abs/1904.11955.

34.

Zhang

Bengio

Hardt

Recht

Vinyals

. Understanding deep learning requires rethinking generalization. ArXiv. 2016; abs/1611.03530.

35.

Olson

Wyner

Berk

. Modern Neural Networks Generalize on Small Data Sets. In: Bengio

Wallach

Larochelle

Grauman

Cesa-Bianchi

Garnett

, eds. Advances in Neural Information Processing Systems 31; 2018. pp. 3619–3628.

36.

Ahmadlou

Adeli

. Enhanced probabilistic neural network with local decision circles: a robust classifier. Integrated Computer-Aided Engineering. 2010 8; 17: 197–210.

37.

Rafiei

Adeli

. A new neural dynamic classification algorithm. IEEE Trans Neural Networks Learn Syst. 2017; 28(12): 3074–3083.

38.

Soares

Fernandes

BJT

Filho

CJAB

. Structured pyramidal neural networks. Int J Neural Syst. 2018; 28(5): 1750021:1–1750021:22.

39.

Antoniades

Spyrou

Martín-Lopez

Valentín

Alarcón

Sanei

, et al. Deep neural architectures for mapping scalp to intracranial EEG. International Journal of Neural Systems. 2018; 28(8): 1850009:1–1850009:15.

40.

Bîlbîe

Paun

Pan

Neri

. Simplified and yet turing universal spiking neural p systems with communication on request. International Journal of Neural Systems. 2018; 28(8): 1850013:1–1850013:19.

41.

Pupo

OGR

Ventura

. Performing multi-target regression via a parameter sharing-based deep network. International Journal of Neural Systems. 2019; 29(9): 1950014:1–1950014:22.

42.

Nguyena

Kashania

Bordasb

Ngo

. Deep neural network with higher order neuron for the prediction of foamed concrete strength. Computer-Aided Civil and Infrastructure Engineering. 2019; 34(5): 316–332.

43.

Liang

. Image-based post-disaster inspection of reinforced concrete bridge systems using deep learning with bayesian optimization. Computer-Aided Civil and Infrastructure Engineering. 2019; 34(5): 415–430.

44.

Maeda

Ogawa

Haseyama

Takahashi

. Convolutional sparse coding-based deep random vector functional link network for distress classification of road structures. Computer-Aided Civil and Infrastructure Engineering. 2019; 34(8): 654–676.

45.

Gao

Kong

Mosalam

. Deep leaf-bootstrapping generative adversarial network for structural image data augmentation. Computer-Aided Civil and Infrastructure Engineering. 2019; 34(9): 755–773.

46.

Ankush Singla

Jahanshahi

Bertino

Verma

. Pruning deep convolutional neural networks for efficient edge computing in condition assessment of civil infrastructures. Computer-Aided Civil and Infrastructure Engineering. 2019; 34(9): 774–789.

47.

Zhang

Cheng

Ren

. A graph deep learning method for short-term traffic forecasting on large road networks. Computer-Aided Civil and Infrastructure Engineering. 2019; 34(10): 431–449.

Deep support vector neural networks

Abstract

Keywords

1. Introduction

2.1 Primal and dual SVC and SVR

Table 1 Number of train and (when available) test patterns and dimensions in the two-class (top) and regression (bottom) problems

Table 6 Mean absolute errors and p values for DSVR and DNNR models with one, three and five hidden layers. Smaller values in bold face when absolute error distributions are statistically different

Table 8 Number of train and test patterns and dimension in one versus rest MNIST, MNIST fashion and CIFAR 10 problems

Table 9 Accuracy comparisons of DNNC, DSVC and LibLinear for LeNet5 nets on MNIST (left: train vs test, right: test vs train; larger accuracies in bold face)

Table 10 Accuracy comparisons of DNNC, DSVC and LibLinear for LeNet5 nets on MNIST fashion (left: train vs test, right: test vs train; larger accuracies in bold face)

Table 11 Accuracy comparisons of DNNC, DSVC and LibLinear for ResNet32 V1 nets on CIFAR 10 (left: train vs test, right: test vs train; larger accuracies in bold face)

Footnotes

Acknowledgments

References

Table 1
Number of train and (when available) test patterns and dimensions in the two-class (top) and regression (bottom) problems

Table 6
Mean absolute errors and $p$ values for DSVR and DNNR models with one, three and five hidden layers. Smaller values in bold face when absolute error distributions are statistically different

Table 8
Number of train and test patterns and dimension in one versus rest MNIST, MNIST fashion and CIFAR 10 problems

Table 9
Accuracy comparisons of DNNC, DSVC and LibLinear for LeNet5 nets on MNIST (left: train vs test, right: test vs train; larger accuracies in bold face)

Table 10
Accuracy comparisons of DNNC, DSVC and LibLinear for LeNet5 nets on MNIST fashion (left: train vs test, right: test vs train; larger accuracies in bold face)

Table 11
Accuracy comparisons of DNNC, DSVC and LibLinear for ResNet32 V1 nets on CIFAR 10 (left: train vs test, right: test vs train; larger accuracies in bold face)