An application of a practical neural network model for solving support vector regression problems

Abstract

This paper offers a recurrent neural network to support vector machine (SVM) learning in regression arising widespread applications in a variety of setting. The SVM learning problem in regression is first converted into an equivalent quadratic programming (QP) formulation. An artificial neural network for SVM learning is then proposed. The presented neural network framework guarantees to obtain the optimal solution of the support vector regression (SVR). The existence and convergence of the trajectories of the network are studied. The Lyapunov stability for the considered neural network is also shown. Two illustrative examples provide a further demonstration of the effectiveness of the method.

Keywords

Neural network support vector regression quadratic optimization convergent stability

1. Introduction

Support vector machines (SVMs) are powerful tools for data classification and regression [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21]. In the recent years, many fast algorithms for SVMs have been developed [2, 4, 9, 10, 11, 12, 16, 17, 19]. In many engineering and military applications, the demands on real-time data processing is often needed, such as classification in complex electromagnetic environments, recognition in medical diagnostics radar object recognition in strong background clutter, etc. [7]. However, the conventional numerical methods require much more computation time and cannot satisfy real-time requirement (see [22, 23, 24, 25]). One possible and promising approach to train SVMs in real-time is to employ recurrent neural networks based on circuit implementation [26]. Recurrent neural network is hardware-implementable, since it can be implemented physically by designated hardware, such as application-specific integrated circuits.

In the past decades, recurrent neural networks as a software and hardware implementable approach for optimization have been widely investigated. By employing artificial neural networks based on analog circuits implementation, the computing procedures are physically parallel and distributed. After the pioneering work in this field by Hopfield and Tank[27, 28], great interests have been raised for designing neural networks with analog circuits implementation in a variety of engineering applications (see [29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67] and references therein). It is well known that SVMs can be modeled as QP problems, and therefore can be solved by some recurrent neural networks capable of solving this type of optimization problems, e.g. [29, 41, 53, 56, 60, 65, 67]. These papers present several recurrent neural networks to train SVMs. These models can also reach the exact saddle point of the QP problem in SVM problems and are globally stable in the Lyapunov sense. But the structure of some of them is rather complicated and further simplification can be achieved. Moreover, the QP formulation of SVM problems may not be strictly convex in many applications (see remark 1 in [65]). In fact some other networks can potentially train SVMs without strict convexity assumption (e.g. [33, 36, 37]). Thus, on the basis of the above notations, proposing an efficient neural network for solving the SVR problems as a class of SVMs with a simple structure, lower model complexity, good stability and convergence results which can also solve QP formulation with only convexity assumption is very necessary and meaningful.

With motivation from the above discussions, in this paper we suggest a suitable neural network for analog hardware implementation which gives a good solution of SVR learning. According to the Karush-Kuhn-Tucker (KKT) optimality conditions of convex programming [74], a neural network model for solving the QP formulation of the SVR is studied. The equilibrium point of the proposed neural network is proved to be equivalent to the KKT point of the QP formulation. The existence and uniqueness of an equilibrium point of the network is also analyzed. By constructing a suitable Lyapunov function, a sufficient condition to ensure the existence and global asymptotical stability for the unique equilibrium point of the neural network is obtained.

The remainder of this paper is organized as follows. In Section 2, we briefly describe the learning problem of SVMs for regression. In Section 3, the system model for SVR problem and some necessary preliminaries are given. A comparison with some existing models is given in Section 4. The stability and convergence properties of the designed model are studied in Section 5. Two numerical experiments are provided in Section 6. Finally, some concluding remarks are given in Section 7.

2. Support vector regression problem

Consider the problem of approximating a set of data

$\{(x_{1},y_{1}),(x_{2},y_{2}),\dots(x_{N},y_{N})\},$

with a regression function of the form

$\phi(x)=\sum_{i=1}^{N}\alpha_{i}\Phi_{i}(x)+\beta,$ (1)

where $\{\Phi_{i}(x)\}_{i=1}^{N}$ are called feature functions defined in a high dimensional space, $\beta$ and $\{\alpha_{i}\}_{i=1}^{N}$ are parameters of the model to be estimated. Here we will construct a continuous-time recurrent neural network to estimate these parameters. Utilizing the Huber loss function [60], the regression function defined in Eq. (1) can be represented as

$\phi(x)=\sum_{i=1}^{N}\theta_{i}K(x,x_{i})+\beta,$ (2)

where $K(x,x_{i})$ is a kernel function satisfying $K(x,x_{i})=\Phi(x)^{T}\Phi(x_{i})(i=1,...,N).$

Based on the problem formulation in [60], $\theta_{i}(i=1,2,...,N)$ is the optimal solution of the following QP problem:

$\displaystyle{\rm min}∼{}\frac{1}{2}\sum_{i=1}^{N}\sum_{j=1}^{N}\theta_{i}% \theta_{j}K(x_{i},x_{j})-\sum_{i=1}^{N}\theta_{i}y_{i}+\frac{\epsilon}{2\zeta}% \sum_{i=1}^{N}\theta_{i}^{2}$ (3)

$\displaystyle{\rm subject\ to}\left\{\begin{array}[]{ll}\displaystyle\sum_{i=1% }^{N}\theta_{i}=0,\\ -\zeta\leqslant\theta_{i}\leqslant\zeta,∼{}∼{}i=1,...,N,\end{array}\right.$ (4)

where $\epsilon>0$ is an accuracy parameter required for the approximation and $\zeta>0$ is a pre-specified parameter. From [41], $\beta$ in Eq. (2) is achieved by

$\displaystyle\beta=\frac{1}{N}\sum_{j=1}^{N}\left(y_{j}-\sum_{i=1}^{N}\theta_{% i}K(x_{j},x_{i})\right).$

Let

$\displaystyle\left\{\begin{array}[]{ll}Q=\{K(x_{i},x_{j})\}_{N\times N}+\frac{% \epsilon}{\zeta}I_{N\times N},\\ d=-(y_{1},y_{2},...,y_{N})^{T},\\ e=(1,1,...,1)^{T}\in\mathbb{R}^{N},\\ D=diag({1,1,...,1})\in\mathbb{R}^{N\times N},\\ A=\left(\begin{array}[]{cccccc}D\\ -D\end{array}\right),\\ b=(\zeta,\zeta,...,\zeta)^{T}\in\mathbb{R}^{2N},\\ \theta=(\theta_{1},\theta_{2},...,\theta_{N})^{T}.\end{array}\right.$

Then the optimization problem Eqs (3) and (4) can be equivalently written as the following compact form:

$\displaystyle{\rm min}∼{}∼{}∼{}\frac{1}{2}\theta^{T}Q\theta+d^{T}\theta$ (5) $\displaystyle\text{subject∼{} to}\left\{\begin{array}[]{ll}A\theta-b\leqslant 0% ,\\ e^{T}\theta=0.\\ \end{array}\right.$ (6)

Throughout this paper, we assume that the problem Eqs (5) and (6) has a unique optimal solution. In the next section, we will try to propose a high performance neural network model for solving QP problem Eqs (5) and (6) and discuss its architecture.

3. A neural network model

This section shows how the dynamic system in this study is formed. It is well known [44] that the key steps in the neural network method lie in constructing the dynamic system and Lyapunov function. To this end, we first look into the KKT conditions of the problem Eqs (5) and (6) which are presented as below:

$\left\{\begin{array}[]{ll}\nu^{*}\geqslant 0,\ \ A\theta^{*}-b\leqslant 0,\ \ % {\nu^{*}}^{T}(A\theta^{*}-b)=0,\\ Q\theta^{*}+d+A^{T}\nu^{*}+e\vartheta^{*}=0,\\ e^{T}\theta^{*}=0.\end{array}\right.$ (7)

.

[73] $\theta^{*}\in\mathbb{R}^{N}$ is an optimal solution of Eqs (5) and (6) if and only if there exist $\nu^{*}\in\mathbb{R}^{2N}$ and $\vartheta^{*}\in\mathbb{R}$ such that $({\theta^{*}}^{T},{\nu^{*}}^{T},{\vartheta^{*}}^{T})^{T}$ satisfies the KKT system Eq. (7).

$\theta^{*}$ is called a KKT point of Eqs (5) and (6) and a pair $({\nu^{*}}^{T},{\vartheta^{*}}^{T})^{T}$ is called the Lagrangian multiplier vector corresponding to $\theta^{*}$ .

.

$(\nu+A\theta-b)^{+}=\nu$ if and only if

$\displaystyle\nu\geqslant 0,\ \ A\theta-b\leqslant 0,\ \ {\nu}^{T}(A\theta-b)=0,$

where

$\displaystyle(\nu+A\theta-b)^{+}=\left([(\nu+A\theta-b)_{1}]^{+},[(\nu+A\theta% -b)_{2}]^{+},\ldots,[(\nu+A\theta-b)_{m}]^{+}\right)^{T},$ $\displaystyle[(\nu+A\theta-b)_{k}]^{+}={\rm max}\{(\nu+A\theta-b)_{k},0\},∼{}k% =1,2,\ldots,m.$

Proof..

If $(\nu+A\theta-b)^{+}=\nu$ , then $\nu=0$ or $\nu>0.$ We have the following cases:

If $\nu=0,$ then $(\nu+A\theta-b)^{+}=(A\theta-b)^{+}=0;$ thus $A\theta-b\leqslant 0$ and ${\nu}^{T}(A\theta-b)=0.$

If $\nu>0,$ then $(\nu+A\theta-b)^{+}=\nu+A\theta-b=\nu;$ thus $A\theta-b=0$ and ${\nu}^{T}(A\theta-b)=0.$

∎

Now, let $\theta(.),$ $\nu(.)$ and $\vartheta(.)$ be some time dependent variables. The aim is to construct a continuous-time neural network that will settle down to the KKT point of the problem Eqs (5) and (6). We propose a neural network for solving Eqs (5) and (6) as

$\displaystyle\frac{dy}{dt}=\tau\Psi(y),$ (8) $\displaystyle y(t_{0})=y_{0}=(\theta_{0}^{T},\nu_{0}^{T},\vartheta_{0}^{T})^{T% },∼{}\tau>0,$ (9)

where $y=(\theta^{T},\nu^{T},\vartheta^{T})^{T}\in\mathbb{R}^{3N+1},$ $\tau$ is a scale parameter and

$\displaystyle\Psi(y)=\left(\begin{array}[]{cccccc}-\left(Q\theta+d+A^{T}(\nu+A% \theta-b)^{+}+e\vartheta\right)\\ (\nu+A\theta-b)^{+}-\nu\\ e^{T}\theta\end{array}\right).$ (10)

It should be noted that $\tau$ indicates the convergence rate of the neural network Eqs (8) and (9). For simplicity of our analysis, we let $\tau=1.$ An indication on how the neural network Eqs (8) and (9) can be implemented on hardware is provided in Fig. 1.

Figure 1.

A simplified block diagram for the neural network Eqs (8) and (9).

4. Comparing with some existing models

The SVR Eqs (5) and (6) is a special case of degenerate convex quadratic programming (DCQP) problem [63] as

$\displaystyle{\rm min}∼{}∼{}∼{}∼{}\frac{1}{2}x^{T}Qx+D^{T}x$ (11)

subject to

$\displaystyle Ax-b\leqslant 0,$ (12) $\displaystyle Ex-f=0,$ (13)

where $Q\in\mathbb{R}^{n\times n}$ is a symmetric and positive semidefinite matrix, $b\in\mathbb{R}^{m}$ , $E\in\mathbb{R}^{l\times n},$ $f\in\mathbb{R}^{l},$ $x\in\mathbb{R}^{n}$ and ${\rm rank}(A)=m(0<m<n).$ A general model for solving the problem Eqs (11)–(13) is as

$\displaystyle\frac{dy}{dt}=\tau\psi(y),$ (14) $\displaystyle y(t_{0})=y_{0}=(x_{0}^{T},u_{0}^{T},v_{0}^{T})^{T}\in\mathbb{R}^% {n+m+l},∼{}\tau>0,$ (15)

where

$\displaystyle\psi(y)=\left(\begin{array}[]{cccccc}-\left(Qx+D+A^{T}(u+Ax-b)^{+% }+E^{T}v\right)\\ (u+Ax-b)^{+}-u\\ Ex-f\end{array}\right).$ (16)

It is clear that the proposed model Eqs (8) and (9) is a special case of the model Eqs (14) and (15). In order to see how well the presented neural network Eqs (14) and (15) can be applied to solve DCQP problems, we compare it with some existing neural network models.

Let us consider the following problem

$\displaystyle{\rm min}∼{}∼{}\ \frac{1}{2}x^{T}Qx+D^{T}x$ (17)

subject to

$\displaystyle Ax-b\leqslant 0,∼{}∼{}x\in\Omega,$ (18)

where $\Omega=\{x\in\mathbb{R}^{n},∼{}l_{i}\leqslant x_{i}\leqslant h_{i}∼{}{\rm for}% ∼{}i=1,2,\ldots,n\},$ some $h_{i}$ (or $-l_{j}$ ) could be $+\infty$ and $Ax-b\in\mathbb{R}^{m}$ . In [32], the Friesz projection neural network for solving Eqs (17) and (18) is given by

$\displaystyle\frac{dx}{dt}=-\left(x-P_{\Omega}(x-(Qx+D)-A^{T}u)\right),$ (19) $\displaystyle\frac{du}{dt}=-\left(u-(u+Ax-b)^{+}\right),$ (20)

where $\Omega\subset\mathbb{R}^{n}$ is a closed convex set and $P_{\Omega}:\mathbb{R}^{n}\rightarrow\Omega$ is a projection operator [33] defined by

$P_{\Omega}(x)={\rm argmin}_{v\in\Omega}||x-v||.$

It is well known [61] that the asymptotical stability of the dynamic model Eqs (19) and (20) for a monotone and a symmetric mapping could not be guaranteed. Thus, the system described by Friesz et al. cannot solve problem Eqs (17) and (18) where $Q$ is positive semidefinite matrix similar to linear programming (LP) problems. For instance, one can see the LP example in [49].

Under the condition that $Q$ is only positive definite matrix, we use a lagrangian network model in [45]for solving Eqs (11)–(13) as

$\displaystyle\frac{dx}{dt}=-\left(Qx+D+\frac{1}{2}A^{T}u^{2}+E^{T}v\right),$ (21) $\displaystyle\frac{du}{dt}={\rm diag}(u_{1},\ldots,u_{m})(Ax-b),$ (22) $\displaystyle\frac{dv}{dt}=Ex-f,$ (23)

with an initial point $(x_{0}^{T},u_{0}^{T},v_{0}^{T})^{T}$ and $u_{k}(t_{0})>0∼{}(k=1,\ldots,m).$ The global convergence of the neural network Eqs (21)–(23) is proved. However, this model can not LP optimization problems. One can see the LP Example in [48] for more clarification.

A gradient based neural network for solving Eqs (11)–(13) called Kennedy and Chua’s model, is given in [3] as

$\displaystyle\frac{dx}{dt}=-\left(Qx+D+\varsigma[A^{T}(Ax-b)^{+}+E^{T}(Ex-f)]% \right),$ (24)

where $\varsigma$ is a penalty parameter. This model is not able to find an exact optimal solution due to a finite penalty parameter and is difficult to implement when the penalty parameter is very large [30]. Thus, this network only converges to an approximate solution of Eqs (11)–(13) for any given finite penalty parameter. It can be also shown that neural network Eq. (24) is not globally convergent to an exact optimal solution of DCQP problems. For instance, one can see solved LP problem in [42]where this LP problem is exactly solved in [49].

Utilizing an NCP-function and the gradient method, authors in [46] formulated a neural network model for solving strictly convex quadratic programming problems as

$\displaystyle\frac{dy(t)}{dt}=-k\nabla E(y(t)),∼{}∼{}k>0,$ (25) $\displaystyle y(0)=y_{0},$ (26)

where

$\displaystyle E(y)=\frac{1}{2}\|\eta(y)\|^{2},$ (27) $\displaystyle\eta(y)=\left[\begin{array}[]{cccccc}Qx+D+E^{T}v+A^{T}u\\ f-Ex\\ \phi_{\rm FB}^{\varepsilon}\left(u,b-Ax\right)\\ \end{array}\right]=0,$ (28) $\displaystyle\phi_{\rm FB}^{\varepsilon}(a,b)=(a+b)-\sqrt{a^{2}+b^{2}+% \varepsilon},∼{}∼{}\varepsilon\rightarrow 0_{+}.$ (29)

It is investigated that the offered model is Lyapunov stable and asymptotically stable, but this model has more neurons with high computational complexity and requires stronger convergence conditions. Moreover, this model can not solve LP problems since in QP problem, the objective is assumed to be strictly convex rather than LP which objective is only convex.

Xia in [58] proposed a neural network to solve the problem

$\displaystyle\text{min}∼{}∼{}∼{}∼{}\frac{1}{2}x^{T}A_{0}x+a^{T}x$ (30)

subject to

$\displaystyle x\in\Omega_{0},$ (31) $\displaystyle Dx=b,$ (32)

$\displaystyle\frac{dx}{dt}=(I+A_{0})[P_{0}(x-A_{0}x+D^{T}y-a)-x]-D^{T}(Dx-b),$ (33) $\displaystyle\frac{dy}{dt}=-DP_{0}(x-A_{0}x+D^{T}y-a)+b,$ (34)

where $A_{0_{m\times m}}$ is a symmetric positive semidefinite matrix, $D$ in an ${n\times m}$ matrix and $\Omega_{0}\subset\mathbb{R}^{m}$ is a closed convex set. This network requires too many expensive analog multipliers and it is not economical to be implemented for large scale optimization problems. In order to overcome this problem, Tao et al. in [57] proposed a modified model as

$\displaystyle\frac{dx}{dt}=[P_{0}(x-A_{0}x+D^{T}y-a)-x]-D^{T}(Dx-b),$ (35) $\displaystyle\frac{dy}{dt}=-DP_{0}(x-A_{0}x+D^{T}y-a)+b.$ (36)

The network offered by Tao et.al. has not been proved to be convergent in finite time. If we use the network in [61] to solve problem Eqs (30)–(32), the dimension of the network is higher. Furthermore, it has not been proved to be convergent in finite time, too. Afterwards, to overcome the finite time convergence and exponential convergence, Xia et al. [62] exhibited a recurrent neural network for solving the strict convex quadratic optimization. The studied neural network is shown to have a finite time convergence and exponential convergence to the unique optimal solution of the strict convex quadratic optimization with general linear constraints. But it cannot be used to solve the DCQP Eqs (11)–(13) in which $Q\in\mathbb{R}^{n\times n}$ is only a symmetric and positive semidefinite matrix. Thus, this model can not solve LP problems.

Recently in [63], the results obtained in [62] for strictly convex quadratic programming have been improved considerably. Xue and Bian in [63] considered the following quadratic optimization problem:

$\displaystyle\text{min}∼{}∼{}∼{}∼{}\frac{1}{2}x^{T}Qx+c^{T}x$ (37)

subject to

$\displaystyle b^{1}\leqslant Bx\leqslant b^{2},$ (38) $\displaystyle d^{0}\leqslant x\leqslant h^{0},$ (39)

where $Q\in\mathbb{R}^{n\times n}$ is a symmetric and positive semidefinite matrix, $B\in\mathbb{R}^{m\times n},$ $b^{1},b^{2}\in\mathbb{R}^{m}$ , $d^{0},h^{0}\in\mathbb{R}^{n}$ and $c\in\mathbb{R}^{n}.$ Xue and Bian in [63] developed a project neural network for solving Eqs (37)–(39), which is shown to have complete convergence and finite-time convergence. However, the proposed neural network model has more variables and neurons, which makes circuit realization more difficult. In order to reduce the complexity of the suggested network in [63] to solve Eqs (37)–(39), Xu in [64] present a new network to solve Eqs (37)–(39), whose author claims the domain is wider than that of the other papers. However, with a deep focus we see that the structure of the projection operator defined for solving DCQP problem Eqs (37)–(39) is complex and thus the offered model is with high computational complexity.

Forti et al. in [31] considered trajectory convergence for a class of neural networks aimed at solving linear and convex quadratic programming problems. The main result in Forti et al. is that each trajectory of the studied network is convergent in finite time toward a singleton belonging to the set of constrained critical points. However, the quadratic problems in Forti et al. are only with affine constraints and the feasible region of them is a bounded closed convex polyhedron with non-empty interior.

5. Stability and convergence properties

In this section, we study some stability and convergence analysis for Eqs (8) and (9).

.

Let $y^{*}=(\theta{{{}^{*}}^{T}},\nu{{{}^{*}}^{T}},\vartheta{{{}^{*}}^{T}})^{T}$ be the equilibrium point of the neural network Eqs (8) and (9). Then $\theta^{*}$ is a KKT point of the problem Eqs (5) and (6). On the other hand, if $\theta^{*}\in\mathbb{R}^{N}$ is an optimal solution of problem Eqs (5) and (6), then there exist $\nu^{*}\in\mathbb{R}^{2N}$ and $\vartheta^{*}\in\mathbb{R}$ such that $y^{*}=(\theta{{{}^{*}}^{T}},\nu{{{}^{*}}^{T}},\vartheta{{{}^{*}}^{T}})^{T}$ is an equilibrium point of the network Eqs (8) and (9).

Proof..

Assume $y^{*}$ be the equilibrium of Eqs (8) and (9). Then $\frac{d\theta^{*}}{dt}=0,$ $\frac{d\nu^{*}}{dt}=0$ and $\frac{d\vartheta^{*}}{dt}=0.$ It follows easily that

$\displaystyle Q\theta^{*}+d+A^{T}(\nu^{*}+A\theta^{*}-b)^{+}+e\vartheta^{*}=0,$ (40) $\displaystyle(\nu^{*}+A\theta^{*}-b)^{+}-\nu^{*}=0,$ (41) $\displaystyle e^{T}\theta^{*}=0.$ (42)

From Lemma 1, $(\nu^{*}+A\theta^{*}-b)^{+}=\nu^{*}$ if and only if

$\displaystyle\nu^{*}\geqslant 0,\ \ A\theta^{*}-b\leqslant 0,\ \ {\nu^{*}}^{T}% (A\theta^{*}-b)=0.$ (43)

Substituting Eqs (41) into (40) we have

$\displaystyle Q\theta^{*}+d+A^{T}\nu^{*}+e\vartheta^{*}=0.$ (44)

From Eqs (42)–(44), it is seen that $y^{*}=(\theta{{{}^{*}}^{T}},\nu{{{}^{*}}^{T}},\vartheta{{{}^{*}}^{T}})^{T}$ satisfies the KKT conditions Eq. (7).

The converse is straightforward. ∎

.

The Jacobian matrix $\nabla\Psi(y)$ of the mapping $\Psi$ defined in Eq. (10) is a negative semidefinite matrix.

Proof..

Without loss of generality, assume that there exists $0<p<2N$ such that

$(\nu+A\theta-b)^{+}=({(\nu+A\theta-b)}_{1},{(\nu+A\theta-b)}_{2},...,{(\nu+A% \theta-b)}_{p},\underbrace{0,0,...,0}_{2N-p})^{T}.$

With a simple calculation, it is clearly shown that

$\displaystyle\nabla\Psi(y)=\left(\begin{array}[]{cccccc}-(Q+{(A^{p})}^{T}A^{p}% )&-{(A^{p})}^{T}&-{e}&\\ A^{p}&{{\cal W}_{2N\times 2N}}&{O_{2N\times 1}}\\ e^{T}&{O_{1\times 2N}}&{0}\end{array}\right),$

where $O$ indicates a zero matrix,

$\displaystyle A^{p}=\left(\begin{array}[]{ccccccc}U_{p\times N}\\ O_{(2N-p)\times N}\\ \end{array}\right)=\left(\begin{array}[]{ccccccc}A_{1.}\\ A_{2.}\\ ...\\ ...\\ A_{p.}\\ O_{1\times N}\\ O_{1\times N}\\ ...\\ ...\\ O_{1\times N}\end{array}\right),$

and

$\displaystyle{\cal W}_{2N\times 2N}=\left(\begin{array}[]{ccc}O_{p\times p}&O_% {p\times(2N-p)}\\ O_{(2N-p)\times p}&-I_{(2N-p)\times(2N-p)}\end{array}\right).$

From [69] we see that $(A^{p})^{T}A^{p}$ is a positive semidefinite matrix. Matrix $Q$ is also assumed to be positive semidefinite. Moreover, it is clear that matrix ${\cal W}_{2N\times 2N}$ is negative semidefinite. From the stated observations, we can derive that the Jacobian matrix $\nabla\Psi(y)$ is a negative semidefinite matrix.

If $p=2N,$ i.e. $(\nu+A\theta-b)^{+}=({(\nu+A\theta-b)}_{1},{(\nu+A\theta-b)}_{2},\ldots,{(\nu+% A\theta-b)}_{2N})^{T},$ then

$\displaystyle\nabla\Psi(y)=\left(\begin{array}[]{cccccc}-(Q+A^{T}A)&-A^{T}&-e&% \\ A&{O_{2N\times 2N}}&{O_{2N\times 1}}\\ e^{T}&{O_{1\times 2N}}&{0}\end{array}\right).$

Similar to the previous case, it is easily proved that $\nabla\Psi(y)$ is a negative semidefinite matrix.

Finally, if $p=0,$ i.e. $(\nu+A\theta-b)^{+}=(\underbrace{0,0,\ldots,0}_{2N})^{T},$ then we obtain

$\displaystyle\nabla\Psi(y)=\left(\begin{array}[]{cccccc}-Q&{O_{N\times 2N}}&-e% &\\ {O_{2N\times N}}&-I_{2N\times 2N}&{O_{2N\times 1}}\\ e^{T}&{O_{1\times 2N}}&{0}\end{array}\right).$

In this case also it is easy to verify that $\nabla\Psi(y)$ is a negative semidefinite matrix. This completes the proof. ∎

.

The function $\|(\nu+A\theta-b)^{+}\|^{2}$ of $(\nu+A\theta-b)^{+}$ defined in Eq. (10) is convex and continuously differentiable on $\mathbb{R}^{N}\times\mathbb{R}^{2N}.$

Proof..

Obviously, $\|(\nu+A\theta-b)^{+}\|^{2}=\sum_{k=1}^{2N}\left([(\nu+A\theta-b)_{k}]^{+}% \right)^{2}$ and for $k=1,\ldots,2N,$

$\displaystyle\left([(\nu+A\theta-b)_{k}]^{+}\right)^{2}=\left\{\begin{array}[]% {ll}\left((\nu+A\theta-b)_{k}\right)^{2}&\mbox{if $(\nu+A\theta-b)_{k}% \geqslant 0$}\\ 0&\text{otherwise.}\end{array}\right.$

Then, according to [75], the result is obtained from the differentiable convexity of $(A\theta-b)_{k}∼{}(k=1,\ldots,2N)$ . We also have

$\displaystyle\nabla\left(\|(\nu+A\theta-b)^{+}\|^{2}\right)=\left[\begin{array% }[]{cccccc}2A^{T}\left(\nu+A\theta-b\right)^{+}\\ 2\left(\nu+A\theta-b\right)^{+}\end{array}\right].$

This completes the proof. ∎

We now establish our main results as follows.

.

The neural network model in Eqs (8) and (9) is stable in the sense of Lyapunov.

Proof..

Consider the Lyapunov function $E:\mathbb{R}^{3N+1}\rightarrow\mathbb{R}$ as follows

$\displaystyle E(y)=E_{1}(y)+E_{2}(y),$ (45)

where $E_{1}(y)=\|\Psi(y)\|^{2}$ and $E_{2}(y)=\frac{1}{2}\|y-y^{*}\|^{2}.$ From Lemma 3, we know that $E_{1}(y)$ is a differentiable function. From Eq. (10), it is seen that

$\displaystyle\frac{d\Psi}{dt}=\frac{\partial\Psi}{\partial y}\frac{dy}{dt}=% \nabla\Psi(y)\Psi(y).$

Calculating the derivative of $E(t)$ along the solution $y(t)$ of the neural network Eqs (8) and (9), we have

$\displaystyle\frac{dE(y(t))}{dt}=\left(\frac{d\Psi}{dt}\right)^{T}\Psi+\Psi^{T% }\left(\frac{d\Psi}{dt}\right)+(y-y^{*})^{T}\frac{dy(t)}{dt}=\Psi^{T}(\nabla% \Psi(y)^{T}+\nabla\Psi(y))\Psi+(y-y^{*})^{T}\Psi(y).$ (46)

Employing Lemma 2, we attain

$\displaystyle\Psi^{T}(y)(\nabla\Psi(y)^{T}+\nabla\Psi(y))\Psi(y)\leqslant 0,∼{% }∼{}\forall y\neq y^{*}.$ (47)

Moreover, from Definition 2.2 and Lemma 2.3 in [44], we have

$\displaystyle(y-y^{*})^{T}(\Psi(y)-\Psi(y^{*}))=(y-y^{*})^{T}\Psi(y)\leqslant 0% ,∼{}∼{}\forall y\neq y^{*}.$

Thus

$\displaystyle\frac{dE(y(t))}{dt}\leqslant 0.$ (48)

This means that the neural network Eqs (8) and (9) is stable in the sense of Lyapunov. ∎

.

(i) For any initial point $y(t_{0})=(\theta(t_{0})^{T},\nu(t_{0})^{T},\vartheta(t_{0})^{T})^{T},$ there exists a unique continuous solution $y(t)=(\theta(t)^{T},\nu(t)^{T},\vartheta(t)^{T})^{T}$ for system Eqs (8) and (9).

(ii) Let $y(t)=(\theta(t)^{T},\nu(t)^{T},\vartheta(t)^{T})^{T}$ be the state trajectory of Eqs (8) and (9) with the initial point $y(t_{0})=(\theta(t_{0})^{T},\nu(t_{0})^{T},\vartheta(t_{0})^{T})^{T}.$ If $\nu(t_{0})\geqslant 0,$ then $\nu(t)\geqslant 0.$

Proof..

(i) It is easy to verify that $Q\theta+d+A^{T}(\nu+A\theta-b)^{+}+e\vartheta,$ $(\nu+A\theta-b)^{+}-\nu$ and $e^{T}\theta$ are locally Lipschitz continuous. According to the local existence of ordinary differential equations [70], neural network Eqs (8) and (9) has a unique continuous solution $y(t),t\in[t_{0},\eta)$ for some $\eta>t_{0}.$

By the proof of Theorem 3, we know that $E$ is a non-increasing function with respect to $t,$ so

$\displaystyle\frac{1}{2}\|y-y^{*}\|^{2}\leqslant E(y(t))\leqslant E(y(t_{0})),% ∼{}∼{}\forall t\geqslant t_{0}.$ (49)

This shows that the state trajectory of the neural network Eqs (8) and (9) is bounded. Thus $\eta\rightarrow+\infty.$

(ii) For the given initial point $y(t_{0})$ with $\nu(t_{0})\geqslant 0$ we have

$\displaystyle\frac{d\nu}{dt}+\nu=(\nu+A\theta-b)^{+},$ $\displaystyle\int_{t_{0}}^{t}\left(\frac{d\nu}{dt}+\nu\right)e^{s}ds=\int_{t_{% 0}}^{t}e^{s}(\nu+A\theta-b)^{+}ds.$

It follows

$\displaystyle\nu(t)=e^{-(t-t_{0})}\nu(t_{0})+e^{-t}\int_{t_{0}}^{t}e^{s}(\nu+A% \theta-b)^{+}ds.$

Since $(\nu+A\theta-b)^{+}\geqslant 0,$ $\nu(t)\geqslant 0$ for any $t\geqslant t_{0}.$ ∎

.

The state trajectory of the neural network Eqs (8) and (9) converges to an equilibrium point for any initial point $y_{0}=(\theta(t_{0})^{T},\nu(t_{0})^{T},\vartheta(t_{0})^{T})^{T}\in\mathbb{R}% ^{3N+1}.$ In particular, neural network Eqs (8) and (9) with any initial point $y_{0}=(\theta(t_{0})^{T},\nu(t_{0})^{T},\vartheta(t_{0})^{T})^{T}\in\mathbb{R}% ^{3N+1}$ is globally asymptotically stable when $D^{*}$ has unique equilibrium point, where $D^{*}$ is the optimal point set of Eqs (5) and (6) and its dual.

Proof..

From the proof of Lemma 4, we have the state trajectory $(\theta(t)^{T},\nu(t)^{T},\vartheta(t)^{T})^{T}$ of neural network Eqs (8) and (9) is bounded. Therefore, there exists an increasing sequence $\{t_{k}\}$ with $t_{k}\rightarrow\infty∼{}{\rm as}∼{}k\rightarrow\infty,$ and a limit point $(\bar{\theta}^{T},\bar{\nu}^{T},\bar{\vartheta}^{T})^{T},$ such that

$\lim_{k\rightarrow\infty}(\theta(t_{k})^{T},\nu(t_{k})^{T},\vartheta(t_{k})^{T% })^{T}=(\bar{\theta}^{T},\bar{\nu}^{T},\bar{\vartheta}^{T})^{T}.$

Using the LaSalle invariant set theorem [71], one has that $\{(\theta(t)^{T},\nu(t)^{T},\vartheta(t)^{T})^{T}\rightarrow M\}$ as $t\rightarrow\infty,$ where $M$ is the largest invariant set in $K=\{(\theta(t)^{T},\nu(t)^{T},\vartheta(t)^{T})^{T}|\frac{dE(y(t))}{dt}=0\}.$ From Eqs (8), (9) and (48), it follows that $\frac{d\theta}{dt}=0,$ $\frac{d\nu}{dt}=0$ and $\frac{d\vartheta}{dt}=0\Leftrightarrow\frac{dE(y(t))}{dt}=0.$ Thus $(\bar{\theta}^{T},\bar{\nu}^{T},\bar{\vartheta}^{T})^{T}\in D^{*}$ by $M\subseteq K\subseteq D^{*}$ .

Secondly, we prove the state trajectory $(\theta(t)^{T},\nu(t)^{T},\vartheta(t)^{T})^{T}$ globally converges to the equilibrium point $(\bar{\theta}^{T},\bar{\nu}^{T},\bar{\vartheta}^{T})^{T}$ . Define another Lyapunov function

$\displaystyle\bar{E}(y)=\|\Psi(y)\|^{2}+\frac{1}{2}\|y-\bar{y}\|^{2},$ (50)

where we substitute $\theta^{*}=\bar{\theta},$ $\nu^{*}=\bar{\nu}$ and $\vartheta^{*}=\bar{\vartheta}$ in Eq. (45). Then $\bar{E}(y)$ is continuously differentiable and $\bar{E}(\bar{y})=0.$ Noting that

$\lim_{k\rightarrow\infty}(\theta(t_{k})^{T},\nu(t_{k})^{T},\vartheta(t_{k})^{T% })^{T}=(\bar{\theta}^{T},\bar{\nu}^{T},\bar{\vartheta}^{T})^{T}.$

We therefore have $\lim_{k\rightarrow\infty}\bar{E}(\theta(t_{k})^{T},\nu(t_{k})^{T},\vartheta(t_% {k})^{T})^{T}=\bar{E}(\bar{\theta}^{T},\bar{\nu}^{T},\bar{\vartheta}^{T})^{T}.$ So, $\forall\epsilon>0$ there exists $q>0$ such that for all $t\geqslant t_{q},$ we have $\bar{E}(y(t))<\epsilon.$ Similarly, we can obtain $\frac{d\bar{E}(y(t))}{dt}\leqslant 0.$ It follows that for $t\geqslant t_{q},$

$\frac{1}{2}\|y(t)-\bar{y}\|^{2}\leqslant\bar{E}(y(t))\leqslant\epsilon.$

It follows that $\lim_{t\rightarrow\infty}\|y(t)-\bar{y}\|=0$ and $\lim_{t\rightarrow\infty}y(t)=\bar{y}.$ Therefore, the suggested framework Eqs (8) and (9) is globally convergent to an equilibrium point $\bar{y}=(\bar{\theta}^{T},\bar{\nu}^{T},\bar{\vartheta}^{T})^{T},$ where $\bar{\theta}$ is the optimal solution of Eqs (5) and (6).

In particular, if $D^{*}=\{(\theta{{{}^{*}}^{T}},\nu{{{}^{*}}^{T}},\vartheta{{{}^{*}}^{T}})^{T}\}$ , then the neural network Eqs (8) and (9) for solving Eqs (5) and (6) is globally asymptotically stable to the unique equilibrium point $y^{*}=(\theta{{{}^{*}}^{T}},\nu{{{}^{*}}^{T}},\vartheta{{{}^{*}}^{T}})^{T}$ , where $D^{*}$ is the optimal point set of Eqs (5) and (6) and its dual. ∎

.

The convergence rate of the neural network in Eqs (8) and (9) increases as $\tau$ increases.

Proof..

From Eqs (46)–(48), it can be seen that for any $\tau>0$ in Eqs (8) and (9)

$\displaystyle\frac{dE(y)}{dt}\leqslant\tau\Psi(y)^{T}\left(\nabla\Psi(y)^{T}+% \nabla\Psi(y)\right)\Psi(y)\leqslant 0.$

Then

$\displaystyle E(y(t))\leqslant E(y(t_{0}))+\tau\int_{t_{0}}^{t}\Psi(y(s))^{T}% \left(\nabla\Psi(y(s))^{T}+\nabla\Psi(y(s))\right)\Psi(y(s))ds.$

Since $E(y)\geqslant\frac{1}{2}\|y-y^{*}\|^{2},$ where $y^{*}$ is a KKT point of Eqs (5) and (6), we get

$\displaystyle\|y(t)-y^{*}\|^{2}\leqslant 2E(y(t_{0}))+2\tau\int_{t_{0}}^{t}% \Psi(y(s))^{T}\left(\nabla\Psi(y(s))^{T}+\nabla\Psi(y(s))\right)\Psi(y(s))ds.$

Therefore, the convergence rate of the trajectory $y(t)$ increases as $\tau$ increases. ∎

6. Numerical simulations

In order to demonstrate the effectiveness of the proposed neural network, in this section we test two examples. The numerical implementation is coded by Matlab 7.0 and the ordinary differential equation solver adopted here is ode23, which uses the Ruge-Kutta (2;3) formula. As mentioned earlier, the parameter $\tau$ is set to be 1.

Example 1: We consider the regression data in Table 1. We use the offered model Eqs (8) and (9) with an RBF kernel to train an SVM for the regression problem. We choose the following Gaussian function

$\displaystyle K(x,y)=\exp\left({-\frac{\|x-y\|^{2}}{2\delta^{2}}}\right).$

Figures 2 and 3 show the convergence behavior of $\theta(t)$ based on the proposed model Eqs (8) and (9) with the initial point $\theta_{0}=(1,-2,3,-4,5,-6,7,-8,9)^{T}$ . Figure 4 shows the SVR results with $\zeta=10,$ $\delta=1$ and three different $\epsilon$ parameters.

Table 1
Regression data

$x$	$-$ 5	$-$ 3	$-$ 2	1.6	3.8	10.2	11	11.5	12.7
$y$	$-$ 1.6	1.8	$-$ 1	$-$ 1.2	2.2	$-$ 6.8	9	10	10

Figure 2.

Transient behaviors $\theta_{1}(.),\ldots,\theta_{5}(.)$ of the neural network Eqs (8) and (9) with the initial point $\theta_{0}=(1,-2,3,$ $-4,5,-6,7,-8,9)^{T}$ in Example 1.

Figure 3.

Transient behaviors $\theta_{6}(.),\ldots,\theta_{9}(.)$ of the neural network Eqs (8) and (9) with the initial point $\theta_{0}=(1,-2,3,-4,$ $5,-6,7,-8,9)^{T}$ in Example 1.

Figure 4.

Results of support vector regression using the neural network Eqs (8) and (9) with an RBF kernel where $\zeta=10,$ $\delta=1$ and three different $\epsilon$ parameters in Example 1.

Figure 5.

Results of support vector regression using the neural network Eqs (8) and (9) with an RBF kernel where $\zeta=10,$ $\delta=6$ and three different $\epsilon$ parameters in Example 2.

Example 2: In this example, we show a regression result based on titanium regression data [72] for the SVR learning by using the presented network. The kernel of the SVM is selected similar to Example 1. Figure 5 shows the SVR result with $\zeta=10$ , $\delta=6$ and three different $\epsilon$ parameters. It is included that a small $\epsilon$ leads to a better regression. Figure 6 also shows the regression results with $\delta=6,$ $\epsilon=0.1$ and three different $\zeta$ parameters. It indicates that a larger $\zeta$ is better than small one. Consequently, for real applications, we need set these parameters properly.

Figure 6.

Results of support vector regression using the neural network Eqs (8) and (9) with an RBF kernel where $\delta=6$ and $\epsilon=0.1$ and three different $\zeta$ parameters in Example 2.

Finally, a natural question arises: what are the practical and computational advantages of the proposed neural network, compared to existing generally available algorithms for optimization problems? To answer this, we summarize what we have observed from numerical experiments and theoretical results as follows:

Compared with traditional numerical optimization algorithms, the neural network approach has several potential advantages in real-time applications [59]. First, the structure of a neural network can be implemented effectively using very large scale integration and optical technologies. Second, neural networks can solve many optimization problems with time-varying parameters. Third, the dynamical techniques and the numerical ODE techniques can be applied directly to the continuous-time neural network for solving constrained optimization problems effectively. Fourth, the neural networks have fast convergence rate in real-time solutions. Therefore, neural network methods for optimization have been received considerable attention

We can compare our neural network in Eqs (8) and (9) with some existing models which also work for quadratic programming problems, for instance, the Friesz projection neural network in [32], the Kennedy and Chua’s model [3], the lagrangian network model in [44, 45], and the gradient–based neural network model [46]. At first glance, these neural network models look having lower complexity. However, we can observe that the difference of the numerical performance is very marginal by testing some quadratic optimization problems. In fact, the presented models in [32, 3, 44, 45, 46] can not solve convex quadratic programming problems where objective function is only convex, i.e. linear programming, degenerate convex quadratic programming problem. But the neural network Eqs (8) and (9) in this manuscript can solve all these optimization problems.

From Theorem 4, we see that the solution converges with any initial point. Thus, changing initial points may not have much effect for our neural network model. In fact, our model is globally convergent to the optimal solution of the problem.

In Lemma 5, we also analyze the influence of the parameter $\tau$ in the dynamic model Eqs (8) and (9) on the convergence rate of the trajectory and the convergence behavior of $\|y(t)-y^{*}\|^{2}$ and obtain that a larger $\tau$ leads to a better convergence rate.

The proposed model Eqs (8) and (9) only requires to convexity of the problem rather than strict convexity, which is in contrast to the models suggested in [53, 60].

The other advantages of the proposed neural network are that it can be implemented without a penalty parameter and can be convergent to an exact solution of the QP problem related to the SVR problem.

According to the the above discussions, proposing an efficient neural network for solving SVR problems with good stability properties and convergence results is very necessary and meaningful.

7. Conclusion

In this paper, an analog neural network architecture for SVM learning for regression is presented. A convergence analysis is given, which shows that the suggested model can be guaranteed to obtain the solution of SVR. The structure of the proposed network is reliable and efficient. Two simulation examples are elaborated to show the excellent performance of the proposed network for SVM in regression.

Footnotes

Acknowledgments

The author is very grateful to the editor and anonymous reviewers for their suggestions in improving the quality of the paper.

References

Ben-Yacoub

Abdeljaoued

and Mayoraz

, Fusion of face and speech data for person identity verification, IEEE Transactions on Neural Networks 10 (1999) 1065–1074.

Burges

, A tutorial on support vector machines for patter recognition, Data Mining and Knowledg Discovery 2 (1998), 1–47.

Cortes

and Vapnik

, Support-vector networks, Machine Learning 20 (1995), 273–297.

Chuang

C.C.

S.F.

Jeng

J.T.

and Hsiao

C.C.

, Robust support vector regression networks for function approximation with outliers, IEEE Transactions on Neural Networks 13 (2002), 1322–1330.

Fletcher

, Support Vector Machines Explained, UCL, 2009.

Gunn

S.R.

, Support Vector Machines for Classification and Regression, Univ. Southampton, Southhampton, U.K., Tech. Rep. May 1998. Image Speech and Intelligent Systems Research Group.

Kalouptisidis

, Signal Processing Systems, Theory and Design, New York: Wiley, 1997.

Keerthi

and DeCoste

, A modified finite Newton method for fast solution of large scale linear SVMs, Journal of Machine Learning Research 6(1) (2006), 341–361.

Lin

C.J.

, On the convergence of the decomposition methods for support vector machines, IEEE Transactions on Neural Networks 12 (2001), 1288–1298.

10.

Lin

C.J.

, A formal analysis of stopping criteria of decomposition methods for support vector machines, IEEE Transaction on Neural Networks 13 (2002), 1045–1052.

11.

Mangasarian

O.L.

and Musicant

D.R.

, Successive overrelaxation for support vector machines, IEEE Transactions on Neural Networks 10 (1999), 1032–1037.

12.

Mangasarian

, A finite Newton method for classification problems, Journal: Optimization Methods and Software 17 (2002), 913–929.

13.

Theiler

and Perkins

, Accurate on-line support vector regression, Neural Computation 15 (2003), 2683–2703.

14.

Martin

, On-line support vector machine regression, In Machine Learning: ECML 2002, Springer Berlin Heidelberg, 2002, pp. 282–294.

15.

Osuna

Freund

and Girosi

, Training support vector machines: An application to face detection, in Proc. Computer Vision Pattern Recognition, San Juan, Puerto Rico, June 1997.

16.

Platt

, Sequential minimal optimization: A fast algorithm for training support vector machines, in: Advances in Kernel Methods Support Vector Learning Scholkopf

Burges

C.J.C.

and Smola

A.J.

, eds, Cambridge, MA: MIT Press, 1999, pp. 185–208.

17.

Shevade

S.K.

Keerthi

S.S.

Bhattacharyya

and Murthy

K.R.K.

, Improvements to the SMO algorithm for SVM regression, IEEE Transactions on Neural Networks 11 (2000), 1188–1193.

18.

Suykens

J.A.K.

and Vandewalle

, Training multilayer perceptron classifiers based on a modified support vector method, IEEE Transactions on Neural Networks 10 (1999), 907–911.

19.

Scholkopf

Smola

Williamson

R.C.

and Bartlett

P.L.

, New support vector algorithms, Neural Computing 12 (2000), 1207–1245.

20.

Vapnik

, The Nature of Statistical Learning Theory, New York: Springer-Verlag, 1995.

21.

Vapnik

Golowich

and Smola

, Support vector method for function approximation, regression estimation and signal processing, in Advances in Neural Information Processing Systems. Cambridge, MA: MIT Press, 9, 1997.

22.

Harker

P.T.

and Pang

J.S.

, Finite-dimensional variational inequality and nonlinear complementarity problems: A survey of theory, algorithms, and applications, Mathematical Programming Series B 48 (1990), 161–220.

23.

B.S.

and Liao

L.-Z.

, Improvements of some projection methods for monotone nonlinear variational inequalities, Journal of Optimization Theory and Applications 112 (2002), 111–128.

24.

Solodov

M.V.

and Tseng

, Modified projection-type methods for monotone variational inequalities, SIAM Journal on Control and Optimization 2 (1996), 1814–1830.

25.

Konnov

I.V.

, A class of combined iterative methods for solving variational inequalities, Journal of Optimization Theory and Applications 94 (1997), 677–693.

26.

Cichocki

and Unbehauen

, Neural networks for optimization with bounded constraints, IEEE Transactions on Neural Networks 4 (1993), 293–304.

27.

Hopfield

J.J.

and Tank

D.W.

, Computing with neural circuits: a model, Science 233 (1986), 625–633.

28.

Tank

D.W.

and Hopfield

J.J.

, Simple neural optimization networks: An A/D converter, signal decision circuit, and a linear programming pircuit, IEEE Transactions on Circuits and Systems 33 (1986), 533–541.

29.

Anguita

and Boni

, Improved neural network for SVM learning, IEEE Transactions on Neural Networks 13 (2002), 1243–1244.

30.

Effati

and Nazemi

A.R.

, Neural network models and its application for solving linear and quadratic programming problems, Applied Mathematics and Computation 172 (2006), 305–331.

31.

Forti

Nistri

and Quincampoix

, Convergence of neural networks for programming problems via a nonsmooth Lojasiewicz inequality, IEEE Transactions on Neural Networks 17 (2006), 1471–1486.

32.

Friesz

T.L.

Bernstein

D.H.

Mehta

N.J.

Tobin

R.L.

and Ganjlizadeh

, Day-to-day dynamic network disequilibria and idealized traveler information systems, Operation Research 42 (1994), 1120–1136.

33.

Gao and

Liao

, A new projection-based neural network for constrained variational inequalities, IEEE Transactions on Neural Networks 20 (2009), 373–388.

34.

, Applications of the general projection neural network in solving extended linear-quadratic programming problems with linear constraints, Neurocomputing 72 (2009), 1131–1137.

35.

and Wang

, Design of general projection neural networks for solving monotone linear variational inequalities and linear and quadratic optimization problems, IEEE Transactions on Systems, Man, and Cybernetics, Part B 37 (2007), 1414–1421.

36.

Sun

and Zhang

, Design of recurrent neural networks for solving constrained least absolute deviation problems, IEEE Transactions on Neural Networks 21 (2010), 1073–1086.

37.

and Zhang

, An alternative recurrent neural network for solving variational inequalities and related optimization problems, IEEE Transactions on System Man Cybernetics, B 39 (2009), 1640–1645.

38.

Kennedy

M.P.

and Chua

L.O.

, Neural networks for nonlinear programming, IEEE Transactions on Circuits and Systems 35 (1988), 554–562.

39.

Lillo

W.E.

Loh

M.H.

Hui

and Zăk

S.H.

, On solving constrained optimization problems with neural networks: A penalty method approach, IEEE Transactions on Neural Networks 4 (1993), 931–939.

40.

Liu

Q.S.

and Wang

, A one-layer recurrent neural network with a discontinuous hard-limiting activation function for quadratic programming, IEEE Transactions on Neural Networks 19 (2008), 558–570.

41.

Liu

and Zhao

, A continuous-time recurrent neural network for real-time support vector regression, Computational Intelligence in Control and Automation (CICA), 2013 IEEE Symposium on, pp 189–193.

42.

Maa

C.Y.

and Shanblatt

M.A.

, Linear and quadratic programming neural network analysis, IEEE Transactions on Neural Networks 3 (1992), 580–594.

43.

Malek

Hosseinipour-Mahani

and Ezazipour

, Efficient recurrent neural network model for the solution of general nonlinear optimization problems, Optimization Methods and Software 25 (2010), 1–18.

44.

Nazemi

A.R.

, A dynamic system model for solving convex nonlinear optimization problems, Communications in Nonlinear Science and Numerical Simulation 17 (2012), 1696–1705.

45.

Nazemi

A.R.

, A neural network model for solving convex quadratic programming problems with some applications, Engineering Applications of Artificial Intelligence 32 (2014), 54–62.

46.

Nazemi

A.R.

and Nazemi

, A gradient-based neural network method for solving strictly convex quadratic programming problems, Cognitive Computation 6 (2014), 484–495.

47.

Nazemi

A.R.

, A dynamical model for solving degenerate quadratic minimax problems with constraints, Journal of Computational and Applied Mathematics 236 (2011), 1282–1295.

48.

Nazemi

A.R.

, Solving general convex nonlinear optimization problems by an efficient neurodynamic model, Engineering Applications of Artificial Intelligence 26 (2013), 685–696.

49.

Nazemi

A.R.

and Omidi

, A capable neural network model for solving the maximum flow problem, Journal of Computational and Applied Mathematics 236 (2012), 3498–3513.

50.

Nazemi

A.R.

and Omidi

, An efficient dynamic model for solving the shortest path problem, Transportation Research Part C 26 (2013), 1–19.

51.

Nazemi

A.R.

and Tahmasbi

, A computational intelligence method for solving a class of portfolio optimization problems, Soft Computing 18 (2014), 2101–2117.

52.

Nazemi

A.R.

Abbasi

and Omidi

, Solving portfolio selection models with uncertain returns using an artificial neural network scheme, Applied Intelligence (2014) (In Press).

53.

Perfetti

and Ricci

, An alog neural network for support vector machine learning, IEEE Transactions on Neural Networks 17 (2006), 1085–1091.

54.

Qin

Fan

and Liu

, A simplified recurrent neural network for pseudoconvex optimization subject to linear equality constraints, Communications in Nonlinear Science and Numerical Simulation 19 (2014), 789–798.

55.

Sun

Chen

J-S.

and Ko

C.-H.

, Neural networks for solving second-order cone constrained variational inequality problem, Computational Optimization and Applications 51 (2012), 623–648.

56.

Tan

, Xia

and Wang

, Neural network realization of support vector methods for pattern classification, in: Proceding IEEE International Joint Conference Neural Networks, 2000, pp. 411–416.

57.

Tao

Cao

and Sun

, A simple and high performance neural network for quadratic programming problems, Applied Mathematics and Computation 124 (2001) 251–260.

58.

Xia

, A new neural network for solving linear and quadratic programming problems, IEEE Transactions on Neural Networks 7 (1996), 1544–1547.

59.

Xia

and Wang

, A recurrent neural network for Solving nonlinear convex programs subject to linear constraints, IEEE Transactions on Neural Networks 16 (2005), 379–386.

60.

Xia

and Wang

, A one layer recurrent neural network for support vector machine learning, IEEE Transactions on Systems, Man Cybernetics – Part B 34 (2004) 1261–1269.

61.

Xia

Y.S.

and Wang

, A recurrent neural network for solving linear projection equations, Neural Networks 13 (2000), 337–350.

62.

Xia

Feng

and Wang

, A recurrent neural network with exponential convergence for solving convex quadratic program and related linear piecewise equation, Neural Networks 17 (2004), 1003–1015.

63.

Xue

and Bian

, A project neural network for solving degenerate convex quadratic program, Neurocomputing 70 (2007), 2449–2459.

64.

, Projection neural networks for solving constrained convex and degenerate quadratic problems (2010) Proceedings – 2010 IEEE International Conference on Intelligent Computing and Intelligent Systems, ICIS 2010, 3, art. no. 5658537, pp.91–96.

65.

Yang

and Hu

, A compact neural network for training support vector machines, Neurocomputing 86 (2012), 193–198.

66.

Yang

, Cao Xua

Hua

and Gao

, A new neural network for solving quadratic programming problemswith equality and inequality constraints, Mathematics and Computers in Simulation 101 (2014), 103–112.

67.

Zhao

and Liu

, Generalized recurrent neural network for ϵ-insensitive support vector regression, Mathematics and Computers in Simulation 86 (2012), 2–9.

68.

Ortega

T.M.

and Rheinboldt

W.C.

, Iterative solution of nonlinear equations in several variables, Academic Press, New York, 1970.

69.

Quarteroni

Sacco

and Saleri

, Numerical mathematics, Volume 37 of Texts in Applied Mathematics, Springer-Verlag, Berlin, second edition, 2007.

70.

Miller

R.K.

and Michel

A.N.

, Ordinary Differential Equations, Academic Press, NewYork, 1982.

71.

Hale

J.K.

, Ordinary Differential Equations, New York: Wiley-Interscience, 1969.

72.

Dierckx

, Curve and Surface Fitting With Splines. Oxford, U.K.: Clarendon, 1993.

73.

Bazaraa

M.S.

Sherali

H.D.

and Shetty

C.M.

, Nonlinear Programming-Theory and Algorithms, 2nd ed. New York: Wiley, 1993.

74.

Boyd

and Vandenberghe

, Convex Optimization, Cambridge University Press, Cambridge, 2004.

75.

Avriel

, Nonlinear Programming: Analysis and Methods, Englewood Cliffs, NJ: Prentice-Hall, 1976.

An application of a practical neural network model for solving support vector regression problems

Abstract

Keywords

1. Introduction

2. Support vector regression problem

.

.

Proof..

.

Proof..

.

Proof..

.

Proof..

.

Proof..

.

Proof..

.

Proof..

.

Proof..

Table 1 Regression data

Footnotes

Acknowledgments

References

Table 1
Regression data