Critical dynamical analysis for α-UAM RNNs without diagonal nonlinear requirements

Abstract

Critical dynamics research of recurrent neural networks (RNNs) is very meaningful in both theoretical importance and practical significance. Recently, because of the application requirements, the study on the critical dynamics behaviors of RNNs has drawn special attention. The critical condition is that a discriminant matrix M₁ (Γ) related with an RNN is nonnegative definite. Due to the essential difficulty in analysis, there were only a few critical results up to now. Further, nearly all of the existing dynamic results are with diagonally nonlinear requirements on the activation mappings, i.e., the activation mapping G should satisfy the strict necessary condition that G (x) = (g₁ (x₁) , g₂ (x₂) , ⋯ , g_N (x_N)) ^T. This is because of the essential difficulty on the analysis of the energy function. The requirement is so strict and it limits the applications of RNNs. In this paper, under the critical conditions, some new global asymptotically stable conclusions are presented for RNNs without the diagonally nonlinear requirement on the activation mappings. The results present here not only improve substantially upon the existing relevant critical stability results, but also provide some further cognizance on the essentially dynamical behavior of RNNs, and further, enlarge the application fields of them.

Keywords

Without diagonal nonlinear critical dynamics analysis global asymptotically stable recurrent neural networks

1 Introduction

Recurrent neural networks are the neural networks with feedback loops and whose neurons send feedback signals to each other. An RNN is one kind of dynamic systems that their states vary with time pasting by. Their internal memory can be used to process arbitrary sequences of inputs, and this makes them applicable to model dynamic process associated with solving learning, pattern recognition, image processing, associative memory as well as solving optimization problems. The crucial foundation of the RNNs consists in their dynamical properties, such as the global convergence, asymptotic stability and exponential stability, therefore, the analysis of such dynamical behaviors is a first and necessary step for any practical design and application of RNNs.

In recent years, considerable efforts have been devoted to the neural network modeling and control, and for different model individuals, there have been numerous analysis on the stability of RNNs with or without time-delay (see, e.g. [1 –6] and the references therein). Further, the stability results of RNNs can be used for different analysis and applications, e.g., the adaptive neural networks are proposed for a class of non-linear second-order multi-agent systems and greatly reduces the online computation burden, or for stabilizing the uncertain nonlinear strict-feedback systems with full-state constraints [7]. A fuzzy-neural network is used to approximate the unknown functions of a class of nonlinear stochastic systems [8]. The Radial basis function neural networks are utilized to approximate the unknown nonlinear function [9].

For RNNs study, two fundamental modeling approaches are commonly adopted: either using the neuron states or using the local field states of neurons as basic variables to describe the dynamical evolution rules of the neural network. Correspondingly, the local field RNNs and static RNNs typically represent two fundamental modeling approaches in current neural network research [10], which are respectively modeled by $τ \frac{du}{dt} = - Du + WG (u) + q, u (0) = u_{0} \in R^{N}$ (1) and $τ \frac{dx}{dt} = - Dx + G (Wx + b), x (0) = x_{0} \in R^{N}$ (2) where u = (u₁, u₂, ⋯ , u_N) ^T is the local field vector, x = (x₁, x₂, ⋯ , x_N) ^T is the neural state vector, D = diag {d₁, d₂, ⋯ d_N} is a positive matrix with each d_i being the state feedback coefficients, W = (ω_ij) _N×N is the synaptic weight matrix, τ is a positive constant, q and b are fixed external bias vectors and G : R^N → R^N is the nonlinear activation mapping.

Model (1)–(2) summarize most of the existing continuous-time RNNs specials, e.g., Hopfield-type neural networks, brain-state-in-a-box neural networks, bound-constraints optimization solvers, recurrent back-propagation neural networks, mean-field neural networks,convex optimization solvers, recurrent correlation associative memories neural networks, cellular neural networks, etc.

For a given recurrent neural network, if we define M₁ (Γ) = L^-1DΓ - (ΓW + W^TΓ)/2, where both D and Γ are positive definite diagonal matrices, and W is the weight matrix of the network, then by generalizing these existing stability results of RNNs, it should be noticed that most of them are on the exponential stability analysis under the conditions. That is, there exists a positive definite diagonal matrix Γ, such that M₁ (Γ) is positive definite, where L = diag {L (g₁) , L (g₂) , …, L (g_N)} with each L (g_i) >0 being the Lipschitz constant of g_i and G = (g₁, g₂, …, g_N) ^T is the activation mapping of the network. On the other hand, [11, 12] have proved that an RNN will be globally exponentially unstable if there is a positive definite diagonal matrix Γ, such that M₂ (Γ) = l^-1DΓ - (ΓW + W^TΓ)/2 is negative definite, where l = diag {l (g₁) , l (g₂) , …, l (g_N)} with each l (g_i) >0 being the inversely Lipschitz constant of g_i, i.e., |g_i (t) - g_i (s) | ≥ l (g_i) |t - s| for all s, t ∈ R^N. By the definitions of Lipschitz constant and inversely Lipschitz constant, we have l (g_i) ≤ L (g_i) and, in the sense of nonnegative definition, the inequality relation M₁ (Γ) ≤ M₂ (Γ) holds. From what have been mentioned above, we get that M₁ (Γ) >0 is sufficient for the globally exponential stability of RNNs, and M₂ (Γ) ≥0 is necessary for RNNs to have globally stable dynamics. The question then arise: what kinds of dynamic behaviors will be when M₁ (Γ) ≤0 and M₂ (Γ) ≥0? Since M₁ (Γ) >0 is a sufficient condition that the network is stable, and when M₁ (Γ) ≥0, one can get M₂ (Γ) ≥0 directly, then recently special attention has been paid on the dynamics behavior under the condition that M₁ (Γ) ≥0. This condition is called as the critical condition, and the dynamics analysis under such condition is called as the critical analysis. It is clear that the critical condition is really an essential gap between stable and unstable for RNNs. The goal of the critical analysis is to find the least restrictions of the networks to assure the stability. To extend the application fields, especially to loosen the design for the structure of RNNs, it is quite important to study the critical dynamics of RNNs. While, it is by no means easy to conduct a meaningful critical dynamics analysis, since such analysis is much more difficult than the dynamics analysis under the non-critical condition, i.e. M₁ (Γ) >0. It should be also noticed that the study of dynamics behaviors for RNNs on the critical case is very valuable in both theory and application, that is because, for RNNs, the critical condition can characterize the essential boundary line between the stability region and instability region [14].

Up to now, there are only a few critical stability and convergence analysis on RNNs. For RNN with hyperbolic tangent activation function, in [15], the globally asymptotical stability and globally exponential stability of the unique equilibrium point of the network under some specific conditions of M₁ (Γ) ≥0 have been conducted. The authors of [16] have gotten the globally exponential stability of RNN with projection operator under the condition that I - W is nonnegative definite (which is a special case of M₁ (Γ) ≥0). In [12], the authors have proved that an RNN with Sigmoidal activation mapping has a globally attractive equilibrium state, and when W is quasi-symmetric (i.e., there exists a positive definite diagonal matrix Γ, such that ΓW is symmetric), then RNN with nearest point projection activation mapping is global convergence on a region defined by the network. The quasi-symmetric requirement of W in [12] has been removed in [17, 18]. Further, the authors of [19] have gotten results that an RNN with general projection mapping is globally convergent under the condition that M₁ (Γ) + P ≥ 0 (here M₁ (Γ) <0 and P ≥ 0) if the nonlinear norm defined by the network is bounded, and some further study of such RNNs has been conducted in [13 , 20–23].

On the other hand, it should be pointed out that almost all of the dynamics conclusions in literature are based on a very strong hypothesis that activation mappings must be diagonally nonlinear. The diagonally nonlinear requirement in dynamics analysis is due to the fact that when considering the derivative of a constructed energy function, an inner product is always produced and it is hard to deal with it if one do not use the diagonally nonlinear property. The diagonally nonlinear requirement has great limitation in application, since actually, each g_i should have responds to all x_j, not only just to the corresponding x_i. Obviously, the requirement for activation mappings is quite strict, which does not obey the biological and applications.

In this present, we devote to answer the question that what kinds of critical dynamics behaviors will be for RNNs without the diagonally nonlinear requirement. That is, we consider the activation mapping G has its original form G (x) = (g₁ (x) , g₂ (x) , ⋯ , g_N (x)) ^T and networks are under the critical condition that M₁ (Γ) ≥0. Since many commonly used activation mappings naturally process the uniformly anti-monotone property defined in [24], i.e., sigmoidal mapping, nearest point projection mapping, linear saturating mapping, signum mapping, symmetric multi-valued step mapping, etc., we focus on studying the critical dynamics behaviors of RNNs with uniformly anti-monotone property. By applying Lyapunov functional method and Barbalat Lemma, we achieve the critical global asymptotically stable results for such type RNNs without diagonal nonlinear requirement. The results in Section 3 only need networks satisfy the critical condition, i.e., M₁ (Γ) ≥0, and do not require any other additional prerequisites, such as quasi-symmetric limitation on W [12], the nonlinear norm or matrix norm being bounded [19], and other restraints of RNNs in [15 –22]. Furthermore, because the critical dynamics results obtained here are for general activation mappings without diagonally nonlinear limitation, they can be directly applied to RNNs and improve deeply the existing ones for RNNs, such as Hopfield-type neural networks, Recurrent Back-propagation (ReBP) neural networks, Recurrent Back-propagation neural networks, Brain-State-in-a-Box/Domain type neural networks, Cellular neural networks, Bidirectional Associative Memory neural networks, Bound-constraints Optimization neural networks, and so on.

2 Preliminaries

For nonlinear activation mapping G : R^N → R^N, the range and fixed-point set of G are respectively defined by R (G) and T (G). Here we give some definitions about the nonlinear activation mapping G.

Definition 1. The nonlinear activation mapping $\begin{matrix} G (x) = (g_{1} (x), g_{2} (x), \dots, g_{N} (x))^{T} \end{matrix}$ is said to be diagonally nonlinear, if it has the strict component form $\begin{matrix} G (x) = (g_{1} (x_{1}), g_{2} (x_{2}), \dots, g_{N} (x_{N}))^{T} . \end{matrix}$

In the previous analysis of RNNs, especially in the critical analysis, the diagonally nonlinear property of RNNs is a necessary requirement for dealing with the derivative of a constructed energy function. While, the diagonally nonlinear requirement has an obviously limitation, i.e., for each g_i, we should consider the whole vector x, not only its component x_i.

Definition 2. [24] A mapping G : R^N → R^N is said to be α-uniformly anti-monotone (α-UAM) if there is a positive constant α such that for any x, y ∈ R^N, $〈 G (x) - G (y), x - y 〉 \geq α ∥ G (x) - G (y) ∥^{2}$ (3)

Many commonly used activation mappings naturally process the uniformly anti-monotone property. For example, sigmoidal mapping, nearest point projection mapping, signum mapping, symmetric multi-valued step mapping, linear saturating mapping, etc. Naturally, most of the RNN models being widely applied in various fields of science and engineering belong to the family with uniformly anti-monotone property.

Definition 3. We assume that each g_i : R^N → R (i = 1, 2, …, N) is Lipschitz continuous. L (g_i), the minimum Lipschitz constant of g_i, is defined as follows for any t, s ∈ R^N, $L (g_{i}) = sup_{t \neq s} \frac{| g_{i} (t) - g_{i} (s) |}{∥ t - s ∥_{2}} .$ (4)

Without loss of generality, through out this paper, we assume that each L (g_i) >0 and let L = diag {L (g₁) , L (g₂) , …, L (g_N)}, which is said to be minimum Lipschitz matrix of operator G (x) = (g₁ (x) , g₂ (x) , …, g_N (x)) ^T.

3 Critical global convergence results

In this section, we will establish the global asymptotically stable results for RNNs with α-UAM operators of both system (1) and (2), which are under the critical condition that the discriminant matrix defined by the network is positive semi-definite. It should be pointed out that all of the results don’t need the mapping G to be diagonal nonlinear, while the diagonal nonlinear property of the activation mapping is quite crucial for most of the critical dynamical analysis. We consider the networks of form (1) first. To be simple, we denote the range of the nonlinear activation operator, i.e. R (G), by Θ.

Suppose that Θ is bounded, closed and convex on R^N. For any y ∈ R (G), define T (y) = WG (y) + q and Fix(T) to be the fixed point set of T (y), then by Brouwer’s fixed point theorem, T has at least one fixed point y^* ∈ Fix (T). That is, T^-1 (0), the equilibrium state set of (1) is not empty. The following is the globally asymptotically stable theorem for system (1).

Theorem 1. Assume that G (x) = (g₁ (x) , g₂ (x) , …, g_N (x)) ^T is α-UAM, with Θ being a bounded, closed and convex subset of R^N. g_i is continuous and monotonically increasing. Let L = diag {L (g₁) , L (g₂) , ⋯ , L (g_N)}, where each L (g_i) is the minimum Lipschitz constant of g_i. Whenever there is a positive diagonal matrix Γ = diag {ξ₁, ξ₂, ⋯ , ξ_N} with ΓD being an identity matrix, such that the matrix

$M_{1} (Γ) = L^{- 1} D Γ - \frac{Γ W + W^{T} Γ}{2}$ (5) is nonnegative definite, then system (1) has a unique equilibrium state u^*, and u^* is globally asymptotically stable on W (Θ) + q.

Proof 1. Choose $u^{*} = (u_{1}^{*}, u_{2}^{*}, \dots, u_{N}^{*}) \in T^{- 1} (0)$ arbitrarily, and for any given trajectory u (t) = (u₁ (t) , u₂ (t) , ⋯ , u_N (t)) of system (1), we define

$\begin{matrix} \frac{dE (u (t))}{dt} & = & (Γ (G (u (t)) - G (u^{*})))^{T} (\frac{d (u (t) - u^{*})}{dt} \\ - {DL}^{- 1} (G (u (t)) - G (u^{*}))) \end{matrix}$ (6)

We will complete the proof in several steps.

Step 1) We show that dE (u (t))/dt ≤ 0.

Now, a direct calculation using system (1) gives

$\begin{matrix} \frac{dE (u (t))}{dt} & = & (G (u (t)) - G (u^{*}))^{T} {- Γ D (u (t) - u^{*}) \\ + Γ W (G (u (t)) - G (u^{*}))} \\ - (G (u (t)) - G (u^{*}))^{T} Γ D L^{- 1} \\ \times (G (u (t)) - G (u^{*})) \\ = & - (G (u (t)) - G (u^{*}))^{T} Γ D (u (t) - u^{*}) \\ + (G (u (t)) - G (u^{*}))^{T} \frac{Γ W + W^{T} Γ}{2} \\ \times (G (u (t)) - G (u^{*})) \\ - (G (u (t)) - G (u^{*}))^{T} Γ D L^{- 1} \\ \times (G (u (t)) - G (u^{*})) \\ = & - (G (u (t)) - G (u^{*}))^{T} Γ D (u (t) - u^{*}) \\ - (G (u (t)) - G (u^{*}))^{T} M_{1} (Γ) \\ \times (G (u (t)) - G (u^{*})) \end{matrix}$ (7)

Noting that M₁ (Γ) is nonnegative definite, so we have

$\begin{matrix} \frac{dE (u (t))}{dt} & \leq & - (G (u (t)) - G (u^{*}))^{T} \\ \times Γ D (u (t) - u^{*}) \end{matrix}$ (8)

Since G is α-UAM, we then have

$\begin{matrix} (G (u (t)) - G (u^{*}))^{T} (u (t) - u^{*}) \\ \geq α ∥ G (u (t)) - G (u^{*}) ∥^{2} \end{matrix}$ (9)

Because ΓD is an identity matrix, Then by (8) and (9), we get

$\begin{matrix} \frac{dE (u (t))}{dt} & \leq & - (G (u (t)) - G (u^{*}))^{T} (u (t) - u^{*}) \\ \leq & - α ∥ G (u (t)) - G (u^{*}) ∥^{2} \\ \leq & 0 \end{matrix}$ (10)

Step 2) We want to show that $lim_{t \to + \infty} ∥ G (u (t)) - G (u^{*}) ∥ = 0$ .

Since dE (u (t))/dt is a continuous function and u (t) ∈ Θ is a bounded and closed set, it follows that dE (u (t))/dt is a uniformly continuous function of t. Furthermore, by (10) we have dE (u (t))/dt ≤ 0, combined with the fact that E (u (t)) is bounded, implies that $lim_{t \to + \infty} E (u (t))$ exists. Thus, applying the well-known Barbalat Lemma, we obtain that $lim_{t \to + \infty} dE (u (t)) / dt = 0$ .

So we have

$\begin{matrix} 0 & = & lim_{t \to + \infty} \frac{dE (u (t))}{dt} \\ \leq & lim_{t \to + \infty} - α ∥ G (u (t)) - G (u^{*}) ∥^{2} \\ \leq & 0 \end{matrix}$ (11)

Consequently,

$lim_{t \to + \infty} - α ∥ G (u (t)) - G (u^{*}) ∥^{2} = 0$ (12)

Then it can be deduced that $lim_{t \to + \infty} ∥ G (u (t)) - G (u^{*}) ∥^{2} = 0$ , for the fact that α is greater than zero.

So, the result $lim_{t \to + \infty} ∥ G (u (t)) - G (u^{*}) ∥ = 0$ is thus proved.

Step 3) To show that $lim_{t \to + \infty} u (t) = u^{*}$ .

By the differential equation theory, we can also solve the following integral equation: $\begin{matrix} u (t) - u^{*} \\ = e^{- (t - s) D} (u (s) - u^{*}) \\ + \int_{s}^{t} e^{- (t - τ) D} W [G (u (τ)) - G (u^{*})] d τ, t > s \end{matrix}$

Let $d_{\min} = min_{1 \leq j \leq N} d_{j}$ . Then, ∀t > s, there holds

$\begin{matrix} ∥ u (t) - u^{*} ∥ & \leq & e^{- (t - s) d_{\min}} ∥ u (s) - u^{*} ∥ \\ + \int_{s}^{t} e^{- (t - τ) d_{\min}} ∥ W ∥ \\ \times ∥ G (u (τ)) - G (u^{*}) ∥ d τ \end{matrix}$ (13)

By Step 2), $lim_{t \to + \infty} ∥ G (u (t)) - G (u^{*}) ∥ = 0$ , then, for any ɛ > 0, there is a T_ɛ > 0 such that, whenever t ≥ T_ɛ, $\begin{matrix} ∥ G (u (t)) - G (u^{*}) ∥ \leq \frac{d_{\min}}{∥ W ∥} \cdot ɛ \end{matrix}$

Therefore, we conclude from (8) that, when t > s ≥ T_ɛ, $\begin{matrix} ∥ u (t) - u^{*} ∥ & \leq & e^{- (t - s) d_{\min}} ∥ u (s) - u^{*} ∥ \\ + & ɛ \cdot d_{\min} \int_{s}^{t} e^{- (t - τ) d_{\min}} d τ \\ < & e^{- (t - s) d_{\min}} ∥ u (s) - u^{*} ∥ + ɛ \end{matrix}$

Letting t→ + ∞ in the above inequality yields $lim_{t \to + \infty} ∥ u (t) - u^{*} ∥ \leq ɛ$ , which implies $lim_{t \to + \infty} u (t) = u^{*}$ since ɛ is arbitrary.

Step 4) Finally, we prove that the equilibrium state of system (1) is unique.

Without loss of generality, we assume that v^* ≠ u^* is also the equilibrium state of system (1), i.e., $lim_{t \to + \infty} ∥ u (t) - v^{*} ∥ \leq ɛ$ . Then, obviously combined with the conclusion got from step 3), we have $\begin{matrix} lim_{t \to + \infty} ∥ u^{*} - v^{*} ∥ \\ = lim_{t \to + \infty} ∥ u^{*} - u (t) + u (t) - v^{*} ∥ \\ \leq lim_{t \to + \infty} ∥ u (t) - u^{*} ∥ \\ + lim_{t \to + \infty} ∥ u (t) - v^{*} ∥ \\ \leq 2 ɛ \end{matrix}$

Similarly, because ɛ is arbitrary, the equation $lim_{t \to + \infty} ∥ u^{*} - v^{*} ∥ = 0$ is correct. Further, as a matter of fact that u^*, v^* are constant, we can get that u^* = v^*, which contradicts to the assumption (v^* ≠ u^*). The equilibrium state of system (1) is unique. This completes the proof of the theorem.

Theorem 1 gives the critically globally asymptotically stable results of system (1) without diagonal nonlinear requirment, that is, the activation mapping G has the form G (x) = (g₁ (x) , g₂ (x) , ⋯ , g_N (x)) ^T. Correspondingly, we can also deduce the critical globally asymptotically stable conclusion for RNN system (2).

Corollary 1. Assume that G (x) = (g₁ (x) , g₂ (x) , …, g_N (x)) ^T is α-UAM, with Θ being a bounded, closed and convex subset of R^N. g_i is continuous and monotonically increasing. Let L = diag {L (g₁) , L (g₂) , ⋯ , L (g_N)}, where L (g_i) is the minimum Lipschitz constant of g_i. Then system (2) has a unique equilibrium state x^*, and x^* is globally asymptotically stable on Θ, if there is a positive diagonal matrix Γ = diag {ξ₁, ξ₂, ⋯ , ξ_N} with ΓD is an identity matrix, such that the matrix M₁ (Γ) = L^-1DΓ - (ΓW + W^TΓ)/2 is nonnegative definite.

Proof. By [10, 11], the convergence of x (t) to an equilibrium state of (2) can be shown by studying the asymptotic behavior of u (t). Then we can get the conclusion of Corollary 3 from Theorem 1 directly.

Remark 1. In this section, we present the globally asymptotical stability results for both local field RNNs and static RNNs with uniformly anti-monotone property under the critical conditions. Since the critical analysis for RNNs can be considered as the essentially discovery for RNNs in the distinct region between stability and non-stability, studying the critical dynamic behaviors of RNNs can find broad applications. While, due to the difficulty in the dynamical analysis of RNNs under the critical condition, most of the critical analysis are with the crucial hypothesis that the activation mappings should be diagonally nonlinear. Theorem 1 and Corollary 1 remove the limitation directly. For all kinds of RNNs with uniformly anti-monotone property (this property is quite common in most of the widely used RNNs), we get the globally asymptotical stability results for the two basic RNN models under the critical conditions without the diagonally nonlinear requirement on activation mappings. In addition, since RNNs with uniformly anti-monotone property can formalize many existing RNNs individuals, thus the achieved analysis of dynamics behaviors in this section present some uniform results, and them can discriminate the similarity and redundant of the dynamics results among many of the known RNNs individuals and improve them deeply, let alone those under the critical conditions (see, e.g., [10–12 , 20–23] and the references therein).

Remark 2. The main difficulty of dealing with the activation function without diagonal nonlinearity lies in how to avoid the inner product term in the derivation of the energy function. That is because, once the derivation of the energy function has an inner product term, the activation function must have the diagonal nonlinearity property, otherwise the non-positive requirement of the derivation of the energy function cannot be deduced anymore under the critical condition.

In this paper, when there exists a positive diagonal matrix Γ, such that ΓD is an identity matrix (here D is a matrix defined by the network) and the critical condition holds (i.e., M₁ (Γ) is a nonnegative definite matrix), then the neural network has a unique equilibrium state and which is globally asymptotically stable. The results directly removed the requirement of diagonally nonlinear property on the activation function.

4 Illustrative examples

In this section, we provide two illustrative examples to demonstrate the validity of the critical stability results formulated in the previous section. Since most of the existing dynamical results for RNNs are based on the assumption that activation mappings of the network are diagonal nonlinear, thus, most of the known dynamical results developed in literature can not be applied here.

Example 1. Consider the following recurrent neural network of system (1): $\frac{du (t)}{dt} = - Du (t) + WG (u (t)) + q$ (14) where the activation mapping G = (g₁, g₂, g₃) ^T, g_i (i = 1, 2, 3) is defined as follows: $\begin{matrix} g_{i} (u) = {\begin{matrix} u_{i}, & ∥ u ∥_{2} \leq 1 \\ \frac{u_{i}}{∥ u ∥_{2}}, & ∥ u ∥_{2} > 1 \end{matrix} \end{matrix}$ (15)

The weight matrix and the external bias vector are defined as follows: $\begin{matrix} W = (\begin{matrix} 0.8 & 0.3 & 0.15 \\ 0.1 & 0.7 & - 0.4 \\ - 0.15 & 0.4 & 1 \end{matrix}), & q = (\begin{matrix} 1 / 10 \\ - \sqrt{2} / 5 \\ 1 / 4 \end{matrix}) \end{matrix}$

In this example, it is easy to verify that for any positive diagonal matrix Γ, the matrix M₁ (Γ) is not positive definite. That is to say, Lemma 3 in [11] is not suitable here. And for this example, it is established on a general projection operator, so the diagonally nonlinear results, i.e., Theorem 3 in [12], Theorem 2 in [18], and Corollary 3.2 in [19] cannot be used here. We will show the conditions in Theorem 1 proposed herein is satisfied. Actually, in this example, we have α = 1 by the definition of G, thus L = I. Let v₁ ≠ v₂ ∈ R³, and since G is α-UAM, we can get that ∥G (v₁) - G (v₂) ∥ ₂ ≤ α^-1 ∥ v₁ - v₂ ∥ ₂ = ∥ v₁ - v₂ ∥ ₂. In addition, since ∥G (v₁) - G (v₂) ∥ ₂ ≥ ∥ g₁ (v₁) - g₁ (v₂) ∥ ₂, so ∥g₁ (v₁) - g₁ (v₂) ∥ ₂ ≤ ∥ v₁ - v₂ ∥ ₂, and one can get that L (g₁) ≤1. On the other hand, by taking v₁ = (- δ, 0, - 1) , v₂ = (δ, 0, - 1), thus $∥ g_{1} (v_{1}) - g_{1} (v_{2}) ∥_{2} = (1 + δ^{2})^{- \frac{1}{2}} ∥ v_{1} - v_{2} ∥_{2}$ . Further, for any ɛ > 0, there exists $0 < δ \leq \sqrt{(1 - ɛ)^{- 2} - 1}$ , such that, ∥g₁ (v₁) - g₁ (v₂) ∥ ₂ ≥ (1 - ɛ) ∥ v₁ - v₂ ∥ ₂. Since ɛ is arbitrary, we have L (g₁) ≥1. As a result, L = I. Choosing D = Γ = I, it is clear that L^-1DΓ - (Γ + W^TΓ)/2 is nonnegative definite. Hence, by Theorem 1, system (14) is global asymptotically stable on W (Θ) + q (here Θ = {u ∈ R³ : ∥ u ∥ ₂ ≤ 1}).

Figure 1 depicts the time responses of state variables of system (14) starting randomly fromW (Θ) + q.

Fig.1

Transient behaviors of RNN in system (14) with random initial points u₀ ∈ W (Θ) + q.

Example 2. Consider the following recurrent neural network with the form of system (2): $\frac{dx (t)}{dt} = - Dx (t) + G (Wx (t) + b)$ (16) where G is defined as a generic projection operator, i.e. $G (x) = \underset{z \in Θ}{argmin} ∥ x - z ∥_{2}$ . The weight matrix and the external bias vector are defined as follows: $\begin{matrix} W = (\begin{matrix} - 1 / 6 & 1 & - 0.45 & - 1 & 1 & 1 \\ - 1 & 1 / 5 & 1 & 0.3 & - 1 & - 1 \\ 0.45 & - 1 & - 1 / 4 & 1 & - 1 & 1 \\ 1 & - 0.3 & - 1 & 1 / 3 & 1 & - 1 \\ - 1 & 1 & 1 & - 1 & - 1 / 2 & - 0.8 \\ - 1 & 1 & - 1 & 1 & 0.8 & 1 \end{matrix}) \end{matrix}$ and $\begin{matrix} q = {(\begin{matrix} - 5 / 6, & 4 / 5, & 3 / 4, & - 4 / 3, & 1 / 2, & - 2 \end{matrix})}^{T} \end{matrix}$

Obviously, the activation mapping G is a general projection operator and for any positive diagonal matrix Γ, M₁ (Γ) is not positive definite in the present case.

Since the conclusions about global convergence of the system (16) are proved under the condition that G is diagonally nonlinear in Theorem 1 of [18] and Theorem 3.1 [19], so both Theorem 1 in [18] and Theorem 3.1 in [19] cannot be used here. Furthermore, Lemma 2 in literature [11] is unsuitable here, for the global convergence of the system (16) being guaranteed under special conditions of M₁ (Γ) is positive definite and G is diagonally nonlinear in the Lemma 2. We show that conditions in Corollary 1 proposed herein is satisfied. Actually, just like the proof in Example 1, here α = 1 and L = I. By taking D = Γ = I, it is clear that L^-1DΓ - (ΓW + W^TΓ)/2 is nonnegative definite. Hence, by Corollary 1, system (16) is global asymptotically stable on Θ (here Θ = {v ∈ R⁶ : ∥ v ∥ ₂ ≤ 1}). Figure 2 depicts the time responses of neural state variables of the system starting randomly from Θ.

Fig.2

Transient behaviors of RNN in system (14) with random initial points x₀ ∈ Θ.

5 Conclusion

We have developed the critical stability results for both the local field RNNs and the static RNNs with α-UAM mappings and without diagonal nonlinear requirement. Based on exploring some intrinsic properties of RNNs, and by combining Lyapunov functional method and Barbalat Lemma, it has been proved that the RNN has a unique equilibrium state and it is globally asymptotically stable in the sense that a discriminant matrix M₁ (Γ) is nonnegative definite. Compared with the existing dynamics analysis, the results here extend most of the dynamics conclusions achieved, and what is the most important, the limitation of the activation mappings have been deeply relaxed.

How to improve the critical stability results without the diagonal nonlinearity requirement on the activation function by some other technique, e.g., by matrix theory and nonlinear analysis methods, and how to present some new results by avoiding the inner product term in the derivation of the energy function with some looser condition than ΓD being an identity matrix, are still full of challenge. Moreover, in this paper, we did not discuss the exponential stability for RNNs, nor consider the dynamics analysis for RNNs under some more critical conditions, e.g., M₁ (Γ) + P ≥ 0. All of them are quite meaningful in applications of RNNs, and they are under our current investigation.

Footnotes

Acknowledgments

This research was supported by NSFC Nos. 11471006 and 11101327, National Science and Technology Program of China (No. 2015DFA81780), the Fundamental Research Funds for the Central Universities (No. xjj2017126) and was partly Supported by HPC Platform, Xi’an Jiaotong University.

References

Zhang

H.G.

, Wang

Z.S.

and Liu

D.R.

, A comprehensive review of stability analysis of continuous-time recurrent neural networks, IEEE Transactions on Neural Networks & Learning Systems 25(7) (2014), 1229–1262.

Wang

Z.S.

, Ding

S.B.

, Shan

Q.H.

and Zhang

H.G.

, Stability of recurrent neural networks with time-varying delay via flexible terminal method, IEEE Transactions on Neural Networks & Learning Systems PP(99) (2016), 1–8.

Chen

T.P.

, Global convergence of delayed dynamical systems, IEEE Trans Neural Networks 12(6) (2001), 1532–1536.

Liang

X.B.

and Si

, Global exponential stability of neural networks with globally Lipschitz continuous activation and its application to linear variational inequality problem, IEEE Trans Neural Networks 12(2) (2001), 349–359.

Qiao

, Peng

and Xu

Z.B.

, Nonlinear measures: a new approach to exponential stability analysis for Hopfield-type neural networks, IEEE Trans Neural Networks 12(2) (2001), 360–370.

Liu

and Chen

, A new result on the global convergence of Hopfield neural networks, IEEE Transactions on Circuits and Systems 49(10) (2002), 1514–1516.

Liu

Y.J.

, Li

, Tong

S.C.

and Chen

C.L.P.

, Neural network control-based adaptive learning design for nonlinear systems with full-state constraints, IEEE Transactions on Neural Networks & Learning Systems 27(7) (2016), 1–10.

Chen

C.L.P.

, Liu

Y.J.

and Wen

G.X.

, Fuzzy neural network-based adaptive control for a class of uncertain nonlinear stochastic systems, IEEE Transactions on Cybernetics 44(5) (2014), 583–593.

Liu

Y.J.

, Tong

, Chen

C.L.P.

, et al., Neural controller design-based adaptive control for nonlinear MIMO systems with unknown hysteresis inputs, Cybernetics IEEE Transactions on 46(1) (2016), 9–19.

10.

Z.B.

, Qiao

, Peng

and Zhang

, A comparative study of two modeling approaches in neural networks, Neural Networks 17(1) (2004), 73–85.

11.

Qiao

, Peng

, Xu

Z.B.

and Zhang

, A reference model approach to stability analysis of neural networks, IEEE Trans Syst Man Cybern 33(6) (2003), 925–936.

12.

Peng

, Xu

Z.B.

, Qiao

and Zhang

, A critical analysis on global convergence of Hopfield-type neural networks, IEEE Trans Circuits Syst 52(4) (2005), 804–814.

13.

Chen

, Mao

H.Z.

and Qiao

, Without diagonal nonlinear requirements: The more general-critical dynamical analysis for UPPAM recurrent neural networks, Mathematical Problems in Engineering 2013 (2013), 1–10.

14.

Qiao

and Xu

Z.B.

, Critical dynamics study on recurrent neural networks: Globally exponential stability, Neurocomputing 77(1) (2012), 205–211.

15.

Liu

X.W.

and Chen

T.P.

, A new result on the global convergence of Hopfield neural networks, IEEE Trans Circuits Syst 49(10) (2002), 1514–1516.

16.

Yang

Y.Q.

and Cao

, Solving quadratic programming problems by delayed projection neural network, IEEE Trans Neural Networks 17(6) (2006), 1630–1634.

17.

Qiao

and Xu

Z.B.

, New critical analysis on global convergence of recurrent neural networks with projection mappings, ISNN 07, Lecture Notes in Computer Science, Springer, Berlin 4493 (2007), 131–139.

18.

Qiao

and Xu

Z.B.

, A critical global convergence analysis of recurrent neural networks with general projection mappings, Neurocomputing 72(7) (2009), 1878–1886.

19.

Qiao

and Xu

Z.B.

, On the P-critical dynamics analysis of projection recurrent neural networks, Neurocomputing 73(13) (2010), 2783–2788.

20.

Qiao

, Jing

W.F.

and Xu

Z.B.

, The UPPAM continuoustime RNN model and its critical dynamics, Neurocomputing 106 (2013), 158–166.

21.

Qiao

, Liang

and Sun

K.F.

, Dynamics analysis for generic projection continuous-time RNNs with bounded matrices, Journal of Systems Science and Complexity 28(4) (2015), 799–812.

22.

Qiao

, Chen

H.B.

, Jing

W.F.

and Sun

K.F.

, Towards establishing a meaningful and practical dynamics results for the unified RNN model, Neurocomputing 157(1) (2015), 315–322.

23.

Qiao

, Jing

W.F.

and Wang

Y.P.

, The general critical analysis for continuous-time UPPAM recurrent neural networks, Neurocomputing 175(Part A) (2016), 40–46.

24.

Z.B.

and Qiao

, Towards a unified feedback neural network theory: The uniformly pseudo-projection-anti-monotone net, Acta Mathematica Sinica English 27(2) (2011), 377–396.