An adaptive incremental TSK fuzzy system based on stochastic configuration and its approximation capability analysis

1 Introduction

In the past few decades, Takagi-Sugeno-Kang (TSK) fuzzy system [1], as one of the most important and popular fuzzy systems, has received more and more attention and has been successfully applied in many different fields, such as data modeling [2], pattern recognition [3], model approximation [4], adaptive control [5], etc. However, as the dimension and scale of the dataset increase, the rule base and the parameters of TSK fuzzy system increase exponentially. How to determine a TSK fuzzy system with appropriate architecture and good learning and generalization performance is a significant research problem.

The task of designing a TSK fuzzy system includes structure recognition and parameters optimization of both fuzzy antecedents and functional consequents. Fuzzy clustering methods [6, 7] have been widely applied for structure recognition, optimization techniques [8, 9], feature selection [10] and back propagation (BP) algorithm are used for parameters optimization.

In the process of using fuzzy clustering methods to determine the system structure, it is a crucial problem to determine an appropriate number of clustering in advance, which determines the number of rules. The performance of fuzzy systems with few rules is poor, while large rules may cause overfitting.

The parameters learning of TSK fuzzy systems can be realized by data-driven techniques similar to artificial neural network training method. Randomized approaches which have great potential in developing fast training algorithms become popular as a result of their good modeling performance and training speed. In [11], The ELM method is applied to optimize the fuzzy system parameters. In [12], a regularization ELM is introduced to randomly select fuzzy layer parameters and to determine neural layer parameters by solving optimization problems in regularization framework. The scope of random parameters is important to the training process of randomized approaches. In [13], IRVFLN’s universal approximation capacity is found to be affected by the scope of random parameters. In [14], stochastic configuration network (SCN) is presented to solve the problem. It ensures universal approximation by assigning random parameters under a supervisory mechanism. Various SCN models [15–17] have been proposed for data modeling to obtain universal approximation. In [18], a complex multidimensional numerical integration method based on SCN is proposed to deal with engineering problems. Deep stacked SCN is presented for data streams modeling in [19]. In [20], SCN’s structural parameters are improved by incorporating the driving amount.

According to current results, determining an appropriate number of fuzzy rules is a critical step in improving fuzzy system approximation. The traditional learning methods of fuzzy system are to set the number of rules in advance. This may affect the approximation ability and generalization ability of fuzzy system. In addition, although the randomized approaches have been used to optimize the parameters of fuzzy system, how to ensure the universal approximation of randomly generated fuzzy systems is still a problem.

Motivated by these facts, in this study an adaptive incremental TSK fuzzy system based on stochastic configuration (SCFS) is proposed. We determine the architecture by an adaptive incremental learning approach and randomly assign the premise parameters with inequality constraints so as to establish a TSK fuzzy system with appropriate fuzzy rules and good approximation capability. The incremental learning approach of SCFS starts with an initial structure and then gradually adds fuzzy rules until predefined termination criteria are met. Obtaining a compact fuzzy system and an appropriate number of fuzzy rules are straightforward. Fuzzy rules are added randomly using a supervisory mechanism which ensures universal approximation. The proposed SCFS is evaluated by a number of popular regression benchmarks and compared with some start-of-the-art approaches. It appears that SCFS is capable of achieving satisfactory performance based on numerical examples.

The rest of paper starts with some preliminaries for TSK Fuzzy system in section 2. The construction of SCFS is introduced in section 3. Section 4 investigates approximation properties of SCFS and section 5 proposes the parameters learning algorithm of SCFS. In section 6, a series of synthetic data and benchmarks datasets are carried out to illustrate the performance of SCFS. Section 7 gives some conclusions.

2 Preliminaries

TSK fuzzy system is the most popular fuzzy system, and its fuzzy rules have the following form: R k : If x 1 is A k 1 and ⋯ and x m is A km , then y = f k ( x ) , k = 1 , 2 , ⋯ , K ,(1) where x = ( x 1 , x 2 ⋯ , x m ) T ∈ ℝ m is the system input vector, A _ki is a fuzzy set subscribed by x _i , consequent part f _k  (x) is a function of input x.

The rule R _k of TSK fuzzy system maps antecedent fuzzy sets (A_k1, ⋯ , A _km ) to consequent function f _k  (x). By sum-product inference, the defuzzified output is written as follows: y = ∑ k = 1 K τ k ( x ) f k ( x ) ∑ k = 1 K τ k ( x ) ,(2) where τ k ( x ) = ∏ i = 1 m A ki ( x i ) is the activation level of the rule R _k .

Centralized TSK fuzzy system which omit the denominator in (2) is suggested in [21–23] and the output can be simplified into y = ∑ k = 1 K τ k ( x ) f k ( x ) ,(3) which is adopt in this study.

The consequent function f _k  (x) usually is a polynomial of the input variables. The zeros-order and first-order TSK fuzzy systems are two popular TSK fuzzy system, in which constant and first order polynomial are chosen as consequent function respectively. However, the consequent function can also be other function. In [24], Chebyshev polynomial is used to improve the nonlinear expression ability of the consequent function, which is employed in this study and has the following form: f k ( x ) = φ ( x ) T a k , ( 4 )(4) where φ (x) =   (1, T₁ (x₁) , ⋯ , T₁ (x _m ) , T₂ (x₁) , ⋯ , T _n  (x _m ))  ^T , T_l+1 (x _i ) is Chebyshev polynomial with T_l+1 (x _i ) = 2x _i T _l  (x _i ) - T_l-1 (x _i ), T₀ (x _i ) = 1 and T₁ (x _i ) = x _i , n is the degree of Chebyshev polynomial and a _l is the consequent parameter.

The construction of SCFS is elaborated in this section. SCFS retains the form (3) of a centralized TSK fuzzy system, but adopts the idea of stochastic configuration to gradually add rules until specified condition is met.

Given a target function f : ℝ m → ℝ and an initial SCFS S _{L 0} with L₀ fuzzy rules, R k : If x 1 is A k 1 and ⋯ and x m is A km , then y = f k ( x ) , k = 1 , 2 , ⋯ , L 0 , where the consequent function f _k  (x) adopts the function of Chebyshev polynomial of form (4), i.e., f _k  (x) = φ (x)  ^T a _k , the membership functions of antecedent fuzzy set A _ki is of the form A ki ( x i ) = exp [ - ( ( x i - c ki ) σ ki ) 2 ] where c _ki and σ _ki are center and width of fuzzy set A _ki . The premise parameters of R _k are denoted as c k = ( c k 1 , ⋯ , c km ) and σ k = ( σ k 1 , ⋯ , σ km ) .

Hence, the output of S _{L 0} can be represented as S L 0 ( x ) = ∑ k = 1 L 0 τ k ( x ) φ ( x ) T a k and the residual error ofS _{L 0} is e L 0 ( x ) = f ( x ) - S L 0 ( x ) , where the activation level τ k ( x ) = ∏ i = 1 m A ki ( x i ) .

If ∥e _{L 0} ∥ fails to reach the tolerance level ɛ, a new fuzzy rule R_L0+1 will be added based on a supervisory mechanism ( ∑ i = 1 N τ L 0 + 1 ( x i ) e L 0 ( x i ) ) 2 ∑ i = 1 N τ L 0 + 1 ( x i ) 2 ∑ i = 1 N e L 0 ( x i ) 2 ≥ r L 0 + 1 where r_L0+1 is a given real number. The supervisory mechanism can ensure the new SCFS S_L0+1 have an improved residual error. The new rule R_L0+1 has the following form: R L 0 + 1 : If x 1 is A ( L 0 + 1 ) 1 and ⋯ and x m is A ( L 0 + 1 ) m , then y = φ ( x ) T a L 0 + 1 where c_L0+1 andσ_L0+1 are randomly allocated based the supervisory mechanism, and a_L0+1 can be evaluated easily by using Moore-Penrose generalized inverse to solve a least square problem.

Hence, the output of S_L0+1 can be represented as S L 0 + 1 ( x ) = ∑ k = 1 L 0 + 1 τ k ( x ) φ ( x ) T a k ,(5) and the residual error of S_L0+1 is e L 0 + 1 ( x ) = e L 0 ( x ) - τ L 0 + 1 ( x ) φ ( x ) T a L 0 + 1 .(6)

The incremental learning approach of SCFS, which repeats the above incremental process and gradually adds new rules until the predefined tolerance level ɛ is met, is illustrated in Fig. 1.

OPEN IN VIEWER

Fig. 1

The incremental learning approach of SCFS.

The initial system S _{L 0} with L₀ fuzzy rules can either be generated by traditional methods or is a null system. For given target function f, new rules are produced incrementally under a supervisory mechanism for improving the performance of the SCFS until specified performance ɛ is met.

It can be found that different form the traditional fuzzy system which requires a given number of fuzzy rules in advance, SCFS increases rules gradually by incremental learning to obtain a suitable number of rules. And the supervision mechanism of SCFS can ensure universal approximation which is proved in the next section.

In this section, the approximation capability of SCFS will be analyzed theoretically.

Given dataset with inputs X = ( x 1 , x 2 , ⋯ , x N ) T ∈ ℝ N × m and outputs Y = f ( X ) = ( y 1 , y 2 , ⋯ , y N ) T ∈ ℝ N , where x i = ( x i 1 , x i 2 , ⋯ , x im ) ∈ ℝ m . The output of the SCFS S_L-1 with L - 1 fuzzy rules on dataset {X, Y} can be represented as S L - 1 ( X ) = ( S L - 1 ( x 1 ) , S L - 1 ( x 2 ) , ⋯ , S L - 1 ( x N ) ) T and the residual error of S_L-1 is e L - 1 ( X ) = ( e L - 1 ( x 1 ) , e L - 1 ( x 2 ) , ⋯ , e L - 1 ( x N ) ) T where S L - 1 ( x i ) = ∑ k = 1 L - 1 τ k ( x i ) φ ( x i ) T a k , e L - 1 ( x i ) = y i - S L - 1 ( x i ) .

If e_L-1 (X) ≥ ɛ, new fuzzy rule R _L is generated and added to the old SCFS S_L-1 to get the new SCFS S _L with L fuzzy rules. The output and the residual error of the S _L on dataset {X, Y} can be represented as S L ( X ) = ∑ l = 1 L η l T a l = S L - 1 ( X ) + η L T a L . ∥(7) e L ( X ) = Y - S L ( X ) = e L - 1 ( X ) - η L T a L(8) where η _l  = (τ _l  (x₁) φ (x₁) , ⋯ , τ _l  (x _N ) φ (x _N ))  ^T  .

The premise parameters c _L and σ _L of new added rule R _L are randomly assigned under inequality constraint.

Then, we will give the first theorem of SCFS and analyze the convergence of the approximation error as the number of rules increases.

Theorem 1. Given r L ∈ [ 1 N + 1 , e L - 1 ( x q ) 2 ∑ i = 1 N e L - 1 ( x i ) 2 ) , if the randomly generated premise parameters c _L , σ _L satisfy the following inequality ( ∑ i = 1 N τ L ( x i ) e L - 1 ( x i ) ) 2 ∑ i = 1 N τ L ( x i ) 2 ∑ i = 1 N e L - 1 ( x i ) 2 ≥ r L ,(9) and the consequent parameters a _L ∈ ℝ nm + 1 are evaluated as a L = ( ∑ i = 1 N τ L ( x i ) e L - 1 ( x i ) ∑ i = 1 N τ L ( x i ) 2 , 0 , ⋯ , 0 ) T , then, we have lim L → + ∞ e L ( X ) = 0 .

where τ L ( x i ) = exp [ - ∑ j = 1 m ( ( x ij - c Lj ) σ Lj ) 2 ] and x q ∈ { x j | | e L - 1 ( x j ) | = max i | e L - 1 ( x i ) | } .

Proof: From (8), we have ∥ e L ( X ) ∥ 2 = ∥ e L - 1 ( X ) - η L T a L ∥ 2 = ∥ e L - 1 ( X ) ∥ 2 - 2 e L - 1 ( X ) T η L T a L + a L T η L η L T a L .

Since a L = ( ∑ i = 1 N τ L ( x i ) e L - 1 ( x i ) ∑ i = 1 N τ L ( x i ) 2 , 0 , ⋯ , 0 ) T , then η L T a L = ( τ L ( x 1 ) , ⋯ , τ L ( x N ) ) T ∑ i = 1 N τ L ( x i ) e L - 1 ( x i ) ∑ i = 1 N τ L ( x i ) 2 .

So, ∥e _L  (X) ∥ ² = ∥ e L - 1 ( X ) ∥ 2 ( 1 - ( ∑ i = 1 N τ L ( x i ) e L - 1 ( x i ) ) 2 ∑ i = 1 N τ L ( x i ) 2 ∑ i = 1 N e L - 1 ( x i ) 2 ) ≤ ∥ e L - 1 ( X ) ∥ 2 ( 1 - r L ) ≤ ∥ e 0 ( X ) ∥ 2 Π l = 1 L ( 1 - r L ) which implies that ∥ e L ( X ) 2 ≤ e 0 ( X ) ∥ 2 ( 1 - 1 1 + N ) L . and lim L → + ∞ ∥ e L ( X ) ∥ = 0 . □

Remark 1. We present a constructive scheme for consequent parameters in Theorem 1, which leads to the convergence of approximation error of SCFS. The inequality constraint (9) is proposed for finding appropriate premise parameters c _L and σ _L . According to the proof of Theorem 1, the inequality constraint (9) can ensure the universal approximation, which is the theoretical basis of data modeling using SCFS.

Then, the next question is the existence of c _L and σ _L satisfying the supervisory mechanism (9), which is studied in the following theorem 2.

Theorem 2. Suppose dataset X = (x₁, x₂, ⋯ , x _N )  ^T satisfying x _i  ≠ x _q for any i ≠ q, for any r L < e L - 1 ( x q ) 2 ∑ i = 1 N e L - 1 ( x i ) 2 , there exist c _L and σ _L , such that ( ∑ i = 1 N τ L ( x i ) e L - 1 ( x i ) ) 2 ∑ i = 1 N τ L ( x i ) 2 ∑ i = 1 N e L - 1 ( x i ) 2 ≥ r L . where τ L ( x i ) = exp [ - ∑ j = 1 m ( ( x ij - c Lj ) σ Lj ) 2 ] and x q ∈ { x j | | e L - 1 ( x j ) | = max i | e L - 1 ( x i ) | } .

Proof: Let c _L  = x_q, then τ _L  (x_q) = 1. Since dataset X satisfy that x _i  ≠ x _q for any i ≠ q, we can get lim σ L → 0 τ L ( x i ) = 0 , which implies that there exists σ _L such that max i ≠ q τ L ( x i ) ≤ u for any u ∈ ( 0 , 1 2 ( N - 1 ) ] .

Thus, we have ∑ i = 1 N τ L ( x i ) 2 = 1 + ∑ i ≠ q τ L ( x i ) 2 ≤ 1 + ( N - 1 ) u 2 .

On the other hand, ( ∑ i = 1 N τ L ( x i ) e L - 1 ( x i ) ) 2 = ( | e L - 1 ( x q ) | + ∑ i ≠ q ( τ L ( x i ) · δ iq · | e L - 1 ( x i ) | ) ) 2 where δ iq = sign ( e L - 1 ( x i ) ) sign ( e L - 1 ( x q ) ) , sign (·) is sign function. Since e_L-1 (X)  > ɛ implies e_L-1 (x _q ) ≠ 0 and δ _iq  ≥ -1, we can get ( ∑ i = 1 N τ L ( x i ) e L - 1 ( x i ) ) 2 ≥ ( | e L - 1 ( x q ) | - ∑ i ≠ q u | e L - 1 ( x i ) | ) 2 . ≥ ( 1 - 2 u ( N - 1 ) ) | e L - 1 ( x q ) | 2 .

Hence, we have ( ∑ i = 1 N τ L ( x i ) e L - 1 ( x i ) ) 2 ∑ i = 1 N τ L ( x i ) 2 ∑ i = 1 N e L - 1 ( x i ) 2 ≥ ( 1 - 2 u ( N - 1 ) ) e L - 1 ( x q ) 2 ( 1 + ( N - 1 ) u 2 ) ∑ i = 1 N e L - 1 ( x i ) 2 .

We notice that ( 1 - 2 u ( N - 1 ) ) e L - 1 ( x q ) 2 ( 1 + ( N - 1 ) u 2 ) ∑ i = 1 N e L - 1 ( x i ) 2 goes up to e L - 1 ( x q ) 2 ∑ i = 1 N e L - 1 ( x i ) 2 when u → 0⁺. So, we can know that for any r L < e L - 1 ( x q ) 2 ∑ i = 1 N e L - 1 ( x i ) 2 , there exists u 0 ∈ ( 0 , 1 2 ( N - 1 ) ] such that ( 1 - 2 u ( N - 1 ) ) e L - 1 ( x q ) 2 ( 1 + ( N - 1 ) u 2 ) ∑ i = 1 N e L - 1 ( x i ) 2 ≥ r L for any u < u₀. □

Remark 2. Theorem 2 proves the existence of c _L and σ _L satisfying the supervisory mechanism which is used in Theorem 1 to ensure the continuous approximation improvement. In fact, according to the proof of Theorem 2, we can easily find the feasible parameter values c L = x q and σ Lj = min k ∈ Q j ( x kj - x qj ) 2 where Q j = { k ∈ { 1 , 2 , ⋯ , N } | ( x kj - x qj ) 2 ≠ 0 } , which means that the premise parameters of fuzzy rule satisfying the inequality constraint are sure to be found.

The consequent parameters a _L in Theorem 1 are given and remain fixed. This may lead a slow convergence of incremental learning. In fact, there is a recalculation scheme. The consequent parameters are determining by minimizing global residual and recalculating as rule increases. Theorem 3 gives the convergence when the consequent parameters a _L are updated by the recalculation scheme.

Theorem 3. Given r L ∈ [ 1 N + 1 , e L - 1 ( x q ) 2 ∑ i = 1 N e L - 1 ( x i ) 2 ) , if the randomly generated premise parameters c _L , σ _L satisfy the following inequality ( ∑ i = 1 N τ L ( x i ) e L - 1 * ( x i ) ) 2 ∑ i = 1 N τ L ( x i ) 2 ∑ i = 1 N e L - 1 * ( x i ) 2 ≥ r L , and the consequent parameters are evaluated by [ a 1 * , ⋯ , a L * ] = arg min a ∥ Y - ∑ l = 1 L η l T a l ∥ , ( 10 )(10) rthen we have lim L → + ∞ e L * ( X ) = 0 . where e L * ( X ) = Y - ∑ l = 1 L η l T a l * .

Proof: Denote e ˆ L ( X ) = e L - 1 * ( X ) - η L T a ˆ L where a ˆ L = ( ∑ i = 1 N τ L ( x i ) e L - 1 ( x i ) ∑ i = 1 N τ L ( x i ) 2 , 0 , ⋯ , 0 ) .

It is obvious that ∥ e L * ( X ) ∥ 2 ≤ ∥ e ˆ L ( X ) ∥ 2 ≤ ( 1 - r L ) ∥ e L - 1 * ( X ) ∥ 2 which implies that ∥ e L * ( X ) ∥ 2 ≤ ∥ e 0 * ( X ) ∥ 2 ( 1 - 1 N + 1 ) L and lim L → + ∞ ∥ e L * ( X ) ∥ = 0 . □

Remark 3. The optimization of consequent parameters in Theorem 3 can be evaluated by using Moore-Penrose generalized inverse, which can improve the convergence rate of SCFS compare with those directly given and remain fixed in Theorem 1.

Remark 4. According to the proof of Theorem 3, we can find that for any given ɛ > 0, if the rule number L satisfies L ≥ 2 ln ( ɛ ∥ Y ∥ ) / ln ( 1 - 1 N + 1 ) then the residual error norm ∥ e L * ( X ) ∥ satisfies ∥ e L * ( X ) ∥ ≤ ɛ . This means that any predetermined tolerance level can be reached when there are enough fuzzy rules, and training process of the proposed method is finite.

Theorem 1 and Theorem 3 have proved the convergence of two kind of parameters learning approach of SCFS. The two approach evaluate consequent parameters by constructing and optimizing schemes, respectively. Note that the performance of the latter is superior to that of the former. Hence, we only give the parameters learning algorithm associated with Theorem 3 here. Generally, the proposed algorithm is composed of the following components.

Configuration of premise parameters: Premise parameters c _L and σ _L are randomly assigned under the supervisory mechanism.

In the process of configuring the premise parameters, the value ranges of random parameters c _L and σ _L are designed as c L , j ∈ [ min i x ij , max i x ij ] , σ L , j ∈ [ 0 , K min k ∈ Q j ( x kj - x qj ) 2 ] , where Q j = { k ∈ { 1 , 2 , ⋯ , N } | ( x kj - x qj ) 2 ≠ 0 } , x q ∈ { x j | | e L - 1 ( x j ) | = max i | e L - 1 ( x i ) | } and K is a volatility multiplier. We randomly generate T pairs of parameters c _L , σ _L at a time from their respective ranges and choose pairs of parameters that satisfy the supervisory mechanism (9). Denote H L = ( ∑ i = 1 N τ L ( x i ) e L - 1 ( x i ) ) 2 ∑ i = 1 N τ L ( x i ) 2 ∑ i = 1 N e L - 1 ( x i ) 2 .(11)

The supervisory mechanism can be represented as H L ≥ r L(12) where r _L is set to 1 N + 1 . if no parameter satisfies the conditions (12), we will select a bigger K from a given set of positive numbers ϒ and repeat randomly generate T pairs of parameters c _L , σ _L until at least one satisfactory pair is found. If more than one satisfactory pair of parameters are found, we choose the pair with the largest value of H _L . Because according to the proof process of Theorem 3, the larger value of H _L means that the new rule R _L will bring more improvement to the approximation error.

In the process of evaluating the consequent parameters, the consequent parameters [ a 1 * , ⋯ , a L * ] are evaluated by solving the optimization problem using Moore-Penrose generalized inverse as a ( L ) * = arg min a ( L ) ∥ Y - η ( L ) a ( L ) ∥ 2 = ( η ( L ) T η ( L ) ) - 1 η ( L ) T Y where η ( L ) = ( η 1 T , ⋯ , η L T ) , a ( L ) = ( a 1 T , ⋯ , a L T ) T . The aim is to obtain the consequent parameters a^(L)* with the minimum learning errors. However, this can lead to overfitting, especially if η^{(L) T} η^(L) is ill-conditioned. Hence, we add a regular term λ ∥ a^(L) ∥ ² and evaluate the consequent parameter as a ( L ) + = arg min a ( L ) ( ∥ Y - η ( L ) a ( L ) ∥ 2 + λ ∥ a ( L ) ∥ 2 ) = ( η ( L ) T η ( L ) + λ I ) - 1 η ( L ) T Y(13) where λ is the regular coefficient.

Thus, the new fuzzy rule R _L is generated and added to old system S_L-1 form a new system S _L . the residual error of new system S _L is e L ( X ) = Y - η ( L ) a ( L ) +(14)

As that, the new fuzzy rules are generated gradually under a supervisory mechanism and added to the system until specified performance ɛ is met. In the parameters training algorithm, in order to avoid too long training process and facilitate the control of training process, we set a maximum rules number L _max .

The parameters training algorithm for SCFS is summarized as follows:

Parameters training algorithm for SCFS

Given dataset {X, Y}; Set the error tolerance ɛ, the maximum configuration times T, a set of positive numbers ϒ ={ K _min  : ΔK, : K _max  } and the maximum number of rules L _max .

1. Initial e = Y, L = 0, two null sets P and W.

2. While L < L _max and e > ɛ, Do

Stage 1: Premise parameters configuration

3. For K ∈ ϒ

4. For t = 1, ⋯ , T

5.   Randomly allocate c _L and σ _L ;

6.  Calculate H _L based on Equation (11);

7.  If H L ≥ 1 N + 1

8.    save c _L and σ _L in W, H _L in P;

9.  End If

10.   End For

11.   If W is not empty

12.  Find c L * , σ L * that maximize H _L in P;

13.  Break and go to step 16;

14.   End If

15. End For

Stage 2: Consequent parameters evaluation

16. Compute a^(L) based on Equation (13);

17. Compute e based on Equation (14);

18. End while

Return c _i , σ _i , a _i , i = 1, 2, ⋯ , L.

Thus, the proposed parameters training algorithm has completed the designing of SCFS.

Remark 5. In the parameter learning algorithm of SCFS, better premise parameters c _L , σ _L are selected from random parameters satisfying inequality constraints, and the consequent parameters are optimized, so the approximation performance of the model is better.

We investigate the performance of the proposed SCFS in this section. Compared with the proposed SCFS, MQ [25], ELM [26], IRVFLN [13], SCN [14], ISCN [27], CSSA-SCN [28], BLS [29], FBLS [30], HPFNN [31] and other models are performed. Root mean squares error (RMSE) is adopted to evaluate the prediction accuracy of the proposed SCFS and the selected comparison models. The statistic measure is denoted by RMSE = 1 N ∑ i = 1 N ( y i - y ˆ i ) 2 where N is the number of sample data, y _i is the actual value while y ˆ i is the predictive value. In the following experiments, every experiment is repeated 30 times and the mean and standard deviations of RMSE are reported.

6.1 Synthetic data (Syn_1D)

A one-dimensional nonlinear function f₁ (x) is considered. f 1 ( x ) = 0.2 e - ( 10 x - 4 ) 2 + 0.5 e - ( 80 x - 40 ) 2 + 0.3 e - ( 80 x - 20 ) 2 , x ∈ [ 0 , 1 ] .

As in [14], training data set consists of 1000 samples uniformly distributed in the interval [0,1], and testing dataset is 300 samples produced from an equally spaced grid on [0, 1].

The parameters are set as follows. Let the degree of Chebyshev polynomial n be 4, the regularization parameter λ be 1e - 10 and the collection of positive constants ϒ be{ 1e3 : 1e3 : 1e4 } in SCFS. Let the predefined maximum configuration times T be 100 in ISCN, SCN, CSSA-SCN and SCFS, the scope of random parameters be [- 1, 1] in IRVFLN and ELM, and the learning rate in MQ be 0.5.

Table 1 compares the training and testing RMSE of the proposed SCFS and other six models, in which L _max is maximum rules number or hidden nodes number (from the perspective of neural fuzzy network, each rule corresponds to a hidden node, so we will not distinguish hidden nodes and fuzzy rules below). From the comparison results in Table 1 where the bold items show the best results, it is obvious that the proposed SCFS has the best training performance and testing performance.

OPEN IN VIEWER

Table 1

RMSE Comparisons on Syn_1D

Dataset	Algorithms	Training RMSE				Testing RMSE
		L max =25		L max =50		L max =25		L max =50
		mean	std	mean	std	mean	std	mean	std
Syn_1D	MQ	0.1031	0.0001	0.103	0.0001	0.1011	0.0003	0.1011	0.0003
	ELM	0.0631	0.0001	0.0631	0.0001	0.0631	0.0001	0.063	0.0001
	IRVFL	0.163	0.0008	0.1626	0.0005	0.1622	0.0012	0.1617	0.0008
	SCN [14]	0.0332	0.0065	0.0097	0.0036	0.0308	0.006	0.01	0.0033
	ISCN [27]	0.0745	0.0049	0.034	0.0091	0.0773	0.0039	0.0367	0.0093
	CSSA-SCN [28]	0.0058	0.0025	0.0009	0.0003	0.0058	0.0025	0.0009	0.0003
	SCFS	8E-05	9E-05	1E-05	1.1E-05	1E-04	0.0001	4.2E-05	7.8E-05

Figure 2 shows the testing performance of the IRVFLN, SCN and SCFS. It can be found that the SCFS has distinctly better generalization ability compared against the IRVFLN and SCN.

OPEN IN VIEWER

Fig. 2

testing performance on Syn_1D when L _max  = 25.

Table 2 presents the result of efficiency comparisons. It can be found that IRVFL are failed in achieving the expected training tolerances while L _max is set as 50 and SCFS requires fewer fuzzy rules than SCN to achieve the expected training tolerances.

OPEN IN VIEWER

Table 2

Efficiency comparisons under the same error tolerance on Syn_1D

Algorithms	error tolerance = 0.05		error tolerance = 0.03
	Hidden nodes		Hidden nodes
	(mean,std)		(mean,std)
IRVFL	–	–	–	–
SCN	19.9	3.617	30.6	3.565
SCFS	2	0	2.5	0.527

The proposed SCFS uses the incremental learning approach to designing TSK fuzzy system. A common problem with incremental learning process is that it may be very long (maybe infinite if unlucky). In Section 4, we explain this problem from theoretically two aspects: the existence of random parameters and sufficient conditions to achieve a given precision. The supervision mechanism of SCFS ensures that the incremental learning process of SCFS is finite. The results in Table 2 also explain this problem from the perspective of simulation experiments.

Figure 3 shows the comparisons of convergence rates for testing data among the proposed SCFS, SCN and IRVFL. It can be found that SCFS has the best convergence performance. That is, SCFS can achieve satisfactory performance with less fuzzy rules.

OPEN IN VIEWER

Fig. 3

Testing RMSE with 50 additive nodes or rules.

6.2 Stock prices of ten aerospace companies (Stock)

The Stock dataset from the KEEL (http://keel.es/) dataset repository contains 950 daily stock prices of ten aerospace companies. 75% of the samples are randomly chosen as training dataset and the rest as testing dataset.

The parameters are set as follows. Let the degree of Chebyshev polynomial n be 3, the regularization parameter λ be 0.1 and the collection of positive constants ϒ be{ 1, 100, 1e4, 1e6, 1e8 } in SCFS. Let the maximum configuration times T be 100 in ISCN, SCN, CSSA-SCN and SCFS, the scope of random parameters (w, b) be [- 1, 1] in BLS andFBLS.

Table 3 compares the training and testing RMSE of the proposed SCFS and other five models on Stock. In Table 3, the bold items show the best RMSE. It is obvious that SCFS performs best in both training and testing performance. This shows that the proposed SCFS performs well not only on synthetic data, but also on real datasets

OPEN IN VIEWER

Table 3

RMSE Comparisons on Stock

Dataset	Algorithms	Training RMSE				Testing RMSE
		L max =25		L max =50		L max =25		L max =50
		Mean	std	mean	std	mean	std	mean	std
Stock	SCN [14]	0.0399	0.001	0.0309	0.0006	0.044	0.0013	0.0374	0.0015
	ISCN [27]	0.0355	0.0012	0.0271	0.0008	0.0471	0.0069	0.0414	0.0058
	CSSA-SCN [28]	0.0401	0.0011	0.0305	0.0006	0.0442	0.0016	0.0365	0.0016
	BLS	0.04862	0.0015	0.0361	0.00303	0.0505	0.00055	0.039379	0.00312
	FBLS	0.03552	0.0012	0.0278	0.0008	0.039	0.00265	0.032202	0.00131
	SCFS	0.01984	0.0001	0.0165	0.00031	0.0269	0.00169	0.025349	0.00102

Figure 4–7 shows the performance comparison between SCFS with different values of paraments n and λ. We can see that although a bigger degree of Chebyshev polynomial n and a smaller regularization parameter λ contribute to better training performance, but on testing performance it is not necessarily and it may occur overfitting. The selection of appropriate parameters can improve the performance of the model. Grid search for the parameters in SCFS and other models to get better performance is used in this paper.

OPEN IN VIEWER

Fig. 4

Training RMSE of SCFS with different n.

OPEN IN VIEWER

Fig. 5

Testing RMSE of SCFS with different n.

OPEN IN VIEWER

Fig. 6

Training RMSE of SCFS with different λ.

OPEN IN VIEWER

Fig. 7

Testing RMSE of SCFS with different λ.

This paper develops SCFS, a kind of adaptive incremental TSK fuzzy system based on stochastic configuration, to improve randomized methods for designing fuzzy system. The proposed SCFS determines the appropriate number of fuzzy rules in TSK fuzzy system by incremental learning approach, which starts from an initial system, and gradually adds randomly generated fuzzy rules to improve the system performance until the specified performance is achieved. The premise parameters of new adding fuzzy rules are randomly assigned under the supervisory mechanism to ensure the continuous improvement of system performance. The universal approximation property and the convergence of approximation error of SCFS have been proved theoretically in Section 4.

Hence, the proposed SCFS has the advantages of adaptive and fast modeling of randomized methods and can guarantee good approximation ability. We compare the performance of the proposed SCFS with the other 14 models on a series of synthetic data and benchmark datasets. The experimental results and statistical analyses show that the proposed SCFS is significantly outperform than other methods in terms of both approximation and generalization performance.

In the future, we can combine broad learning system with SCFS to further improve the approximation performance and training speed of the fuzzy system by referring to the characteristics of broad learning system, such as fast training and strong nonlinear representation ability. In addition, the improvement of the interpretability of fuzzy system and the integration of intelligence optimization algorithm [33–35] are also interesting.