Takagi-sugeno-kang fuzzy systems with dynamic rule weights

Abstract

Fuzzy rules are very important in Takagi-Sugeno-Kang (TSK) fuzzy systems as they not only provide a mapping mechanism for input patterns but also make fuzzy systems interpretable. Current works further introduce rule weights to restrict/strengthen fuzzy rules for more situations. However, most of the embedded rule weights in fuzzy rules are static. In other words, the rule weights keep unchanged once they have been determined by the learning algorithms. In practical applications, it is often expected that each fuzzy rule should deduce different confidence degrees with respect to different input patterns. In this paper, a new TSK fuzzy system is proposed, in which each fuzzy rule is empowered by an individual dynamic rule weight (DRW). DRW is basically a nonlinear function of the input pattern to reflect the confidence degree (acceptability) the fuzzy rule acting on the input pattern. Furthermore, for an input pattern, its “isolation” level can be measured by the aggregated DRW values of the fuzzy rules in the proposed fuzzy system. Specifically, the proposed fuzzy system can be used to identify outliers whose aggregated DRW values of all fuzzy rules are very small. In order to effectively embed DRW to each fuzzy rule, an analogous stacked structure consisting of a basic input-output unit and an augmenting unit is proposed. The stacked architecture is characterized by three features: (i) the augmented information from the augmenting unit can provide indirect pattern information for DRW learning; (ii) the predictive information from the augmenting unit can be differentiated from the interpretability of fuzzy rules; and (iii) the modeling performance can be improved by the stacked generalization principle which leverages the predictive information in the manifold of the input pattern space in the system approximation process. Experimental results on 16 real-life datasets demonstrate the approximation accuracy, interpretability and outlier detection ability of the proposed fuzzy system.

Keywords

TSK fuzzy systems dynamic rule weights stacked structure outlier detection

1 Introduction

TAKAGI-Sugeno-Kang (TSK) fuzzy systems are rule-based fuzzy systems, which have been successfully adopted in different practical applications such as process control, medical diagnosis, financial prediction, and image processing [1 –3] because of its strong approximation capability and good interpretability. Fuzzy rules are the souls and skeletons of TSK fuzzy systems because they not only provide a mapping mechanism for input features but also make fuzzy systems interpretable. As for classical TSK fuzzy systems (e.g., 0-order TSK fuzzy system, one-order TSK fuzzy system [4]), each fuzzy rule is treated fairly. To observe the effect of rule weights in fuzzy rule-based systems, some representative results [5 –9] have been reported in which rule weights are also called certainty factors or certainty grades. In [5], Nauck and Kruse analyzed the interpretation of rule weights both in the Mamdani-type fuzzy systems and the Sugeno-type fuzzy systems from two aspects, i.e., rule weights are applied to complete rules and rule weights are applied only to the then-parts of fuzzy rules. For the learning of rule weights, they revealed that it could be replaced by modifying the membership functions of the antecedent or consequent fuzzy sets. In [6], Ishibuchi and Nakashima used the rule weight to decide the size of the decision area of each fuzzy rule in the Sugeno-type fuzzy systems. Unlike the results reported in [5], they revealed that the learning of certainty grades could be achieved without modifying the membership functions such that the interpretability of fuzzy rules had not been compromised and the classification performance could also be improved. Later, Ishibuchi et al. [7] presented two heuristic algorithms for rule weight specification. In [8], Tsang et al. introduced global weights and local weights to fuzzy rules in order to enrich the knowledge representations and accordingly designed a fuzzy neural network-based learning algorithm to tune global and local weights. In [9], Chen and Huang firstly presented a weighting mechanism that features appearing in the if-parts were endowed with different weights and then employed genetic algorithms to adjust weights of features in order to estimate the null values in relational database systems. More references about rule weights can be found in [10 –13].

From the existing researches on rule weights, it can be seen that most of them concentrate on the idea of restricting/strengthening the corresponding fuzzy rules. In other words, once rule weights are determined by learning algorithms, they will be kept unchanged. However, in practice, rule weights could be much relaxed and judiciously exploited. Figure 1 illustrates a medical diagnostic system, where the knowledge of medical experts can be summarized into each fuzzy rule. The if-part of a fuzzy rule can be represented by symptom descriptions of a patient and the then-part can be represented by the diagnosis result for a patient. Suppose one expert has the knowledge that “If a patient is with photophobia, impaired vision, double images, and hyperglycemia, then the patient may be suffered from diabetic retinopathy (DR).” When the fuzzy rule is applied to a patient and a patient having a family history of diabetes, respectively, the confidence degrees (i.e. rule weights) of diagnosis as DR should be apparently different. The patient having a family history of diabetes may be suffered from DR with 70% confidence degree while the other may be suffered from DR with 10% confidence degree. In other words, fixing a stationary rule weight on each fuzzy rule is not sufficient to model the complex scenarios in the real world. Thus, a dynamic rule weight (DRW) should be exploited to reflect the confidence degree of each fuzzy rule with respect to each input pattern.

Fig.1

An instance of a medical diagnostic system.

Inspired by the practical requirements, this work proposes a new TSK fuzzy system termed as DRW-TSK-FS in which DRW is embedded in each fuzzy rule. DRW-TSK-FS is different from the existing TSK fuzzy systems with rule weights on the following aspects:

(i) In DRW-TSK-FS, the rule weight in each fuzzy rule is dynamic. As for different input patterns being judged by the same fuzzy rule, even all of them match with the if-part of the fuzzy rule, the rule weight of this fuzzy rule acting on each input pattern may be different. In other words, the rule weight can be considered as a function of the input pattern. To the contrary, the rule weight in most existing fuzzy systems is only interpreted as the strength of each fuzzy rule. The larger the rule weight is, the larger the decision area of the fuzzy rule is.

(ii) DRW learning is very simple. From the mathematical view, DRW can be expressed as the product of serval membership functions. Hence, clustering techniques (e.g., FCM [14]) can be employed to achieve DRW learning.

The contributions of this paper can be summarized as follows:

(i) A new TSK fuzzy system DRW-TSK-FS with dynamic rule weights is proposed. The structure of DRW-TSK-FS consists of a basic input-output unit and an augmenting unit. The augmenting unit is used to add predictive information to the original input pattern space that is taken as the new input of the basic input-output unit. Such a structure is similar to the stacked structure such that its performance can be insured by the stacked generalization principle because the predictive information can be leveraged to open the manifold of the feature space to improve the generalization capability. Although the new input pattern space contains predictive information fuzzy rules in the basic input-output unit have the predictive information as augmented inputs from the augmenting unit, they can be understood as those with only the original input patterns in the if-parts together with DRW with respect to each predictive information from the augmenting unit. Hence, the if-parts of all fuzzy rules in DRW-TSK-FS always have concise physical meanings.

(ii) DRW in each fuzzy rule is a nonlinear function of the input pattern which reflects the confidence degree (acceptability) of the fuzzy rule with respect to the input pattern. In addition, for an input pattern, its “isolation” level can be measured by the summed value of DRW of all fuzzy rules in DRW-TSK-FS. Specifically, DRW-TSK-FS can be used to identify outliers whose summed value of DRW is very small.

(iii) DRW-TSK-FS has been evaluated on 16 datasets and the experimental results have demonstrated its enhanced or at least comparable modeling performance as well as its ability in outlier detection.

The following sections are organized as follows. TSK fuzzy systems and their learning algorithms are briefly reviewed in Section II. In Section III, the structure of DRW-TSK-FS is given and its learning process is discussed. Section IV reports the experimental results. Section V concludes the whole paper and describes future works.

2 TSK fuzzy systems

In this section, the TSK fuzzy system and its learning algorithms are briefly reviewed since the TSK fuzzy system is the basic unit of DRW-TSK-FS.

2.1 Concept and principle

TSK fuzzy systems are the most frequently used fuzzy systems in which the kth fuzzy rule can be formulated as $\begin{matrix} If x_{1} is A_{1}^{k} \land x_{2} is A_{2}^{k} \land \dots \land x_{d} is A_{d}^{k}, \\ then f^{k} (x) = p_{0}^{k} + p_{1}^{k} x_{1} + . . . + p_{d}^{k} x_{d}, k = 1, 2, . . ., K . \end{matrix}$ (1)

In (1), $A_{i}^{k}$ is a fuzzy subset subscribed by the ith input pattern x_i for the kth rule, ∧ denotes a fuzzy conjunction operator and K is the number of fuzzy rules. Each fuzzy rule is premised on the pattern space where an input pattern is denoted as x = [x₁, x₂, . . . , x_d] ^T, and maps the fuzzy subspace $A^{k} = [A_{1}^{k}, A_{2}^{k}, . . ., A_{d}^{k}]$ in the input pattern space R^d (A^k ⊂ R^d) to a varying singleton denoted by f^k (x) which is generally a constant or a linear function in (1). The process of defuzzification can be worked out by a simple weighted summation. That is, the output of the TSK fuzzy system can be written as $y^{o} = \sum_{k = 1}^{K} (μ^{k} (x) / \sum_{m = 1}^{K} μ^{m} (x)) f^{k} (x) = \sum_{k = 1}^{K} {\tilde{μ}}^{k} (x) f^{k} (x),$ (2) where μ^k (x) and ${\tilde{μ}}^{k} (x)$ are the compatibility and the normalized compatibility of the input pattern x with the antecedent part A^k of the kth fuzzy rule, respectively, which can be formulated as: $μ^{k} (x) = \prod_{i = 1}^{d} μ_{A_{i}^{k}} (x_{i}) and {\tilde{μ}}^{k} (x) = μ^{k} (x) / \sum_{m = 1}^{K} μ^{m} (x) .$ (3)

Generally speaking, parameters involved in the if-parts and then-parts are usually learned separately. In the if-parts, the Gaussian membership function is adopted as the mapping function and a fuzzy set $A_{i}^{k}$ associating with the Gaussian membership function can be expressed as: $μ_{A_{i}^{k}} (x_{i}) = \exp (- 1 / 2 ((x_{i} - c_{i}^{k}) / (δ_{i}^{k}))^{2}) .$ (4)

In (4), $c^{k} = [c_{1}^{k}, c_{2}^{k}, . . ., c_{d}^{k}]$ and $δ^{k} = [δ_{1}^{k}, δ_{2}^{k}, . . ., δ_{d}^{k}]$ are two kinds of parameters that can be obtained by clustering techniques (e.g., FCM [14]), where the parameter vector $c^{k} = [c_{1}^{k}, c_{2}^{k}, . . ., c_{d}^{k}]$ is the center of the kth cluster and the kernel width vector $δ^{k} = [δ_{1}^{k}, δ_{2}^{k}, . . ., δ_{d}^{k}]$ can be calculated by the following form: $δ_{i}^{k} = h \sum_{j = 1}^{N} u_{jk} (x_{ji} - c_{i}^{k})^{2} / \sum_{j = 1}^{N} u_{jk}, i = 1, . . ., d, k = 1, . . ., K .$ (5)

In (5), u_jk represents the fuzzy membership degree of the object x_j belonging to the kth cluster by FCM clustering, h is a scale parameter that is manually tuned.

When the parameters in the if-parts are determined by clustering techniques, let $x_{e} = (1, x^{T})^{T},$ (6a) ${\tilde{x}}^{k} = {\tilde{μ}}^{k} (x) x_{e},$ (6b) $x_{g} = (({\tilde{x}}^{1})^{T}, ({\tilde{x}}^{2})^{T}, . . ., ({\tilde{x}}^{K})^{T})^{T},$ (6c) $p^{k} = (p_{0}^{k}, p_{1}^{k}, . . ., p_{d}^{k})^{T} and$ (6d) $p_{g} = ((p^{1})^{T}, (p^{2})^{T}, . . ., (p^{K})^{T})^{T},$ (6e)

so the solution to the output of the TSK fuzzy systems in (2) can also be rewritten as the following form $y^{o} = p_{g}^{T} x_{g},$ (7) which obviously is a linear regression problem [14]. Therefore, the learning of the TSK fuzzy system can be considered as the learning of the parameters in the equivalent linear regression model [15, 16].

2.2 Learning algorithms

With respect to the parameter learning in (7), there exist so many outstanding learning algorithms and criteria. In [17], the least square criterion function was introduced for the parameter learning of the then-parts. However, this criterion used in TSK fuzzy systems is not robust for modeling small datasets or noisy datasets [4]. In [18], another one, i.e., ɛ-insensitive criterion was proposed for the parameter learning of the then-parts. With an L1-norm penalty term, the parameter learning can be taken as a quadratic programming problem and the corresponding TSK fuzzy system is called L1-TSK-FC. Comparing with the least square criterion, L1-TSK-FC becomes more robust against small datasets and noisy datasets. Furthermore, in [4], the authors pointed out that an L2-norm penalty term can also be employed to develop ɛ-insensitive criterion-based algorithms (i.e., L2-TSK-FC) for the parameter learning of the then-parts. The insensitive parameter ɛ is involved in the objective function as a penalty term. Comparing with L1-TSK-FC, L2-TSK-FC is more superior because the insensitive parameter ɛ can be obtained automatically.

In (1), if $f^{k} (x) = p_{0}^{k}$ , the TSK fuzzy system is termed as the 0-order TSK fuzzy system. If $f^{k} (x) = p_{0}^{k} + p_{1}^{k} x_{1} + . . . . . . + p_{d}^{k} x_{d}$ , the TSK fuzzy system is called the one -order TSK fuzzy system.

3 DRW-TSK-FS: The proposed TSK fuzzy system with dynamic rule weights

In this section, detailed analyses about DRW-TSK-FS including its structure, fuzzy rules, and the involved dynamic rule weights are given. Then, the learning algorithm of DRW-TSK-FS is presented.

3.1 Structure of DRW-TSK-FS

The structure of DRW-TSK-FS is illustrated in Fig. 2 where x = [x₁, x₂, . . . , x_d] ^T is an input pattern denoting the regression problem with d input features. There exist two components in Fig. 2, i.e., a compulsory TSK fuzzy system taken as the basic input-output unit and an optional augmenting unit used to augment the original input pattern space. Note that the augmenting unit is also learnable until the performance and effectiveness of DRW-TSK-FS meet the requirements.

Fig.2

Structure of DRW-TSK-FS.

As for the augmenting unit, it can be realized by many regression models like SVR [19, 20] and neural networks (NN) [21 –23]. In this paper, the single layer neural network is employed to realize the augmenting unit. Hence, the augmenting unit is termed as the NN unit. As for the TSK fuzzy system in Fig. 2, the 0-order-TSK fuzzy system is adopted in order to keep the interpretability of DRW-TSK-FS. Its justification can be described as follows.

Obviously, when the NN unit involved in DRW-TSK-FS is silent, the TSK fuzzy system in Fig. 2 is a 0-order TSK fuzzy system with the kth rule expressed as

$\begin{matrix} If x_{1} is A_{1}^{k} \land x_{2} is A_{2}^{k} \land \dots \land x_{d} is A_{d}^{k}, \\ then f^{k} (x) = p_{0}^{k}, k = 1, 2, . . ., K, \end{matrix}$ (9) where notations used in (9) follow the same definitions in (1). When the NN unit is activated by users, the predictive information generated by the NN unit will be added to the original input pattern space iteratively and the augmented input pattern space will be taken as the new input of the basic input-output unit. The structure of DRW-TSK-FS is very similar to the stacked structure [24], and thus according to the stacked generalization principle, the predictive information generated by the NN unit can be leveraged to open the manifold of the input pattern space of the training set so as to improve the generalization capability. Besides, the most distinctive benefit exists in the fact that the basic input-output unit can be considered as another equivalent TSK fuzzy system where the predictive information of the NN unit is hidden in the dynamic rule weights (to be clearly elaborated later). With the engaged NN unit, the kth rule of DRW-TSK-FS can be expressed as

If x₁ is $A_{1}^{k} \land x_{2}$ is $A_{2}^{k} \land \dots \land x_{d}$ is

$\begin{matrix} A_{d}^{k} \land y_{1} is A_{y_{1}}^{k} \land \dots \land y_{t} is A_{y_{t}}^{k} \\ then f^{k} (x) = p_{0}^{k}, k = 1, 2, . . ., K, \end{matrix}$ (10)

where y_t is the tth output of the NN unit and $A_{y_{t}}^{k}$ is a fuzzy subset subscribed by y_t for the kth rule.

With the singleton fuzzifier, the product inference and the center average defuzzifier, the output of DRW-TSK-FS can be computed as $y^{0} = \sum_{k = 1}^{K} \frac{\prod_{i = 1}^{d} μ_{A_{i}^{k}} (x_{i}) \prod_{j = 1}^{t} μ_{A_{j}^{k}} (y_{j})}{\sum_{k^{'}}^{K} \prod_{i = 1}^{d} μ_{A_{i}^{k}} (x_{i}) \prod_{j = 1}^{t} μ_{A_{j}^{k}} (y_{j})} f^{k} (x) .$ (11)

From (10), it can be seen that the outputs generated by the NN unit involve in the if-parts of DRW-TSK-FS such that the interpretability of the fuzzy rule in (10) degenerates. This is because the outputs of the NN unit are not endowed with specific physical meanings. Despite the potential advantage that the approximation accuracy of DRW-TSK-FS can be insured by the stacked structure, DRW-TSK-FS becomes incomprehensible. Furthermore, the confidence degree that the fuzzy rule can act on an input pattern cannot be properly modeled. In the following, let discuss another form of DRW-TSK-FS to address the aforementioned issues.

As pointed out in [25, 26], when the Gaussian membership function is employed as the corresponding fuzzy membership function, the activation degrees cannot be normalized. Hence, without taking the denominator in (11) into considerations, the output of DRW-TSK-FC can be expressed as $y^{0} = \sum_{k = 1}^{K} \prod_{i = 1}^{d} μ_{A_{i}^{k}} (x_{i}) \prod_{j = 1}^{t} μ_{A_{j}^{k}} (y_{j} (x)) f^{k} (x) .$ (12)

It is obvious that the output formulated in (12) can be expressed in another form, $y^{0} = \sum_{k = 1}^{K} (\prod_{i = 1}^{d} μ_{A_{i}^{k}} (x_{i}) f^{k} (x)) \prod_{j = 1}^{t} μ_{A_{j}^{k}} (y_{j} (x)),$ (13)

which indicates that there exists another form of DRW-TSK-FS whose output is as same as that in (13). Accordingly, the kth fuzzy rule of the new form of DRW-TSK-FS is defined as

If x₁ is $A_{1}^{k} \land x_{2}$ is $A_{2}^{k} \land \dots \land x_{d}$ is $A_{d}^{k}$ ,

$\begin{matrix} then f^{k} (x) = p_{0}^{k}, \\ with DRW = \prod_{j = 1}^{t} μ_{A_{j}^{k}} (y_{j} (x)), k = 1, 2, . . ., K . \end{matrix}$ (14)

With regards to the if-parts in (14), it is clear that except the original input patterns, there are no other patterns being involved and thus the interpretability keeps compared to the complicated fuzzy rule shown in (10). With respect to the predictive information, a new dynamic concept termed as the dynamic rule weight DRW is defined for each fuzzy rule. Thus, the predictive information from the NN unit now does not complicate the interpretation of the fuzzy rules in DRW-TSK-FS because it can be hidden in DRW as shown in (14).

Please note, as a new concept, DRW here is a function of the input pattern x, which means that when the fuzzy rule in (14) is applied to the input pattern x, its rule weight does not keep unchanged. That is to say, the if-part of each fuzzy rule in (14) reflects the matching degree that the fuzzy rule can be applied to x, and DRW reflects the acceptability of each fuzzy rule for x or the confidence degree that the fuzzy rule can act on x. The smaller the value of DRW, the lower acceptability (confidence degree) the fuzzy rule for x. To the best of our knowledge, the rule weights of fuzzy rules are always assumed to be unchanged in literatures [5 –9]. However, such an assumption does not always hold true in practical application scenarios, see the example (Fig. 1) illustrated in the first section. In addition, for x, if the aggregated value of acceptability of all fuzzy rules is very low, it means that x is not accepted by most fuzzy rules. Therefore, x can be considered as an isolated object or an outlier, and not being accepted by DRW-TSK-FS. In other words, $\sum_{k = 1}^{K} \prod_{j = 1}^{t} μ_{A_{j}^{k}} (y_{j} (x))$ can be employed to reflect the isolation level of x. Conversely, by observing the isolation level of each object, the abnormal patterns or outliers can be identified.

3.2 Learning algorithm

In DRW-TSK-FS, with regards to the parameter learning of the if-parts, FCM is employed to obtain the center vectors and the kernel width vectors for fuzzy membership functions. After all the if-parts of fuzzy rules are determined, one may obtain $p_{g} = (p_{0}^{1}, p_{0}^{2}, . . ., p_{0}^{K})^{T} and$ (15a) $\begin{matrix} w_{g} = (\prod_{i = 1}^{d} μ_{A_{i}^{1}} (x_{i}) \prod_{j = 1}^{t} μ_{A_{y_{j}}^{1}} (y_{j} (x)), . . ., \prod_{i = 1}^{d} μ_{A_{i}^{K}} \\ (x_{i}) \prod_{j = 1}^{t} μ_{A_{y_{j}}^{K}} (y_{j} (x))), \end{matrix}$ (15b) where t in (15b) represents the number of outputs generated by the NN unit. Consequently, the output of DRW-TSK-FS can be rewritten as the following form $y^{o} = p_{g}^{T} w_{g} .$ (16)

Hence, the training problem of DRW-TSK-FS can be transformed into the learning of parameters involved in the linear regression model expressed in (16). Based on the least square (LS) criterion, the solution for the parameter learning of the regression model in (16) can be derived by minimizing the following LS criterion function $\begin{array}{l} \min p_{g} E = \frac{1}{2} \sum_{i = 1}^{N} (y_{i}^{o} - y_{i}^{real})^{2} \\ = \frac{1}{2} (y^{real} - W_{g} p_{g})^{T} (y^{_{real}} - W_{g} p_{g}) \end{array}$ (17) where W_g = [w_g1, w_g2, . . . , w_gN] ^T ∈ R^N×K, $y^{_{real}} = {[y_{1}^{real}, y_{2}^{real}, ..., y_{N}^{real}]}^{T} \in R^{N}$ is a vector where each element denotes a real output corresponding to a training object. By introducing the gradient descent approach, the parameters in the regression model are updated as $\begin{array}{l} p_{g} (i t e r + 1) = p_{g} (i t e r) - η \frac{\partial E}{\partial p_{g} (i t e r)} \\ = p_{g} (i t e r) - η [(- W_{g})^{T} (y^{real} - W_{g} p_{g} (i t e r))] \end{array}$ (18)

where η is the learning rate given by users, iter represents the current moment. The learning algorithm of DRW-TSK-FS is listed in Algorithm 1.

Algorithm 1:
Input: The training set X = [x₁, x₂, . . . , x_N] ^T and its corresponding output set $y^{real} = [y_{1}^{real}, y_{2}^{real}, . . ., y_{N}^{real}]^{T} \in R^{N}$ , where x_i ∈ R^d, $y_{i}^{real} \in R$ , i = 1, 2, . . . . . . , N, d represents the number of features in one object. The number of outputs generated by the NN unit t. The number of fuzzy rules K. The threshold ξ and the maximum iteration times IterMax.
Output: Parameters and DRW of fuzzy rules.
Procedure:
Step 1: Activate the NN unit on the training set and generate t groups of predictive information $[y_{1}^{j} (x_{1}), y_{2}^{j} (x_{2}), . . ., y_{N}^{j}$ (x_N)] ^T, j = 1, 2, . . . , t, then add them to the original feature space and obtain $\tilde{X} = [{\tilde{x}}_{1}, {\tilde{x}}_{2}, . . ., {\tilde{x}}_{N}]^{T}$ , where ${\tilde{x}}_{i} = [x_{i 1}, x_{i 2}, . . ., x_{id}, y_{i}^{1}, y_{i}^{2}, . . ., y_{i}^{t}], i = 1, 2, . . ., N$ .
Step 2: Employ FCM to cluster $\tilde{X}$ and obtain the center vector
c^k and the kernel width vector δ^k for each fuzzy rule.
Step 3: Compute the value of the if-part of a fuzzy rule by
$W (n, k) = \prod_{i = 1}^{d} μ_{A_{i}^{k}} (x_{ni}) \prod_{j = 1}^{t} μ_{A_{y_{j}}^{k}} (y_{nj} (x_{n}))$ ,
where n = 1,2, . . . ,N, k = 1,2, . . . .,K and $DRW = \prod_{j = 1}^{t} μ_{A_{y_{j}}^{k}}$
(y_nj (x_n)).
Step 4: Learn the parameters in the then-part of each fuzzy rule
by using the gradient descent approach, i.e., to carry out
the following steps.
Step 4(a) Initialize the parameter matrix p_g in (18) by
randomly assigning a value between – 1 and 1
to its every element and iter = 1.
Repeat
Step 4(b) Use (18) to compute p_g (iter + 1).
Step 4(c) Set iter=iter+1.
Until E (iter) - E (iter - 1) ⩽ ξ or iter > IterMax
Step 5: Return the parameters and DRW of each fuzzy rule and
accordingly DRW-TSK-FS.

In algorithm 1, t can exert influence on the approximation accuracy of DRW-TSK-FS. Our extensive experimental results in the following section reveal that t is set in a relatively reasonable range from d/3 to d/2 can meet our requirements.

3.3 Time complexity and interpretability

In this subsection, the time complexity of Algorithm 1 will be firstly analyzed, then the interpretability of DRW-TSK-FS will be discussed quantificationally.

The time complexity of DRW-TSK-FS contains four parts, i.e., generating t groups of predictive information, learning parameters involved in the if-parts of fuzzy rules, computing the value of the if-parts of fuzzy rules and learning parameters involved in the then-parts of fuzzy rules. The time complexity of the first part depends on the training approach for NN. For a given training approach, e.g., back-propagation [27], the time complexity is O (THN²), where H is the number of nodes in the hidden layer. The second part is achieved by FCM [14], thus the time complexity is O (NK (d + t) ²). The time complexity of the third part is O (NK). Lastly, with regards to the parameter learning of the then-parts, the upper bound of the time complexity is O (IterMaxNK²).

Therefore, the entire asymptotic time complexity of DRW-TSK-FS is O (N (THN + K (1 + (d + t) ²) + IterMaxNK²)). Obviously, with the augmenting unit (NN), the time complexity of DRW-TSK-FS is at least near to the magnitude of O (N²), which is higher than that of the classic TSK fuzzy system, e.g., 0-order TSK fuzzy system. However, we should keep in mind that the predictive information generated by the augmenting unit (NN) can be leveraged to open the manifold of the feature space to improve the generalization capability of DRW-TSK-FS. Also, with regards to the implementation of NN in Algorithm 1, the neural network toolbox provided by the Matlab environment is employed [40], which is effective and high-efficiency for many application scenarios.

As stated in [37 –39], the interpretability of the TSK fuzzy system can be evaluated by both rule-based level (measured by the number of fuzzy rules) and fuzzy partition level (measured by the number of features and membership functions). In other words, the interpretability of DRW-TSK-FS can be quantificationally evaluated by the number of involved parameters during the training procedure. Therefore, with regards to DRW-TSK-FS with the fuzzy rule shown in (14), it needs 2Kd parameters in the if-parts and K parameters in the then-parts. Thus, the total number of parameters is K (2d + 1). However, if the fuzzy rule in (10) is adopted, the number of parameters involved in the if-parts is 2K (d + t), and hence the total number of parameters is K (2d + 1) +2Kt. That is to say, by introducing DRW, the interpretability is enhanced since the predictive information is hidden in DRW.

4 Experimental results

In this section, DRW-TSK-FS is evaluated from the perspective of its approximation ability and outlier detection performance. The Gas Furnace dataset [28] and 15 KEEL regression datasets [29] are respectively chosen for this evaluation. Five benchmarking regression models, 0-order-TSK-FS [4], L2-TSK-FS [4], FS-FCSVM [30], TSFS-SVR [31] and a single hidden layer neural network (NN) are employed for comparison. The detailed experiments are organized as follows. In subsection IV.A, experimental setups including the parameter setting and the experimental environment configuration are given. The experimental results on the Gas Furnace dataset and KEEL datasets are respectively reported in subsections IV.B and IV.C. In subsection IV.D, the advantages of DRW-TSK-FS are discussed. Lastly, two statistical approaches are employed to analyze the significant differences between DRW-TSK-FS and benchmarking regression models.

4.1 Experimental setup

In our experiments, parameters in all benchmarking regression models and DRW-TSK-FS are determined by 5-fold cross-validation combining with the grid searching strategy on training datasets. In 0-order-TSK-FS, L2-TSK-FS, FS-FCSVM and DRW-TSK-FS, the number of fuzzy rules is set in a relatively small range from 2 to 15 [32, 33] in order to consider the balance between the approximation ability and interpretability of each model. In addition, for NN, the number of hidden nodes is also searched from 2 to 15. Table 1 shows the parameter search in the corresponding models.

Table 1
Parameter setup (search) of different models

L2-TSK-FS, 0-order-TSK-FS, DRW-TSK-FS Number of fuzzy rules K ∈ {2, 3, . . . , 15}, fuzzy index m ∈ {1.1, 1.5, 2, 2.5, 3}, scale parameter h ∈ {0 . 2², 0 . 4², 0 . 8², 1⁰, 1 . 6², 2²}. Number of fuzzy rules K ∈ {2, 3, . . . , 15}.

FS-FCSVM Learning threshold parameter σ ∈ {0 . 2, 0 . 3, . . . , 0 . 8}, regularization parameter C ∈ {2^-3, 2^-1, 2⁰, . . . , 2⁵, 2⁷}, number of rules K ∈ {2, 3, . . . , 15}.

NN Number of hidden nodes H ∈ {2, 3, . . . , 15}.

TSFS-SVR Insensitive parameter ɛ ∈ {0 . 01, 0 . 02, . . . , 0 . 2}, regularization parameter C ∈ {10^-3, 10^-1, 10⁰, . . . , 10⁵, 10⁷}, number of fuzzy rules K ∈ {2, 3, . . . , 15}.

L2-TSK-FS, 0-order-TSK-FS, DRW-TSK-FS	Number of fuzzy rules K ∈ {2, 3, . . . , 15}, fuzzy index m ∈ {1.1, 1.5, 2, 2.5, 3}, scale parameter h ∈ {0 . 2², 0 . 4², 0 . 8², 1⁰, 1 . 6², 2²}. Number of fuzzy rules K ∈ {2, 3, . . . , 15}.
FS-FCSVM	Learning threshold parameter σ ∈ {0 . 2, 0 . 3, . . . , 0 . 8}, regularization parameter C ∈ {2^-3, 2^-1, 2⁰, . . . , 2⁵, 2⁷}, number of rules K ∈ {2, 3, . . . , 15}.
NN	Number of hidden nodes H ∈ {2, 3, . . . , 15}.
TSFS-SVR	Insensitive parameter ɛ ∈ {0 . 01, 0 . 02, . . . , 0 . 2}, regularization parameter C ∈ {10^-3, 10^-1, 10⁰, . . . , 10⁵, 10⁷}, number of fuzzy rules K ∈ {2, 3, . . . , 15}.

To evaluate the performance (model error) quantitatively, the performance measure also used in [28] is hired: $Err = \frac{\sum_{i = 1}^{N} (y_{i}^{real} - y_{i}^{predicting})^{2}}{N \times (\max (y^{real}) - \min (y^{real}))^{2}},$ (19)

where N is the size of the training set, and y^real and y^predicting denote the real output and predicting output, respectively. The smaller the value of Err, the better the modeling performance of the corresponding regression model.

In order to test the statistical significance between DRW-TSK-FS and the benchmarking regression models, two non-parametric statistical tests [34] are introduced in our experiments. More specifically, the Friedman ranking test [35] is used to detect statistical differences among the results generated by different models and the Holm post-hoc test [36] is used to find the models that reject the equality hypothesis with respect to a chosen control model. The two non-parametric statistical tests are both provided by the KEEL toolbox.

All adopted models are implemented in the Matlab environment and all the experiments are conducted on a PC with 4 cores of I5-7200U with 8 G Bytes of memory.

4.2 On the gas furnace dataset

To evaluate the performance of DRW-TSK-FS, the Gas Furnace dataset downloaded from http://openmv.net/info/gas-furnace is firstly employed. The dataset is a time series, containing successive pairs of 296 observations of the gas rate X (b) and the percentage CO₂ in the gas Y (b) generated from continuous records at 9 s intervals.

In the experiments, X (b), X (b) - X (b - 1), Y (b - 1) and Y (b - 1) - Y (b - 2) are taken as the input features affecting the current output Y (b). the first 250 vectors of the form [X (b) , X (b) - X (b - 1) , Y (b - 1) , Y (b - 1) - Y (b - 2)] are selected to train DRW-TSK-FS and the rest is used for model testing.

Suppose that we use t to represent the number of activations of the NN unit, i.e., the number of predictive information augmented to the original feature space of the training set. To observe the influence of t exerted on the model error of DRW-TSK-FS, t is first fixed from 0 to 5 respectively and then DRW-TSK-FS is carried out with each t for 30 trials. Figure 3 illustrates the average model error Err with different t.

Fig.3

Modeling error Err of DRW-TSK-FS with different t.

When t is set to 0, the NN unit is not activated. Hence, DRW-TSK-FS degenerates to a 0-order-TSK-FS, which obtains the modeling error Err as high as 4.221*1e-3. When the NN unit is activated with a non-zero t, the average Err is reduced significantly and reaches to its minimum when t = 2. However, as the value of t continues to grow, the modeling error Err does not further decrease, but increases a bit and then stays stable. This experimental result indicates that augmenting moderate predictive information from the NN unit to the original feature space can indeed reduce the modeling error. However, excessive predictive information may overwhelm the original features and even increases the model error. In addition, excessive predictive information means that a higher time complexity of DRW-TSK-FS. In fact, experimental results (including the following results on KEEL datasets) reveal a guideline that t can be set in a relatively reasonable range from d/3 to d/2, where d is the number of the features in the original feature space.

Table 2 lists the comparative results of the adopted benchmarking models and DRW-TSK-FS in terms of the minimum (Min) and average (Mean) modeling errors, the standard deviation (Std) and the optimal number of fuzzy rules/hidden nodes. The best average modeling errors are marked in bold. Please note that, for NN, the number of hidden nodes instead of the number of fuzzy rules is reported. From Table 2, we can see that comparing with other benchmarking models such as 0-order-TSK-FS, L2-TSK-FS, FS-FCSVM, NN and TSFS-SVR, DRW-TSK-FS wins the best performance in terms of minimum and average modeling errors.

Table 2

The performance of different regression models on the Gas Furnace dataset

Model	Min	Mean	Std	Rules/Hidden Nodes
L2-TSK-FS	6.756*1e-3	6.772*1e-3	0.011*1e-3	11
0-order-TSK-FS	3.946*1e-3	4.221*1e-3	0.226*1e-3	12
FS-FCSVM	7.052*1e-3	7.060*1e-3	0.006*1e-3	13
TSFS-SVR	3.993*1e-3	4.289*1e-3	0.221*1e-3	8
NN	6.737*1e-3	6.764*1e-3	0.021*1e-3	5
DRW-TSK-FS	3.425*1e-3	3.600*1e-3	0.167*1e-3	12

In addition to approximation performance, the interpretability of a fuzzy system is also very important. To observe the interpretability of DRW-TSK-FS more concretely, Table 3 shows 3 interpretable fuzzy rules with t being set to 1 and 2, respectively. It is obvious that the predictive information from the NN unit is not involved in the if-parts and does not complicate the interpretation of the then-parts of the fuzzy rules because it is hidden in DRW of these rules. Each row in Table 3 can be translated into a fuzzy rule with a linear then-part with DRW. For example, when t is set to 1, the first fuzzy rule can be expressed as

If x₁ is $A_{1}^{k} (0.2453, 0.5816) \land x_{2}$ is $A_{2}^{k} (57.2119, 3.0940) \land x_{3}$ is $A_{3}^{k} (0.0047, 0.0947) \land x_{4}$ is $A_{4}^{k} (- 0.6640, 0.6214)$ ,

then f^k (x) =0.0979, with $DRW = e^{- \frac{1}{2} {(\frac{y_{1} (x) - 57.1895}{1 . 4991})}^{2}}$ .

Table 3

Rules obtained from the Gas Furnace dataset

The kth fuzzy rule: If x₁ is $A_{1}^{k} (c_{1}^{k}, δ_{1}^{k}) \land x_{2}$ is $A_{2}^{k} (c_{2}^{k}, δ_{2}^{k}) \land \dots \land x_{d}$ is $A_{d}^{k} (c_{d}^{k}, δ_{d}^{k})$ , then $f^{k} (x) = p_{0}^{k}$ , with $DRW = \prod_{j = 1}^{t} μ_{A_{j}^{k}} (y_{j} (x))$ .
T	No. of rules	If-parts parameters: $c^{k} = [c_{1}^{k}, c_{2}^{k}, . . ., c_{d}^{k}], δ^{k} = [δ_{1}^{k}, δ_{2}^{k}, . . ., δ_{d}^{k}]$	Then-parts parameters: $p^{k} = [p_{0}^{k}]$	y (x)	DRW
1	1	c¹ = [0.2453, 57.2119, 0.0047, - 0.6640] δ¹ = [0.5816, 3.0940, 0.0947, 0.6214]	p¹ = [0 . 0979]	y₁ (x) =53 . 5414	$DRW = e^{- \frac{1}{2} {(\frac{y_{1} (x) - 57.1895}{1 . 4991})}^{2}}$
	2	c² = [0.2120, 58.4629, 0.1601, - 0.8206] δ² = [0.5680, 4.0758, 0.0708, 0.5277]	p² = [0 . 2383]		$DRW = e^{- \frac{1}{2} {(\frac{y_{1} (x) - 58.4185}{1.9355})}^{2}}$
	3	c³ = [-0.2535, 56.0091, 0.1901, 0.1364] δ³ = [0.5194, 2.9508, 0.1007, 0.8722]	p³ = [0 . 1023]		$DRW = e^{- \frac{1}{2} {(\frac{y_{1} (x) - 55.6112}{1.3235})}^{2}}$
2	1	c¹ = [- 0 . 1197, 50 . 3758, - 0 . 2379, - 0 . 2245] δ¹ = [0.3593, 2.2202, 0.1230, 1.1963]	p¹ = [0 . 1679]	y₁ (x) =53.5443 y₂ (x) =53 . 5440	$DRW = e^{- \frac{1}{2} ({(\frac{y_{1} (x) - 50.4254}{1.1265})}^{2} + {(\frac{y_{2} (x) - 50.4148}{1.1324})}^{2})}$
	2	c² = [0.6752, 56.0291, 0.2181, - 0.5909] δ² = [0.7383, 2.6825, 0.0895, 0.8098]	p² = [0.1241]		$DRW = e^{- \frac{1}{2} ({(\frac{y_{1} (x) - 56.4065}{1.3248})}^{2} + {(\frac{y_{2} (x) - 56.4155}{1.3267})}^{2})}$
	3	c³ = [-0.3406, 55.3126, - 0.1074, - 1.4833] δ³ = [0.6075, 2.7491, 0.1108, 1.3543]	p³ = [0.0405]		$DRW = e^{- \frac{1}{2} ({(\frac{y_{1} (x) - 55.0985}{1.2048})}^{2} + {(\frac{y_{2} (x) - 55.1248}{1.2163})}^{2})}$

In practice, data collected from sensors may be contaminated at one point. How to detect the abnormal data (outliers) is also very important. In DRW-TSK-FS, DRW reflects the acceptability of each fuzzy rule for x. The smaller value of DRW indicates the lower acceptability. Therefore, for an arbitrary x, the sum value of DRW of all fuzzy rule is used to denote its “isolation” level.

To investigate DRW-TSK-FS’s ability to detect outliers, we simulate abnormal data (outliers) by randomly selecting 3 vectors (objects) from the testing set and deviating them from their original distributions by adding 20% noises. For an arbitrary object i in the testing set, its isolation level, denoted as DRW(i), can be computed as the sum value of DRW of all fuzzy rules. Figure 4(a) and 4(b) illustrate the isolation level of each object in the testing set with and without noises respectively. Note that in order to magnify the differences of isolation levels, the logarithmic form of the isolation level of each object is plotted.

Fig.4

Isolation level of each object. (a) Testing set without noises. (b) Testing set with noises.

Figure 4 shows that there exist three objects (or at least two objects) whose isolation levels are significantly different from those of the others. We check that the three objects are exactly the ones which we randomly select from the testing set and add noises.

Next, more datasets will be selected to demonstrate the characteristics involved in DRW-TSK-FS.

4.3 On KEEL datasets

15 real-world regression datasets are selected from the KEEL project repository to further demonstrate the characteristics of DRW-TSK-FS. Table 4 lists the detailed information (i.e., number of features, number of objects) of the selected datasets.

Table 5 gives the experimental results of DRW-TSK-FS and the benchmarking models in terms of minimum and average model errors, the standard deviation and the optimal number of fuzzy rules/hidden nodes. Except for BAS, WIZ, MAC and DEL, DRW-TSK-FS obtains the lowest modeling errors among all benchmarking models. The results further demonstrate the promising performance of DRW-TSK-FS. Also, 10 datasets are selected and the relationships between the modeling errors and t are plotted in Fig. 5.

Fig.5

Modeling error of DRW-TSK-FC with different t on KEEL datasets.

As discussed in subsection IV.B, an appropriate t can make DRW-TSK-FS keep a balance between the modeling error and the time complexity. From Fig. 5, we find that t set to 4, 2, 3, 6, 3, 3, 3, 2, 1 and 2 on STO (d = 9), FRI (d = 5), CON (d = 8), MOR (d = 11), LAS (d = 4), WAN (d = 9), AUT (d = 7), ELE (d = 4), QUA (d = 3) and DIA (d = 2) respectively can basically meet our requirements. Also, we find that the adopted t values on these 10 datasets basically locate in the range from d/3 to d/2. The results further demonstrate the effectiveness of the guideline revealed in subsection IV.B.

Similarly, we select 4 datasets (MOR, STO, AUT and ELE) and simulate the outliers by randomly selecting 10 vectors from the testing dataset and adding 20% noises to them in order to further investigate DRW-TSK-FS’s ability in outlier detection. The experimental results are plotted in Fig. 6.

Fig.6

Isolation level of each object on MOR, STO, AUT and ELE.

From Fig. 6, we find that DRW-TSK-FS detects 9 and 10 outliers respectively on MOR and STO based on their isolation levels. However, for AUT and ELE, although the intuitional results are inferior to that on MOR and STO, we can use a threshold to select the outliers dynamically. For example, in ELE, we can consider the objects as outliers when their values of log(DRW(i)) are less than – 1.5 or – 2.0 according to the specific application scenarios.

5 Discussion

The experimental results on all selected datasets demonstrate that with the predictive information from the NN unit, the approximation accuracy of DRW-TSK-FS is enhanced. Also, by introducing DRW, the interpretability of DRW-TSK-FS still keeps. Moreover, the sum value of DRW of all fuzzy rules can help us detect outliers. Overall, based on the experimental results, the main advantages of DRW-TSK-FS over other benchmarking models are highlighted as follows. (i) The structure of DRW-TSK-FS is significantly different from other benchmarking models. It is very similar to the stacked structure such that the predictive information can be leveraged to open the manifold of the training set so as to enhance the approximation accuracy of DRW-TSK-FS. For instance, on the gas furnace dataset, DRW-TSK-FS wins the best performance, and on 15 KEEL regression datasets, DRW-TSK-FS wins the best performance on 11 ones. (ii) Although the original feature space is augmented by the predictive information from the NN unit, with an equivalent transformation, for each fuzzy rule, the predictive information is removed from the if-parts and hidden in DRW such that the interpretability of DRW-TSK-FS keeps. From the fuzzy rules listed in Table 3, it is obvious that the if-parts do not contains the predictive information whose physical meanings are not explicit. (iii) With the introduced DRW, another advantage over other benchmarking models is that the sum value of DRW all fuzzy rules reflects the isolation level of each object. As for outliers, compared with normal objects, their summed DRW values are very small. In other words, the outliers should have very low isolation level in DRW-TSK-FS and they can be probably detected by their summed DRW values with respect to all fuzzy rules.

With the above advantages, DRW-TSK-FS is expected to apply to practical engineering applications. Here, two guidelines are given from the aspects of approximation accuracy and outlier detection based on the above experimental results. (i) The number of predictive information generated by the NN unit should be set in a relatively reasonable range from d/3 to d/2. (ii) As for abnormal data (outliers), firstly, it is strongly suggested to give a decision graph about log(DRW) of each object. Then according to the graph, it is recommended to use a threshold to select the abnormal data dynamically based on the application requirements.

5.1 Non-parametric statistical analysis

In this section, we employ two non-parametric statistical methods, i.e., the Friedman ranking test and the Holm post-hoc test to analyze the statistical significance of the performance of the proposed DRW-TSK-FS model as compared with the introduced benchmarking models. The level of confidence α is set to 0.05.

The Friedman ranking test is firstly employed and the ranking performance is illustrated in Fig. 7, which indicates that DRW-TSK-FS achieves the best ranking performance among all the benchmarking models. The p-value in Fig. 7 is much smaller than the specified α value, illustrating the statistical significance of the performance DRW-TSK-FS as compared with the benchmarking models.

Fig.7

Friedman ranking test.

Next, we also use the Holm post-hoc test to analyze the statistical significance when benchmarking DRW-TSK-FS with the other models. The results are recorded in Tables 6. All models are ordered by the z-value computed during the test procedure. Holm’s procedure rejects those hypotheses having an unadjusted p-value smaller than the Holm (α/i). From Tables 6, we can see that with respect to 0-order-TSK-FS, though the hypothesis is not rejected, their correspondingly very low p-values still reveal the competitiveness of DRW-TSK-FS.

Table 4

Main characteristics of 15 KEEL datasets

Datasets	Number of features	Number of objects
Stock (STO)	9	950
Friedman (FRI)	5	1200
Concrete (CON)	8	1030
Baseball (BAS)	16	337
Mortgage (MOR)	16	1049
Laser (LAS)	4	993
Wankara (WAN)	10	1609
Wizmir (WIZ)	9	1461
AutoMPG8 (AUT)	5	392
Ele2 (ELE)	4	1056
Quake (QUA)	3	2178
machineCPU (MAC)	6	209
Balance (DIA)	2	43
ANACALT (ANA)	7	4052
Delta_ail (DEL)	5	7129

Table 5

The modeling performance in terms of the maximum accuracy, the average number of rules, the average accuracy and the standard deviation on 15 datasets

Datasets	L2-TSK-FC		0-order-TSK-FC		FS-FCSVM		TSFS-SVR		NN		DRW-TSK-FC
	Min Mean	Rules Std	Min Mean	Rules Std	Min Mean	Rules Std	Min Mean	Rules Std	Min Mean	Rules Std	Min Mean	Rules Std
STO	3.523*1e-3	15	1.026*1e-3	11	5.568*1e-3	15	1.345*1e-3	12	3.538*1e-3	12	0.702*1e-3	11
	4.254*1e-3	0.827*1e-3	1.045*1e-3	0.019*1e-3	5.645*1e-3	0.105*1e-3	1.433*1e-3	0.137*1e-3	4.177*1e-3	0.454*1e-3	0.741*1e-3	0.048*1e-3
FRI	2.696*1e-2	3	1.014*1e-2	13	3.507*1e-2	14	1.063*1e-2	13	3.851*1e-2	7	0.880*1e-2	10
	3.041*1e-2	0.290*1e-2	1.024*1e-2	0.006*1e-2	3.519*1e-2	0.009*1e-2	1.078*1e-2	0.014*1e-2	4.058*1e-2	0.231*1e-2	0.913*1e-2	0.014*1e-2
CON	2.516*1e-2	14	1.042*1e-2	15	3.755*1e-2	14	1.307*1e-2	9	2.854*1e-2	10	0.911*1e-2	15
	2.706*1e-2	0.133*1e-2	1.084*1e-2	0.045*1e-2	3.912*1e-2	0.118*1e-2	1.360*1e-2	0.032*1e-2	2.917*1e-2	0.070*1e-2	1.076*1e-2	0.058*1e-2
BAS	2.776*1e-2	12	2.418*1e-2	12	3.424*1e-2	11	3.731*1e-2	11	5.454*1e-2	10	3.740*1e-2	10
	2.870*1e-2	0.114*1e-2	2.427*1e-2	0.014*1e-2	3.564*1e-2	0.134*1e-2	3.855*1e-2	0.117*1e-2	5.463*1e-2	0.011*1e-2	4.120*1e-2	0.140*1e-2
MOR	1.150*1e-4	9	1.820*1e-4	15	1.840*1e-4	9	1.480*1e-4	15	1.668*1e-4	11	0.537*1e-4	15
	1.616*1e-4	0.430*1e-4	2.090*1e-4	0.380*1e-4	2.370*1e-4	0.640*1e-4	2.210*1e-4	0.570*1e-4	2.921*1e-4	1.230*1e-4	0.555*1e-4	0.091*1e-4
LAS	2.187*1e-3	10	2.550*1e-3	15	3.646*1e-3	11	3.118*1e-3	15	2.331*1e-3	12	0.985*1e-3	13
	2.473*1e-3	0.815*1e-3	2.857*1e-3	0.285*1e-3	3.906*1e-3	0.321*1e-3	3.462*1e-3	0.310*1e-3	2.524*1e-3	0.775*1e-3	1.041*1e-3	0.906*1e-3
WAN	7.772*1e-4	12	5.610*1e-4	13	7.520*1e-4	11	4.640*1e-4	15	4.584*1e-4	12	4.534*1e-4	15
	9.947*1e-4	0.517*1e-4	6.430*1e-4	0.460*1e-4	7.530*1e-4	0.010*1e-4	4.950*1e-4	0.090*1e-4	4.968*1e-4	0.223*1e-4	4.780*1e-4	0.625*1e-4
WIZ	5.087*1e-4	15	3.670*1e-4	15	4.650*1e-4	11	3.490*1e-4	12	5.619*1e-4	8	4.321*1e-4	11
	5.333*1e-4	0.388*1e-4	3.680*1e-4	0.010*1e-4	4.670*1e-4	0.020*1e-4	3.521*1e-4	0.023*1e-4	5.735*1e-4	0.123*1e-4	4.464*1e-4	0.087*1e-4
AUT	8.088*1e-3	14	6.821*1e-3	15	6.980*1e-3	14	10.368*1e-3	11	8.420*1e-3	12	5.202*1e-3	13
	8.153*1e-3	0.039*1e-3	7.619*1e-3	0.501*1e-3	7.280*1e-3	0.504*1e-3	10.934*1e-3	0.407*1e-3	8.555*1e-3	0.137*1e-3	5.644*1e-3	0.433*1e-3
ELE	2.962*1e-4	15	4.660*1e-4	12	5.530*1e-4	12	5.174*1e-4	15	7.502*1e-4	7	3.611*1e-4	12
	7.333*1e-4	4.468*1e-4	4.810*1e-4	0.130*1e-4	5.740*1e-4	0.300*1e-4	5.356*1e-4	0.241*1e-4	9.171*1e-4	2.017*1e-3	3.749*1e-4	0.157*1e-4
QUA	3.353*1e-1	15	8.528*1e-1	14	5.538*1e-1	10	4.958*1e-1	15	3.685*1e-1	12	2.881*1e-1	15
	3.408*1e-1	0.070*1e-1	9.836*1e-1	1.464*1e-1	5.813*1e-1	0.404*1e-1	5.829*1e-1	0.981*1e-1	3.767*1e-1	0.057*1e-1	2.926*1e-1	0.457*1e-1
MAC	4.766*1e-3	5	1.780*1e-3	15	7.444*1e-3	15	8.850*1e-3	5	5.220*1e-3	15	2.120*1e-3	11
	7.445*1e-3	1.852*1e-3	1.823*1e-3	0.026*1e-3	7.552*1e-3	0.189*1e-3	8.929*1e-3	0.066*1e-3	5.897*1e-3	0.529*1e-3	2.825*1e-3	0.032*1e-3
DIA	3.898*1e-2	8	5.618*1e-2	8	5.647*1e-2	8	9.963*1e-2	5	6.798*1e-2	9	4.214*1e-2	5
	5.405*1e-2	1.229*1e-2	6.934*1e-2	1.349*1e-2	8.468*1e-2	2.014*1e-2	11.955*1e-2	1.633*1e-2	7.954*1e-2	1.286*1e-2	4.694*1e-2	0.353*1e-2
ANA	8.491*1e-3	5	8.763*1e-3	10	7.325*1e-3	7	8.367*1e-3	5	8.491*1e-3	6	6.782*1e-3	5
	8.658*1e-3	0.157*1e-3	13.741*1e-3	3.422*1e-3	9.230*1e-3	1.519*1e-3	16.402*1e-3	12.504*1e-3	8.658*1e-3	0.157*1e-3	9.826*1e-3	2.702*1e-3
DEL	3.888*1e-3	6	4.078*1e-3	5	3.506*1e-3	5	3.919*1e-3	5	7.987*1e-3	6	3.254*1e-3	5
	4.347*1e-3	0.783*1e-3	4.210*1e-3	0.205*1e-3	4.132*1e-3	0.435*1e-3	4.341*1e-3	0.367*1e-3	8.237*1e-3	0.237*1e-3	3.859*1e-3	0.621*1e-3

Table 6

Holm test results for DRW-TSK-FC vs. FS-FCSVM, 0-order-TSK-FC, NN, TSFS-SVR and L2-TSK-FC with α = 0.05

i	Classifiers	z	p	Holm=α/i	Hypothesis
5	NN	4.2993	1.7e-5	0.01	Rejected
4	FS-FCSVM	4.2521	2.1e-5	0.0125	Rejected
3	TSFS-SVR	3.5907	3.3e-4	0.0167	Rejected
2	L2-TSK-FS	3.0710	2.1e-3	0.0025	Rejected
1	0-order-TSK-FS	1.7953	0.0726	0.05	Not Rejected

6 Conclusion

In this paper, a novel TSK fuzzy system DRW-TSK-FS is proposed in which dynamic rule weights are introduced to guarantee the interpretability and the stacked generalization principle is used to ensure the modeling performance. A basic input-output unit and an augmenting unit are involved in DRW-TSK-FS and form a special structure that is similar to the stacked structure. By means of the stacked structure, the predictive information from the augmenting unit as augmented features can be leveraged to open the manifold structure of the feature space such that the enhanced modeling performance can be expected. Superficially, since the predictive information has no specific physical meaning, it seems to reduce the interpretability of fuzzy rules. However, from the mathematical view, fuzzy rules in DRW-TSK-FS can be rewritten in another form in which the predictive information is removed from the if-parts and hidden in DRW. Hence, the if-parts of all fuzzy rules always have concise physical meanings. More importantly, DRW can be taken as a nonlinear function of the input pattern which reflects the acceptability of a fuzzy rule with regard to an input pattern. In addition, the isolation level of an input pattern can be measured by the summed DRW values of all fuzzy rules in DRW-TSK-FS. The promising modeling performance, high interpretability and the ability of outlier detection are evaluated on 16 real-life datasets.

In our future works, we will gear towards developing a TSK fuzzy system which provides a mechanism to measure the acceptability of fuzzy rules acting on each individual feature involved in the input patterns. Moreover, like the issues discussed in [41, 42], we will extend the current results for control of general nonlinear systems based on fuzzy/piecewise dynamic models and for similar issues.

Footnotes

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China under Grants 81701793, 61272210, 61572236, 61300151 and NSFC-JSPS Grant 6161101250, by the Natural Science Foundation of Jiangsu Province under Grant BK20161268, by the Fundamental Research Funds for the Central Universities (JUDCF13030), and by the Science and Technology Plan Funding Project of Nantong (MS12017016-2).

References

Iglesias

, Angelov

, Ledezma

and Sanchis

, Human activity recognition based on evolving fuzzy systems, Int J Neural Syst 20(5) (2010), 355–364.

Lemos

, Caminhas

and Gomide

, Adaptive fault detection and diagnosis using an evolving fuzzy classifier, Inf Sci 220(1) (2013), 64–85.

Jiang

, Wu

, Deng

, Qian

, Wang

, Chung

F.L.

, Choi

K.S.

and Wang

Shitong

, Seizure classification from EEG signals using transfer learning, semi-supervised learning and TSK fuzzy system, IEEE Transactions on Neural Systems and Rehabilitation Engineering 25(12) (2017), 2270–2284.

Deng

, Jiang

, Chung

F.L.

, Ishibuchi

and Wang

, Knowledge-leverage-based fuzzy system and its modeling, IEEE Trans Fuzzy Syst 21(4) (2013), 597–609.

Nauck

and Kruse

, How the learning of rule weights affect the interpretability of fuzzy systems, 1998 IEEE International Conference on Fuzzy Systems Proceedings, Anchorage, AK 2 (1998), 1235–1240.

Ishibuchi

and Nakashima

, Effect of rule weights in fuzzy rule-based classification systems, IEEE Transactions on Fuzzy Systems 9(4) (2001), 506–515.

Ishibuchi

and Yamamoto

, Rule weight specification in fuzzy rule-based classification systems, in, IEEE Transactions on Fuzzy Systems 13(4) (2005), 428–435.

Tsang

E.C.C.

, Lee

J.W.T.

and Yeung

D.S.

, Tuning certainty factor and local weight of fuzzy production rules by using fuzzy neural network, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) 32(1) (2002), 91–98.

Chen

S-M.

and Huang

C-M.

, Generating weighted fuzzy rules from relational database systems for estimating values using genetic algorithms, IEEE Transactions on Fuzzy Systems 11(4) (2003), 495–506.

10.

Chen

, Shen

, Su

and Shang

, Refinement of fuzzy rule weights with particle swarm optimization, 2014 14th UK Workshop on Computational Intelligence (UKCI), Bradford (2014), pp. 1–7.

11.

Tormási

and Kóczy

L.T.

, Dynamic fuzzy rule weight optimization for a fuzzy based single-stroke character recognizer, 2013 IEEE 17th International Conference on Intelligent Engineering Systems (INES), San Jose, 2013, pp. 119–124.

12.

Karasakal

, Guzelkaya

, Eksin

and Yesil

, Online rule weighting of fuzzy PID controllers, 2010 IEEE International Conference on Systems, Man and Cybernetics, Istanbul, 2010, pp. 1741–1747.

13.

Amaitik

N.M.

and Buckingham

C.D.

, Developing a hierarchical fuzzy rule-based model with weighted linguistic rules: Acase study of water pipes condition prediction, 2017 Computing Conference, London, 2017, pp. 30–40.

14.

Deng

, Jiang

, Chung

F.L.

, Ishibuchi

, Choi

K.S.

and Wang

, Transfer prototype-based fuzzy clustering, IEEE Trans Fuzzy Syst 24(5) (2016), 1210–1232.

15.

Deng

Z.H.

, Choi

K.S.

, Chung

F.L.

and Wang

S.T.

, Scalable TSK fuzzy modeling for very large datasets using minimal-enclosing-ball approximation, IEEE Trans Fuzzy Syst 19(2) (2011), 210–226.

16.

Leski

, TSK-fuzzy modeling based on -insensitive learning, IEEE Trans Fuzzy Syst 13(2) (2005), 181–193.

17.

Jang

J.S.R.

, ANFIS: Adaptive-network-based fuzzy inference systems, IEEE Trans Syst Man Cybern 23(3) (1993), 665–685.

18.

Leski

, TSK-fuzzy modeling based on ɛ-insensitive learning, IEEE Trans Fuzzy Syst 13(2) (2005), 181–193.

19.

Juang

C.F.

and Hsieh

C.D.

, A fuzzy system constructed by rule generation and iterative linear SVR for antecedent and consequent parameter optimization, IEEE Transactions on Fuzzy Systems 20(2) (2012), 372–384.

20.

Qian

Z.L.

, Juan

D.C.

, Bogdan

, Tsui

C.Y.

, Marculescu

and Marculescu

, A support vector regression (SVR)-based latency model for network-on-chip (NoC) architectures, IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems 35(3) (2016), 471–484.

21.

Peng

J.X.

, Li

and Irwin

G.W.

, A new Jacobian matrix for optimal learning of single-layer neural networks, IEEE Transactions on Neural Networks 19(1) (2008), 119–129.

22.

Bhama

and Singh

, Single layer neural networks for linear system identification using gradient descent technique, IEEE Transactions on Neural Networks 4(5) (1993), 884–888.

23.

Zurada

J.M.

and Shen

, Sufficient condition for convergence of a relaxation algorithm in actual single-layer neural networks, IEEE Transactions on Neural Networks 1(4) (1990), 300–303.

24.

Wolpert

D.H.

, Stacked generalization, Neural Netw 5 (1992), 241–259.

25.

Wang

, Chung

F.-L.

, Shen

H.-B.

and Hu

, Cascaded centralized TSK fuzzy system: Universal approximator and high interpretation, Appl Soft Comput 5(2) (2005), 131–145.

26.

Alata

, Su

C.Y.

and Demirli

, Adaptive control of a class of nonlinear systems with a first-order parameterized Sugeno fuzzy approximator, IEEE Trans Syst Man Cybern C Appl Rev 31(3) (2001), 410–419.

27.

Dehuri

and Cho

S.-B.

, A comprehensive survey on functional link neural networks and an adaptive PSO– BP learning for CFLNN, Neural Computing & Applications 19 (2010), 187–205.

28.

Box

E.P.

and Jenkins

G.M.

, Time Series Analysis, Forecasting and Control. San Francisco, CA: Holden Day, 1970.

29.

Alcal-Fdez

, Fernndez

, Luengo

, Derrac

, Garca

, Snchez

and Herrera

, KEEL data-mining software tool: Data set repository integration of algorithms and experimental analysis framework, J Multiple-Valued Logic and Soft Computing 17(2) (2011), 255–287.

30.

Juang

C.F.

, Chiu

S.H.

and Shiu

S.J.

, Fuzzy system learned through fuzzy clustering and support vector machine for human skin color segmentation, IEEE Trans Syst Man Cybern A Syst Humans 37(6) (2007), 1077–1087.

31.

Juang

C.F.

and Hsieh

C.D.

, TS-fuzzy system-based support vector regression, Fuzzy Sets Syst 160(17) (2009), 2486–2504.

32.

, Chung

F.L.

and Wang

, Bayesian Takagi– Sugeno– Kang fuzzy classifier, IEEE Transactions on Fuzzy Systems 25(6) (2017), 1655–1671.

33.

Zhang

, Ishibuchi

and Wang

, Deep Takagi– Sugeno– Kang fuzzy classifier with shared linguistic fuzzy rules, IEEE Transactions on Fuzzy Systems 26(3) (2018), 1535–1549.

34.

Demar

, Statistical comparisons of classifiers over multiple data sets, J Machine Learning Research 7 (2006), 1–30.

35.

Hodges

J.L.

and Lehmann

E.L.

, Ranks methods for combination of independent experiments in analysis of variance, Ann Math Statist 33 (1962), 482–497.

36.

Holm

, A simple sequentially rejective multiple test procedure, Scand J Stat 6(2) (1979), 65–70.

37.

Jiang

, Deng

, Chung

F.L.

and Wang

, Realizing Two-View TSK Fuzzy Classification System by Using Collaborative Learning, IEEE Trans Syst Man Cybern: Syst 47(1) (2017), 145–160.

38.

Deng

, Cao

, Jiang

and Wang

, Minimax probability TSK fuzzy system classifier: A more transparent and highly interpretable classification model, IEEE Trans Fuzzy Syst 23(4) (2015), 813–826.

39.

Juang

C.F.

, Chiu

S.H.

and Chang

S.W.

, A self-organizing TS-type fuzzy network with support vector learning and its application to classification problems, IEEE Trans Fuzzy Syst 15(5) (2007), 998–1008.

40.

Zhao

, Xin

, Ren

and Guo

, Application and Comparison of BP Neural Network Algorithm in MATLAB, 2010 International Conference on Measuring Technology and Mechatronics Automation, Changsha City, 2010, pp. 590–593.

41.

Qiu

, Wei

and Wu

, A Novel Approach to Reliable Control of Piecewise Affine Systems With Actuator Faults, IEEE Transactions on Circuits and Systems II: Express Briefs 64(8) (2017), 957–961.

42.

Wang

, Qiu

, Chadli

and Wang

, A Switched System Approach to Exponential Stabilization of Sampled-Data T– S Fuzzy Systems with Packet Dropouts, IEEE Transactions on Cybernetics 46(12) (2016), 3145–3156.