DFLOWFS: Deep Fast-Learning Optimal Weight Fuzzy System with bottom-up hierarchical structure for high-dimensional data regression

Abstract

As a model for reasoning and decision-making based on fuzzy rules, fuzzy systems have high interpretability. However, when the data dimension increases, the fuzzy system will face the problem of “rule explosion”, making it difficult to learn and predict effectively. In this paper, the fuzzy system trained by the FLOWFS (Fast-Learning with Optimal Weights for Fuzzy Systems) algorithm is used as sub-module in the deep fuzzy system, and the deep fuzzy system DFLOWFS (Deep FLOWFS) is constructed from the bottom-up hierarchical structure as the following three steps. 1) The FLOWFS algorithm assigns weight attributions to each fuzzy rule, and the rule weights are trained by the least square method with regularization terms to shorten training time and improve accuracy. 2) Three strategies of dividing high-dimensional inputs into multiple low-dimensional inputs are proposed as sequential division, random division and correlation division. Then, it is verified by experiments that the correlation division has the best performance. 3) The sub-module discarding method is proposed to discard the sub-modules with poor performance to have a maximum improvement of 13.8% compared to the DFLOWFS without using the sub-module discarding method. Then, the optimized DFLOWFS is verified and compared with the other three classic regression models on the three UCI datasets. Experiments show that with the increase of the data dimension, DFLOWFS not only have good interpretability but also have good accuracy. Furthermore, DFLOWFS performs best among all models in comprehensive scores, with good learning ability and generalization ability. Therefore, the proposed strategies with hierarchical structure for optimal shallow fuzzy systems are effective, which give a new insight for fuzzy system research.

Keywords

Correlation division fuzzy system interpretability rule weights submodule discarding method

1 Introduction

From traditional machine learning to artificial neural networks to deep learning, artificial intelligence has provided us with great convenience in daily life, especially in fields such as facial recognition and autonomous driving [1]. However, its lack of interpretability also brings security risks [2]. As a reasoning Decision model based on fuzzy rules, fuzzy system usually has high interpretability and researchers [3].

Although fuzzy system is a highly interpretable artificial intelligence method, its theoretical research is not mature enough [4]. The extraction of fuzzy rules was mainly based on the knowledge and experience of experts in the early stage, and it is called fuzzy system modeling based on expert knowledge. With the development of the times, the amount of information is gradually increasing, and it is inevitable that there is a lack of expert knowledge. Therefore, research began to turn to data-driven fuzzy system modeling, especially the Wang-Mendel (WM) method proposed by Wang Lixin and Mendel [5–7]. It realized the extraction of fuzzy rules from data, no longer relying mainly on expert knowledge and experience, greatly improving the intelligence level of fuzzy systems. However, the WM method has the problem that the accuracy is not high enough, which inevitably affects the reliability of decision-making. Therefore, how to improve the accuracy of fuzzy system modeling has become the key issue [8].

Many scholars have attempted many new methods, such as those based on fuzzy clustering, genetic algorithms, and neural networks. Guo J [9] proposed an improved WM method based on the approximate fuzzy c-means (AFCM) algorithm to generate fuzzy rules, which improves the performance of fuzzy systems. Zhang R [10] combines the improved genetic algorithm (IGA) with dynamic Auto to propose a regression (ARX) Takagi Sugeno (T-S) fuzzy model with exogenous inputs. The T-S fuzzy model has good modeling accuracy and simple structure. Harifi [11] proposed using the newly released Emperor Penguin Swarm Algorithm to optimize ANFIS (adaptive-network-based fuzzy inference system), achieving higher model accuracy. Liu [12] proposed a Hebbian based rule reduction algorithm (HeRR) and a new rough set based attribute selection algorithm RS-HeRR for fuzzy system rule reduction, which are used to generate effective, interpretable and compact rule sets. Although these methods improve the accuracy of fuzzy system modeling, they all have some shortcomings. On the one hand, the model is relatively complex and the optimization time is too long, and most of it starts from a single direction without considering the domain partition, membership function parameters, number of rules, fuzzy system structure, and other comprehensive factors. On the other hand, it is difficult to solve the problem of dealing with high-dimensional data when fuzzy systems face “dimensional disasters”. The DFLOWFS method proposed in this article is a fuzzy system modeling method that can efficiently and quickly process high-dimensional data. Compared with other methods, it not only has better learning ability and generalization ability, but also higher interpretability; With the increase in data dimensions, DFLOWFS still has good performance and can better fit data; It has significant theoretical and practical significance.

2 FLOWFS: Fast learning-algorithm with optimal weights for fuzzy systems

The algorithm that assigns fuzzy rules weight attributes and trains the rule weights through the least squares method is named as FLOWFS (Fast Learning-algorithm with Optimal Weights for Fuzzy Systems). The least squares method is a numerical optimization technique, which seeks to find the optimal parameters of the function to be fitted by minimizing the sum of squares of errors between all true and fitted values. The use of the least squares method can be easily and quickly solved, and has good mathematical theoretical support, as well as good fitting effect and accuracy. When FLOWFS uses the least squares method to train weights, it adds a regularization term to prevent the Multicollinearity problem of the characteristic matrix, while restraining the over fitting of the model, and improving the generalization ability of the model. It significantly reduces the Time complexity of the algorithm, improves the running time of the algorithm, and improves the accuracy. From this, DFLOWFS is based on fuzzy systems with fast running speed and high accuracy in low dimensional space. High dimensional problems are transformed into multiple low dimensional problems and combined to effectively process high-dimensional data. The specific implementation steps of FLOWFS are shown in Table 1.

Table 1
FLOWFS algorithm implementation steps

Step 1: Data is normalized and divided into training set and test set.

Step 2: Determine the antecedent parameters of fuzzy rules in a complete fuzzy rule base. (1) Set the number of fuzzy sets for each dimensional input space. (2) Divide the input space equally, and determine the membership function form corresponding to each fuzzy set in the input space. (3) Determine the antecedents of all fuzzy rules based on the characteristics of the full arrangement.

Step 3: Determine the consequences of fuzzy rules in a complete fuzzy rule library. (1) Using all the data and the contribution of the data to the rule, determine the output center point of the consequent fuzzy set. (2) Using all the data and the contribution of the data to the rule, determine the output center point of the consequent fuzzy set. (3) Determine the membership function form corresponding to the fuzzy set in the consequent.

Step 4: Training rule weight w. (1) Set the regularization coefficient α (α ≠ 0). (2) Use the least square method to train all rule weights w.

Step 5: After constructing the fuzzy system model, predict the test set.

Through the analysis, the optimal structure is searched from all the fuzzy set structures, thus changing the structure of the initial fuzzy sets to adapt to the data, and then changing the membership degree of the data in the rules [13]. Compared with the initial time, the data in the rules and the membership degree on the network expands or shrinks by a certain multiple. In order to achieve this effect, we assign weighted attributes to the fuzzy rules, and change the membership degree of the data on the rules through the rule weights, which is equivalent to changing the structure of the fuzzy sets [14, 15].

For the extraction of fuzzy rules in the fuzzy system [16], a complete fuzzy rule base is obtained by training the full rule base, so it is only necessary to determine the weight of the rules. At the same time, it is not necessary to reduce redundant rules after adding rule weights. When the weight of the fuzzy rule is 0 or close to 0, the rule will not work in the model, which is equivalent to the rule being reduced. It can be seen that the addition of fuzzy rule weights not only simplifies the optimization of the fuzzy set structure, but also reduces the redundant rules, which greatly reduces the training time of the model, and the time complexity is significantly reduced.

3 DFLOWFS: Construction of deep fuzzy system by FLOWFS

FLOWFS is a fuzzy system modeling algorithm based on rule weight optimization. It assigns weight to each fuzzy rule and trains the rule weights by the least squares method with a regularization term [17]. The accuracy of the model has also been improved. The fuzzy system trained by the FLOWFS is used as a sub-module in the deep fuzzy system and the deep fuzzy system is constructed from the bottom up [18]. The deep fuzzy system is named DFLOWFS, and its algorithm steps are shown in Table 2. Taking the window size of 3 and the sliding step size of 2 as an example, the structure of the DFLOWFS is shown in Fig. 1.

Table 2
DFLOWFS algorithm steps

Step 1: Set the window size w and sliding step s for one-dimensional convolution of data dimensions.

Step 2: Divide the dataset into multiple sub datasets based on parameters w and s. For those with insufficient window size, random filling method is used to fill in the gaps.

Step 3: Train each sub dataset using the FLOWFS method (Table 1) to construct a sub fuzzy system.

Step 4: (1) If the final layer of fuzzy system has been built, go to step 5. (2) If you have not yet built to the last layer, combine the outputs of this layer in order as input to the next layer, and go to step 2.

Step 5: The deep fuzzy system has been constructed and can be used for prediction.

Fig. 1

Structure diagram of deep fuzzy system.

Suppose an n-dimensional input and single-output training data $X^{0} {= [x}_{1}^{0} {, x}_{2}^{0}, \dots x_{n}^{0}; y]$ , where the n is a large number. A sub-module is constructed by using the idea of one-dimensional convolution for each layer. Suppose we build a sub-module with a window size of 3 and a sliding step size of 2. For the first layer, the dataset will be divided into $\begin{matrix} X_{1}^{0} {= [x}_{1}^{0} {, x}_{2}^{0} {, x}_{3}^{0}; y], \\ X_{2}^{0} {= [x}_{2}^{0} {, x}_{3}^{0} {, x}_{4}^{0}; y], \dots, \\ X_{n}^{0} {= [x}_{n - 2}^{0} {, x}_{n - 1}^{0} {, x}_{n}^{0}; y] . \end{matrix}$

Sub-dataset $X_{1}^{0}$ is used to construct the first fuzzy system ${FS}_{1}^{1}$ of the first layer; Sub-dataset $X_{2}^{0}$ is used to construct the second fuzzy system ${FS}_{2}^{1}$ of the first layer; and so on, Sub-dataset $X_{n - 2}^{0}$ is used to construct the n-2th fuzzy system ${FS}_{n - 2}^{1}$ of the first layer. Among them, ${FS}_{1}^{1}, \dots,$ ${FS}_{n - 2}^{1}$ is trained by the FLOWFS learning algorithm proposed in the previous section. Because the dimension of each sub-dataset is 3, the “rule explosion” problem disappears, and the fuzzy system trained by FLOWFS can predict efficiently and accurately. After training and prediction, suppose the predicted output value of $X_{1}^{0}$ on ${FS}_{1}^{1}$ is $y_{1}^{1}$ , and so on, the predicted output value of $X_{n - 2}^{1}$ on ${FS}_{n - 2}^{1}$ is $y_{n - 2}^{1}$ . We sequentially combine the output values predicted by the first-layer fuzzy system into a new vector as the input data X¹ of the second-layer fuzzy system, as shown in Formula (1): $X^{1} {= [y}_{1}^{1} {, y}_{2}^{1}, \dots {, y}_{n - 2}^{1}; y]$ (1) Rewrite it as Formula (2): $X^{1} {= [x}_{1}^{1} {, x}_{2}^{1}, \dots {, x}_{m}^{1}; y]$ (2) $x_{1}^{1} {= y}_{1}^{1}, x_{2}^{1} {= y}_{2}^{1}, \dots, x_{m}^{1} {= y}_{n - 2}^{1}$

We train the second layer in the same way as we train the first layer. A submodule is constructed with a window size of 3 and a window sliding step size of 2. First, the dataset will be divided into $X_{1}^{1} {= [x}_{1}^{1} {, x}_{2}^{1} {, x}_{3}^{1}; y]$ , $X_{2}^{1} {= [x}_{2}^{1} {, x}_{3}^{1} {, x}_{4}^{1}; y], \dots,$ $X_{m - 2}^{1} {= [x}_{m - 2}^{0} {, x}_{m - 1}^{0} {, x}_{m}^{0}; y]$ . Sub-dataset $X_{1}^{1}$ is used to construct the first fuzzy system of the second layer; Sub-dataset ${FS}_{1}^{2}$ is used to construct the second fuzzy system of the second layer; and so on, Sub-dataset $X_{m - 2}^{1}$ is used to construct the second fuzzy system The m-2th fuzzy system ${FS}_{m - 2}^{2}$ of the layer. After training and prediction, it is assumed that the predicted output value of $X_{1}^{1}$ on ${FS}_{1}^{2}$ is $y_{1}^{2}$ ; the predicted output value of $X_{2}^{1}$ on ${FS}_{2}^{2}$ is $y_{2}^{2}$ ; and so on, the predicted output value of $X_{m - 2}^{1}$ on ${FS}_{m - 2}^{1}$ is $y_{m - 2}^{2}$ . Similarly, the prediction output of each dataset on the second layer fuzzy system is used as the input of the third layer fuzzy system to train the third layer, and so on, until the last layer of fuzzy system is constructed [19].

During the construction process, in the case of one-dimensional convolution, the data dimension in the last window of each layer may have the problem of insufficient window size. For example, for a four-dimensional input data, assuming that the window size is 3 and the sliding step size of the window is 2, then after the window is slid once, the data in the window only includes the third dimension and the fourth dimension, less than three dimensions. Therefore, this paper uses random filling to fill the vacancies, and randomly selects the missing number of data dimensions from the data dimensions that are not in the current window to fill. In the above example, this method randomly selects one dimension from the first and the second dimensions, and forms a three-dimensional sub-dataset with the third and the fourth dimensions, which satisfies the window size.

4 Optimization of DFLOWFS

4.1 High-dimensional input partition strategy

In Section 3, we divided the sub-datasets according to the initial order of the data, and we defined this division as sequential division. Sequential division can be regarded as an unguided division, which only constructs sub-datasets in a continuous range based on the initial order, so the initial order of data input features will affect each sub-fuzzy system, thereby affecting the performance of deep fuzzy systems.

In this section, we propose two other partitioning strategies. A division strategy that we define as random division is also an unguided division strategy. The random scramble function is used to randomly scramble the data dimension, which introduces randomness to the division of the dataset, so that it no longer depends on the initial data. order [20]. The other is defined as correlation division. This division strategy first calculates the Pearson correlation coefficient between each dimension of the input feature and the target variable according to Formula (3), and then arranges the data dimensions in descending order according to the correlation coefficient. The data are divided sequentially [21]. This is a guided partitioning strategy. Assuming that there is a four-dimensional input data, we give each dimension a number of 1, 2, 3, and 4, respectively. After the correlation coefficient is sorted in descending order, the order of the data dimensions becomes 3, 2, 4, and 1, then we will According to this order, the sub-dataset is divided according to the sequence division strategy. If the window size is 3 and the sliding step is 1, then the data of the third, second and fourth dimensions will form the first sub-dataset, and the second sub-dataset that, the fourth dimension and the first dimension will build. $r = \frac{\sum_{i = 1}^{n} (X_{i} - \bar{X}) (Y_{i} - \bar{Y})}{\sqrt{\sum_{i = 1}^{n} {(X_{i} - \bar{X})}^{2}} \sqrt{\sum_{i = 1}^{n} {(Y_{i} - \bar{Y})}^{2}}}$ (3)

Among them, $\sqrt{\sum_{i = 1}^{n} {(X_{i} - \bar{X})}^{2}} \sqrt{\sum_{i = 1}^{n} {(Y_{i} - \bar{Y})}^{2}}$ is the standard deviation, $\sum_{i = 1}^{n} (X_{i} - \bar{X}) (Y_{i} - \bar{Y})$ is the covariance of the random variable X and Y, and the quotient of the covariance and standard deviation between the two variables is the correlation coefficient of the random variable X and Y, denoted as r. The correlation coefficient r is a method of measuring the degree of correlation between the random variable X and Y, and the value range of the correlation coefficient r is [-1,1]. The larger the absolute value of the correlation coefficient r, the higher the correlation between X and Y. When X and Y are linearly correlated, the correlation coefficient takes the value of 1 (positive linear correlation) or -1 (negative linear correlation).

We use the Bc dataset in Table 8 to conduct comparative experiments with DFLOWFS with a window size of 3 and a sliding step size of 2, to compare the effects of the three partitioning strategies on the model performance. Figure 2 illustrates the difference between the predicted value and the true value of DFLOWFS on the training set and test set under three partitioning strategies, where order means sequential division, random means random division, and corr means correlation coefficient for guided division (Fig. 2(1) shows the errors of the three partitioning strategies on the training set, and Fig. 2(2) shows the errors of the three partitioning strategies on the test set).

Fig. 2

Error images of DFLOWFS under three partition strategies.

At the same time, we use objective evaluation indicators absolute mean error MAE (Formula (4)), mean square error MSE (Formula (5)), determination coefficient R² (Formula (6)), and symmetric mean absolute percentage error SMAPE (Formula (7)) for evaluation. Validate on three datasets in the UCI Machine Learning Repository (detailed in Section 5.2). The evaluation indicators of DFLOWFS on the training set under three partitioning strategies are shown in Table 3. The evaluation indicators of DFLOWFS on the test set under three partitioning strategies are shown in Table 4.

Table 3

Evaluation index values of DFLOWFS on the training set under the three partition strategies

	MAE	MSE	R²	SMAPE
Order	0.05513	0.00505	0.75662	9.96%
Random	0.05395	0.00499	0.75964	9.72%
Corr	0.05314	0.00474	0.77138	9.62%

Table 4

Evaluation index values of DFLOWFS on the test set under the three partition strategies

	MAE	MSE	R²	SMAPE
Order	0.06880	0.00747	0.65379	12.30%
Random	0.06137	0.00630	0.70801	11.01%
Corr	0.05636	0.00549	0.74582	10.48%

From the above experiments, it can be seen that DFLOWFS using relevant partitioning strategies has the best performance on both the training and testing sets. As shown in Tables 3 and 4, DFLOWFS, which uses relevant partitioning strategies, ranks first among all four evaluation indicators and has good learning and generalization abilities. DFLOWFS under random partitioning strategy also has good fitting performance due to the introduction of randomness. It can be seen that compared to random partitioning strategies and sequential partitioning strategies, correlated partitioning strategies have better performance.

4.2 Submodule discard method

In Section 3, each sub-module of each layer is retained, so that there may be some sub-modules with poor performance participating in the establishment of the model, causing the model to be biased. On the other hand, all sub-modules are retained, which will make the model too complex and easily lead to overfitting.

Therefore, this section proposes the submodule discarding method [22], which selectively discards submodules at each layer. After the training of each layer is completed, the performance of the sub-module is evaluated. Here we use the coefficient of determination R² to evaluate the performance of the sub-module and set a drop threshold. When the R² value of the sub-module is lower than the drop threshold, the sub-module will be discarded and no longer participates in the construction of the next layer of fuzzy systems. Suppose after training, the first layer has four sub-modules, sub-module ${1FS}_{1}^{1}$ , sub-module ${2FS}_{2}^{1}$ , sub-module ${3FS}_{3}^{1}$ , sub-module ${4FS}_{4}^{1}$ , where the value of sub-module 4 is lower than the discard threshold, sub-module 4 will be discarded and will no longer participate in the training of the second layer. The input of the second layer changes the dataset composed of the outputs of the original four sub-modules into a dataset composed of the outputs of sub-module 1, sub-module 2 and sub-module 3, that is, the original $X^{1} = [y_{1}^{1}, y_{2}^{1}, y_{3}^{1}, y_{4}^{1}; y]$ becomes $X^{1} = [y_{1}^{1}, y_{2}^{1}, y_{3}^{1}; y]$ , where $y_{1}^{1}, y_{2}^{1}, y_{3}^{1}, y_{4}^{1}$ represents the predicted output value on submodule 1, submodule 2, submodule 3, and submodule 4, respectively.

In order to verify the effectiveness of the sub-module discarding method, we also use the Bc dataset in Table 8, with the window size of 3, the sliding step size of 2, and the sub-input space using the correlation division of DFLOWFS to conduct comparative experiments, and compare the sub-module discarding. The effect of DFLOWFS model performance with and without the submodule dropping method, where we set the dropout threshold to 0.2 in the submodule dropping method. Figure 3 is an image of the difference between the predicted value and the true value of DFLOWFS on the training set and test set under the two sub-module selection strategies.

Fig. 3

Error images of DFLOWFS under two submodule selection strategies.

At the same time, we use objective evaluation indicators MAE, MSE, R² and SMAPE for evaluation. Validate on three datasets in the UCI Machine Learning Repository (detailed in Section 5.2). The evaluation index values of DFLOWFS under the two submodule selection strategies on the training set are shown in Table 5, and the evaluation index values on the test set are shown in Table 6.

Table 5

Evaluation index values of DFLOWFS on the training set under the two submodule selection strategies

	MAE	MSE	R²	SMAPE
Corr	0.05314	0.00474	0.77138	9.62%
Corr_dropout	0.05431	0.00497	0.76051	9.81%

Table 6

Evaluation index values of DFLOWFS on the test set under the two submodule selection strategies

	MAE	MSE	R²	SMAPE
Corr	0.05636	0.00549	0.74582	10.48%
Corr dropout	0.05191	0.00473	0.78100	9.43%

It can be seen from the above experiments that although the values of each indicator in the training set have decreased, all indicators in the test set have improved, and the proportion of improvement is higher than that of the training set. Among them, MAE on the test set increased by 7.8%, MSE increased by 13.8%, R² increased by 4.7%, and SAMPE increased by 10.1%, while the maximum decrease on the training set was only 4.7%. Therefore, it can be seen that the performance improvement of the model on the test set is greater than the decrease in the training set, and the generalization ability of the model is greatly improved. Compared with the learning ability, the generalization ability of the model is more important in practical applications. The improvement of the ability can better predict the unknown data. From another point of view, the sub-module discarding method reduces the complexity of the model and prevents the model from overfitting by discarding some sub-modules with poor performance.

5 Comparisons and analysis with DFLOWFS and classic algorithms

Through the above research, we improved the learning algorithm of DFLOWFS in Table 2. The complete learning algorithm of DFLOWFS is shown in Table 7 (The implementation process of the FLOWFS module is shown in Table 1).

Table 7
Steps of the optimized DFLOWFS algorithm

Step 1: Set the window size w, sliding step s, and drop threshold prob for one-dimensional convolution of data dimensions.

Step 2: According to the parameters w and s, the dataset is divided into multiple sub datasets using a “correlation partitioning” strategy. For those with insufficient window size, a random filling method is used to fill in the gaps.

Step 3: Train each sub dataset using the FLOWFS method (Table 1).

Step 4: Calculate the R² of the sub-fuzzy system, discard the sub-modules below the discarding threshold prob, and keep them otherwise.

Step 5: (1) If the fuzzy system has been constructed to the last layer, go to step 5. (2) If the last layer has not been constructed, combine the outputs of this layer in order as the input of the next layer, and go to step 2.

Step 6: The deep blur system is built and can be predicted.

5.1 Evaluation indicators

In order to evaluate the proposed DFLOWFS algorithm, the back propagation (BP) neural network, the radial basis function (RBF) neural network [23], and long short-term memory (LSTM) method were introduced for comparison and validation. At the same time, in order to verify the impact of introducing fuzzy rule weights, we will not conduct rule weight training but also add comparisons, that is, the algorithm that does not execute step 4 in Table 4-1 will be named FRFS. In addition, we introduce absolute mean error MAE (Formula (4)), mean square error MSE (Formula (5)), determination coefficient R² (Formula (6)), and symmetric mean absolute percentage error SMAPE (Formula (7)) for quantitative evaluation. $MAE = \frac{\sum_{j = 1}^{n} | y_{j} - {\hat{y}}_{j} |}{n}$ (4) $MSE = \frac{1}{n} \sum_{j = 1}^{n} (⌢ y_{j} - y_{j})$ (5) $R^{2} = \frac{\sum_{j = 1}^{n} {(y_{j} - {\overset{⌢}{y}}_{j})}^{2}}{\sum_{j = 1}^{n} {(y_{j} - y \frac{1}{n} \sum_{j = 1}^{n} {\overset{⌢}{y}}_{j})}^{2}}$ (6) $SMAPE = \frac{100 %}{n} \sum_{j = 1}^{n} \frac{| {\hat{y}}_{j} - y_{j} |}{| {\hat{y}}_{j} | + | y_{j} |}$ (7)

Assuming n m-dimensional training data $X_{i} = [x_{1}^{j}, x_{2}^{j}, \dots, x_{m}^{j}; y_{j}]$ , where. For training data with m-dimension input, it is necessary to partition the input space and allocate fuzzy set. Assuming that each dimension is allocated with K_i (i = 1, 2, ⋯ , m) fuzzy set, a complete fuzzy rule will contain ∏K_i (i = 1, 2 ⋯ , m) rules. From the perspective of data mining, each sample data contributes to every rule in a complete fuzzy rule library. This article uses the membership degree of data on fuzzy rule antecedents to describe the contribution of data to rules. If there are n sample data and N fuzzy rules in the complete fuzzy rule library, then the contribution of C (i, j) table data i to rule j is (i ⩽ n, j ⩽ N). After obtaining the contribution of each data to each rule, involve all data in determining the outcome of each rule. The contribution of each data is taken as the weight of its real output value to calculate the central value ${\overset{⌢}{y}}_{j}$ (Formula (8)) of the subsequent fuzzy set. ${\overset{⌢}{y}}_{j} = \frac{\sum_{i = n, j = M}^{i = n, j = N} y_{i} \times C (i, j)}{\sum_{i = 1}^{m} C (i, j)}$ (8)

For the three evaluation indicators MAE, MSE, and SMAPE, the smaller the indicator value, the smaller the fitting error of the model and the better the fitting effect. For R², the upper limit of R² is 1. When R² is negative, it indicates that the model does not have fitting ability. When R² is between 0 and 1, the model has a certain fitting ability. The closer R² is to 1, the better the fitting effect of the model. For runtime, the shorter the time, the faster the model is established.

5.2 Experiment and analysis

In order to verify the effectiveness of DFLOWFS, we introduce BP neural network, RBF neural network, LSTM for comparison experiments. Similarly, we use four objective evaluation indicators MAE, MSE, R² and SMAPE for quantitative evaluation, and the four evaluation indicators are comprehensively considered by a comprehensive scoring system. The computing environment is as follows: processor: Intel (R) Core (TM) i7-8550U CPU @ 1.80GHz, 1992 Mhz, 4 cores, 8 logical processors.The proposed algorithm is verified on three datasets of the machine learning database UCI Machine Learning Repository. The three datasets are Bias correction of numerical prediction model temperature forecast Dataset (Bc dataset), Superconductivity Data Dataset (Scd dataset) And Smartphone Dataset (Smartphone Dataset). Evaluate the predicted results based on the average of 20 experimental results. The characteristics of each dataset are summarized in Table 8, of which 70% are used as training set and 30% are used as test set.

Table 8
Summary of dataset features

Dataset Input Output Number of

dimension dimension samples

Bc 20 1 7650

Scd 81 1 21263

Smartphone 560 1 7767

Dataset	Input	Output	Number of
Bc	20	1	7650
Scd	81	1	21263
Smartphone	560	1	7767

5.2.1 Application of Bc dataset

Figure 4(1) shows the error between the predicted results and the true values of DFLOWFS, LSTM, BP neural network, and RBF neural network on the Bc training set. Figure 4(2) shows the error between the predicted results and the true values of DFLOWFS, LSTM, BP neural network, and RBF neural network on the Bc test set. The results show that DFLOWFS is superior to the other three methods.

Fig. 4

Prediction error map of each model on the Bc dataset.

Table 9 shows the evaluation index values of each model on the Bc training set, Table 10 shows the evaluation values of each model on the Bc test set, and Table 11 shows the comprehensive scores obtained by each model on the Bc dataset based on the comprehensive scoring system.

Table 9

Evaluation index values of each model on the Bc training set

	DFLOWFS	LSTM	BP	RBF
MAE	0.05431	0.05700	0.05578	0.05415
MSE	0.00497	0.00566	0.00543	0.00511
R²	0.76051	0.72736	0.73825	0.75382
SMAPE	9.81%	10.36%	10.10%	9.84%

Table 10

Evaluation index values of each model on the Bc test set

	DFLOWFS	LSTM	BP	RBF
MAE	0.05191	0.05629	0.05481	0.05522
MSE	0.00473	0.00554	0.00514	0.00525
R²	0.78100	0.74328	0.76212	0.75702
SMAPE	9.43%	10.36%	10.07%	10.21%

Table 11

Comprehensive scores of each model on the Bc dataset

	DFLOWFS	LSTM	BP	RBF
Training Set	15	4	8	13
Test Set	16	4	12	8

In Table 9, our proposed DFLOWFS is significantly better than the other three in terms of MSE, R² and SMAPE, and it is second only to RBF in MSE with a small difference, with a gap of only 0.000165. It can be seen that DFLOWFS has good learning ability on the Bc training set. From Table 10, DFLOWFS ranks first in all four evaluation indicators, and has the best generalization ability on the Bc test set with the indicators that are significantly ahead of the other three models. Finally, according to the comprehensive scores of each model on the Bc dataset in Table 11, DFLOWFS ranks first in both the training set and the test set, with good performance.

5.2.2 Scd dataset application

Figure 5 (1) shows the predicted results of DFLOWFS, LSTM, BP neural network, and RBF neural network on the training set of Scd and the error of the true values of each algorithm. Figure 5 (2) shows the predicted results of DFLOWFS, LSTM, BP neural network, and RBF neural network on the Scd test set and the error of the true values of each algorithm. The results indicate that the DFLOWFS method is superior to the other three methods.

Fig. 5

Prediction error map of each model on the Scd dataset.

Table 12 shows the evaluation index values of each model on the Scd training set, Table 13 shows the evaluation values of each model on the Scd test set, and Table 14 shows the comprehensive scores obtained by each model on the Scd dataset based on the comprehensive scoring system.

Table 12

Evaluation index values of each model on the Scd training set

	DFLOWFS	LSTM	BP	RBF
MAE	0.06339	0.09085	0.06647	0.06871
MSE	0.00803	0.01340	0.00900	0.00973
R²	0.76148	0.60183	0.73274	0.71110
SMAPE	54.60%	73.64%	56.53%	56.54%

Table 13

Evaluation index values of each model on the Scd test set

	DFLOWFS	LSTM	BP	RBF
MAE	0.06922	0.09420	0.70297	0.071262
MSE	0.01030	0.01431	0.00972	0.010332
R²	0.73139	0.60919	0.72778	0.710592
SMAPE	59.20%	74.65%	57.03%	56.85%

Table 14

Comprehensive scores of each model on the Scd dataset

	DFLOWFS	LSTM	BP	RBF
Training Set	16	4	12	8
Test Set	13	4	13	9

First of all, from the training set, it can be seen from Table 12 that DFLOWFS also ranks first in all four evaluation indicators, especially in R² and MAE has obvious advantages. Then from the test set, according to Table 13, it can be seen that DFLOWFS ranks first in MAE and R² two indicators, and ranks second in MSE index, and the gap between it and the first-ranked BP is only 0.000586, which is very small. Finally, from the comprehensive score, it can be seen from Table 14 that DFLOWFS ranks first in the training set and has the best ability to learn data rules from the data. DFLOWFS and BP are tied for the first place in the test set, both have good predictive ability for unknown data, and DFLOWFS is a rule-based model compared to the BP neural network model, which has higher interpretability than the BP model. In summary, the DFLOWFS method has good generalization ability and can better predict unknown data, reducing the complexity of the model and preventing overfitting of the model.

5.2.3 Smartphone dataset application

Figure 6(1) shows the error between the predicted results and the true values of DFLOWFS, LSTM, BP neural network, and RBF neural network on the Smartphone training set. Figure 6(2) shows the error between the predicted results and the true values of DFLOWFS, LSTM, BP neural network, and RBF neural network on the Smartphone test set. The results show that DFLOWFS is superior to the other three algorithms.

Fig. 6

Prediction error map of each model on the Smartphone dataset.

Table 15 shows the evaluation index values of DFLOWFS and the other three algorithms on the Smartphone training set, Table 16 shows the evaluation values of each model on the Smartphone test set, and Table 17 shows the comprehensive scores obtained by each model on the Smartphone dataset based on the comprehensive scoring system.

Table 15

Evaluation index values of each model on the Smartphone training set

	DFLOWFS	LSTM	BP	RBF
MAE	0.01741	0.04692	0.03821	0.02055
MSE	0.00010	0.00496	0.00178	0.00089
R²	0.99947	0.74211	0.90735	0.95377
SMAPE	5.70%	11.95%	9.66%	6.42%

Table 16

Evaluation index values of each model on the Smartphone test set

	DFLOWFS	LSTM	BP	RBF
MAE	0.03234	0.04735	0.03900	0.023646
MSE	0.00116	0.00499	0.00187	0.001226
R²	0.94047	0.74389	0.90431	0.937078
SMAPE	7.01%	12.50%	10.31%	7.39%

Table 17

Comprehensive scores of each model on the Smartphone dataset

	DFLOWFS	LSTM	BP	RBF
Training Set	16	4	8	12
Test Set	15	4	8	13

Compared with the first two datasets, the dimension of Smartphone dataset is larger, reaching 560 dimensions, but DFLOWFS still has better performance. According to Table 15, it can be seen that the value of DFLOWFS R² on the training set reaches 0.99, which fits the data well. Compared with other models, DFLOWFS is better than other models in all indicators, and has obvious advantages. From Table 16, DFLWOFS significantly outperforms other models on MSE, R² and SMAPE on the test set. Finally, from the comprehensive score in Table 17, the comprehensive score of DFLOWFS ranks first in both the training set and the test set, with good learning ability and generalization ability.

6 Conclusions

Artificial intelligence has gradually evolved from expert systems to data-driven algorithms, overcoming the limitations of expert knowledge caused by the increasing amount of information with social development [24]. Fuzzy system is a highly interpretable artificial intelligence method, but its theoretical research is not mature enough. The current methods have the problem that the accuracy is not high enough, and the low accuracy will inevitably affect the reliability of decision-making [25]. This paper proposes a deep fuzzy system DFLOWFS built layer by layer from bottom to top based on the fuzzy system FLOWFS optimized by rule weights, and optimizes the structure of the deep fuzzy system from two aspects. The three main innovations of the DFLOWFS algorithm are as follows:

By giving rule weights, the fuzzy system is optimized constructively, and the least square method, a numerical optimization method, is innovatively introduced to train the weights. At the same time, regularization terms are added in the training process, which inhibits the reduction of the training time of the model overfitting model, and improves the accuracy of the model.

The initial order of data input features will affect each sub fuzzy system, thereby affecting the performance of deep fuzzy systems. Therefore, we propose two other partitioning strategies: random partitioning and correlation partitioning. The experiment shows that DFLOWFS using correlation partitioning strategy performs best on both the training and testing sets, and has good learning and generalization abilities; The random partitioning strategy is second only to correlation partitioning in various indicators, as the introduction of randomness also has good fitting effects.

A submodule discarding method was proposed, which selectively discards each layer’s submodules, greatly improving the model’s generalization ability and enabling better prediction of unknown data. On the other hand, the submodule discarding method reduces the complexity of the model and prevents overfitting of the model by discarding some submodules with poor performance.

Footnotes

Acknowledgments

The main idea of this paper dates back to 10 years ago, when Dewang Chen, the corresponding author of this paper, was visiting University of California at Berkeley as a visiting scholar of Prof. Lotfi Zadeh, father of fuzzy logic and member of Academy of Engineering of USA. In the discussion during the serial seminars named as “Interpretability vs Accuracy” hosted by Prof. Zadeh, the idea of interpretable fuzzy modeling occurred to Chen’ mind. After long-time thinking, coding and writing, this article was formed. We would like to express our gratitude to the late Prof. Zadeh, who inspired us to pursue interpretable AI, not just high-accuracy AI.

This work is jointly supported by the National Natural Science Foundation of China under Grant 61976055, the Special Fund for Education and Scientific Research of Fujian Provincial Department of Finance under Grant GY-Z21001, and Scientific Research Foundation of Fujian University of Technology under Grant GY-Z22071.

References

Chiu

S.L.

, Fuzzy model identification based on cluster estimation[J], Journal of Intelligent and Fuzzy Systems 2(3) (1994), 267–278.

Mukherjee

, Matam

, Shu

, Maglaras

, Ferrag

M.A.

, Choudhury

and Kumar

, Security and privacy in fog computing: Challenges, IEEE Access 5 (1930), 3–4, doi: 10.1109/ACCESS.2017.2749422.

Bhavsar

, Safro

, Bouaynaya

, Polikar

and Dera

, Machine Learning in transportation data analytics, In Data Anal. Intell. Transp. Syst. (2017), 283–307.

Wang

L.X.

and Mendel

J.M.

, Fuzzy basis functions, universal approx.-imation, and orthogonal least-squares learning, IEEE Trans. Neural Netw 3(5) (1992), 807–814, doi: 10.1109/72.159070.

Zheng

, Byeungwoo

, Xu

et al. Image segmentation by generalized hierarchical fuzzy C-means algorithm[J], Journal of Intelligent & Fuzzy Systems 28(2) (2015), 4024–4028.

Wang

L.X.

and Mendel

J.M.

, Generating, Fuzzy rules by learning from examples, IEEE Trans. Syst., Man, Cybern. B 22(6) (1992), 1414–1427, doi: 10.1109/21.199466.

Wang

L.X.

, The WM method completed: a flexible fuzzy system approach to data mining, IEEE T. Fuzzy Syst. 11(6) (2003), 768–782, doi: 10.1109/TFUZZ.2003.819839.

Yamakawa

, A fuzzy inference engine in nonlinear analog mode and its application to a fuzzy logic control, IEEE T. Neur. Net. 4(3) (1993), 496–522, doi: 10.1109/72.217192.

Gou

, Hou

, Chen

et al. Improving Wang– Mendel method performance in fuzzy rules generation using the fuzzy C-means clustering algorithm[J], Neurocomputing 151 (2015), 1293–1304.

10.

Zhang

, Tao

and Gao

, A new approach of takagi-sugeno fuzzy modeling using an improved genetic algorithm optimization for oxygen content in a coke furnace[J], Industrial & Engineering Chemistry Research 55(22) (2016), 6465–6474.

11.

Harifi

, Khalilian

, Mohammadzadeh

et al. Optimizing a neuro-fuzzy system based on nature-inspired emperor penguins colony optimization algorithm[J], IEEE Transactions on Fuzzy Systems 28(6) (2020), 1110–1124.

12.

Liu

, Sekh

, Quek

et al. RS-HeRR: a rough set-based Hebbian rule reduction neuro-fuzzy system[J], Neural Computing and Applications 33 (2021), 1123–1137.

13.

Dong

, Chen

W.Z.

, Xu

X.B.

et al. Determination method of fuzzy membership function based on separability measure, Control and Decision 00(11) (2014), 2089–2093, doi: 10.13195/j.kzyjc.2013.0965.

14.

Sanchez

, Melin

and Castillo

, Fuzzy system optimization using a hierarchical genetic algorithm applied to pattern recognition. Intelligent Systems’ 2014, Springer, Cham, pp. 713–720, (2015), doi: 10.1007/978-3-319-11310-4_62.

15.

Liu

, Sekh

, Quek

et al, RS-HeRR: a rough set-based Hebbian rule reduction neuro-fuzzy system, Neural Comput. Appl. 33 (2020), 1123–1137, doi: 10.1007/s00521-020-04997-2.

16.

Mehrsai

et al., Using metaheuristic and fuzzy system for the optimization of material pull in a push-pull flow logistics network, Math. Probl. Eng. (2013), 130–137. doi: 10.1155/2013/359074.

17.

Haaland

D.M.

and Thomas

E.V.

, Partial least-squares methods for spectral analyses. 1. Relation to other quantitative calibration methods and the extraction of qualitative information, Anal. Chem. 60(11) (1988), 1193–1202, doi: 10.1021/ac00162a020.

18.

Lee

S.J.

, Chen

et al., A neuro-fuzzy system modeling with self-constructing rule generationand hybrid SVD-based learning, IEEE T. Fuzzy Syst. 11(3) (2003), 341–353, doi: 10.1109/TFUZZ.2003.812693.

19.

Huang

, Chen

and Zhao

, Deep fuzzy system algorithms based on deep learning and input sharing for regression application, Int. J. Fuzzy Syst. 23(3) (2021), 727–742, doi: 10.1007/s40815-020-00998-4.

20.

Boland , et al. Comparing partition and random testing via majorization and Schur functions, IEEET. Software Eng. (2003), doi: 10.1109/TSE.2003.1166591.

21.

Copenhaver

D.P.

, An improved sequentially rejective bonferroni test procedure, Biometrics 43(2) (1987), doi: 10.2307/2531823.

22.

Xie

et al., Variational autoencoder bidirectional long and short-term memory neural network soft-sensor model based on batch training strategy, IEEE T. Ind. Inform. 99 (2020), 1–1, doi: 10.1109/TII.2020.3025204.

23.

Yang

and Paindavoine

, Implementation of an RBF Neural Network on Embedded Systems: Real-Time Face Tracking and Identity Verification, IEEE T. Neural Networ. 14(5) (1162), doi: 10.1109/TNN.2003.816035.

24.

Jin

, Ying

and Li

, Gas path fault diagnosis of gas turbine engine based on knowledge data-driven artificial intelligence algorithm, IEEE Access (2021), doi: 10.1109/ACCESS.2021.3101647.

25.

Yager

R.R.

and Filev

D.P.

, Generation of fuzzy rules by mountain clustering[J], Journal of Intelligent and Fuzzy Systems 2(3) (1994), 209–219.

Step 1:	Data is normalized and divided into training set and test set.
Step 2:	Determine the antecedent parameters of fuzzy rules in a complete fuzzy rule base. (1) Set the number of fuzzy sets for each dimensional input space. (2) Divide the input space equally, and determine the membership function form corresponding to each fuzzy set in the input space. (3) Determine the antecedents of all fuzzy rules based on the characteristics of the full arrangement.
Step 3:	Determine the consequences of fuzzy rules in a complete fuzzy rule library. (1) Using all the data and the contribution of the data to the rule, determine the output center point of the consequent fuzzy set. (2) Using all the data and the contribution of the data to the rule, determine the output center point of the consequent fuzzy set. (3) Determine the membership function form corresponding to the fuzzy set in the consequent.
Step 4:	Training rule weight w. (1) Set the regularization coefficient α (α ≠ 0). (2) Use the least square method to train all rule weights w.
Step 5:	After constructing the fuzzy system model, predict the test set.

Step 1:	Set the window size w and sliding step s for one-dimensional convolution of data dimensions.
Step 2:	Divide the dataset into multiple sub datasets based on parameters w and s. For those with insufficient window size, random filling method is used to fill in the gaps.
Step 3:	Train each sub dataset using the FLOWFS method (Table 1) to construct a sub fuzzy system.
Step 4:	(1) If the final layer of fuzzy system has been built, go to step 5. (2) If you have not yet built to the last layer, combine the outputs of this layer in order as input to the next layer, and go to step 2.
Step 5:	The deep fuzzy system has been constructed and can be used for prediction.

Step 1:	Set the window size w, sliding step s, and drop threshold prob for one-dimensional convolution of data dimensions.
Step 2:	According to the parameters w and s, the dataset is divided into multiple sub datasets using a “correlation partitioning” strategy. For those with insufficient window size, a random filling method is used to fill in the gaps.
Step 3:	Train each sub dataset using the FLOWFS method (Table 1).
Step 4:	Calculate the R² of the sub-fuzzy system, discard the sub-modules below the discarding threshold prob, and keep them otherwise.
Step 5:	(1) If the fuzzy system has been constructed to the last layer, go to step 5. (2) If the last layer has not been constructed, combine the outputs of this layer in order as the input of the next layer, and go to step 2.
Step 6:	The deep blur system is built and can be predicted.

DFLOWFS: Deep Fast-Learning Optimal Weight Fuzzy System with bottom-up hierarchical structure for high-dimensional data regression

Abstract

Keywords

1 Introduction

2 FLOWFS: Fast learning-algorithm with optimal weights for fuzzy systems

4.1 High-dimensional input partition strategy

Table 8 Summary of dataset features Dataset Input Output Number of dimension dimension samples Bc 20 1 7650 Scd 81 1 21263 Smartphone 560 1 7767

Footnotes

Acknowledgments

References

Table 8
Summary of dataset features

Dataset Input Output Number of

dimension dimension samples

Bc 20 1 7650

Scd 81 1 21263

Smartphone 560 1 7767