Pruning fuzzy neural networks based on unineuron for problems of classification of patterns

Abstract

This paper presents a learning algorithm for fuzzy neural networks based on unineurons able to generate interpretation provided by the model through fuzzy rules. The learning algorithm is based on ideas from Extreme Learning Machine, to achieve a low time complexity, and pruning method based on F-scores resulting in accurate models using low complexity resources, using only training data in a single step. Experiments considering binary pattern classification are detailed. Results and statistical evaluation suggest the suggested approach as a promising alternative for pattern recognition with a good accuracy and some level of interpretability through a process of pruning performed in simple steps.

Keywords

Fuzzy neural networks fuzzy systems F-Scores pattern classification

1 Introduction

The intelligent models capable of unifying the predictive capacity of artificial neural network training algorithms and the possibility of linguistic interpretation of the model results through the fuzzy systems [1] allowed the creation of fuzzy neural networks. The main contribution of these models is the interpretability of the results obtained through a set of fuzzy rules extracted from the network structure in the training stages of the model [2]. Over the last few decades, fuzzy systems [1] and their hybrid derivations have been shown to be able to simulate the typical human reasoning ability in a computationally efficient way. An important area of current research is the development of such systems with a high level of flexibility and autonomy in order to evolve their structures and knowledge based on changes in the environment, being able to handle modeling, control, prediction and pattern classification in an environment not unmoved, susceptible to constant changes.

The fuzzy neural networks differ from artificial neural networks in the use of neurons, which are used respectively fuzzy and artificial neurons. Examples of fuzzy neurons are and and or neurons [10]. The fuzzy neural network training algorithms stand out in well-defined steps within their layers. In these models, most of the time the first layer is responsible for fuzzyfication input data. Already the next model layers are able to perform many machine learning tasks using techniques and models of neural networks. Finally, an artificial neuron is responsible for presenting the results obtained by fuzzy neural network. These networks have been shown to be highly promising in various application areas, such as fuzzy clustering, modeling of nonlinear dynamic systems, fault detection and diagnosis, among others. Thus, these networks present themselves as strong candidates to be used as the basis for the development of intelligent systems. The most commonly used methods for structure definition are clustering [2 , 9], evolutionary optimization [11 –13] and membership function equally spaced [4]. Once the network structure is defined, free parameters are estimated. A number of distinct methods have already been used in this step including reinforcement learning [2 , 6], gradient based methods [12, 14], genetic algorithms [8], least squares [5, 9] and Extreme Machine Learning [4 , 25].

Regarding network structure optimization, the two most commonly used methods may present significant deficiencies especially regarding the amount of neurons created to solve the pattern classification problems. Approaches that create a high amount of fuzzy neurons may impair the final response of the network because unnecessary information is present in these neurons.

This paper proposes a novel learning algorithm for fuzzy neural networks able generate compact and accurate models. The learning is performed using ideas from Extreme Learning Machine [16] as in [4], to speed-up parameter tuning, and regularization theory [17]. In the model developed in [4] is used the concept of regularization by resampling, which is an excellent formula to improve the model response, but has four variables that need to be defined to fit the model. This paper it is proposed to use, an approach using F-scores [18] is used to define the network topology by selecting a subset of the candidate neurons. Finally, the remaining network parameters are estimated through least squares. The resulting network is a sparse model and can be expressed as a compact set of incomplete fuzzy rules, that is, rules with antecedents defined using only a fraction of the available input variables [15] making it possible to reduce the complexity of the model by eliminating two variables. In this case they must be chosen the percentage of selected neurons and the number of membership functions for each feature of the database used in themodel.

The paper is organized as follows. Next section reviews the necessary basic concepts about fuzzy logic neurons and fuzzy neural networks. Section 3 details the proposal new learning algorithm able to pruning less significant neurons for the model, besides the concepts about the F-scores. Section 4 presents the experimental results for pattern classification problems and comparison with alternative classifiers based on statistical tests. Finally, the conclusions and further developments are summarized in Section 5.

2 Fuzzy neural networks

2.1 Fuzzy logic neurons

Logical neurons are functional units that add logical aspects of processing with learning capacity. They can be seen as multivariate non-linear transformations between unit hypercubes, or [0, 1] ^N → [0, 1] [1], where N is the number of inputs.

Thus, the and and or neurons aggregate the input signals (membership values) a = [a₁, a₂ … a₃] by initially combining them individually with their weights w = [w₁, w₂ … w₃], a, w ∈ [0, 1] n and then globally combine these results asfollows: $h = OR (a, w) = S_{i = 1}^{n} (a_{i} {tw}_{i})$ (1) $h = AND (a, w) = T_{i = 1}^{n} (a_{i} {sw}_{i})$ (2) where T and t are t-norms and S and s are s-norms.

The neuron or is interpreted as a logical expression that performs a t-norm with inputs and weights, and then to perform a global aggregation with the results using an s-norm. The neuron and acts in the opposite directionusing an s-norm with the inputs and the weights and t-norm with the results obtained. The activation functions φ_and and φ_or can, in general, be nonlinear mappings. In this paper φ_and (ξ) = φ_or (ξ) = ξ, i.e., they are defined as the identityfunction [4].

In order to improve the capacity of the fuzzy neurons, it was decided to unite characteristics of neurons and and or in a single logical relationship. This structure is known as uninorm [27].

A uninorm is a binary operator U : [0, 1] ² → [0, 1] which is commutative, monotonic, associative, and such that there exists an element u ∈ [0, 1] called the absorption element that satisfies NU (u, x ; u) = u, for all x ∈ [0, 1]. Unineuron can be seen as an extension of logical neurons and and or [1] in which a uninorm is a generalization of t-norms and s-norms [8], is used in the operations of weighting and aggregation [1]. The uninorm used in this paper is expressed as follows [27]:

$U (x, y) = {\begin{matrix} gT (\frac{x}{g}, \frac{y}{g}), if x, y \in [0, g] \\ g + (1 - g) S (\frac{x - g}{1 - g}, \frac{y - g}{1 - g}), if x, y \in [g, 1] \\ max (x, y) or min (x, y), otherwise \end{matrix}$ (3) where T and t are t-norms and S and s-norms.

The unineuron adopted in this work follows the same concept proposed in [9] that performs the following operations to calculate its output:

each pair (a_i, w_i) is transformed into a single value b_i = q (a_i, w_i);

we calculate the unified aggregation of the transformed values U (b₁, b₂, … b_n), where n is the number of entries. The function h is responsible for transforming the inputs and corresponding weights into individual transformed values [9]. A formulation for the q function can be described as [9]:

q (w, a) = wa + wu

(4) using the weighted aggregation reported in (4) we can write the unineuron as [8]:

z = UNI (w, x; g) = U_{i = 1}^{n} h (w_{i} x_{i})

(5)

2.2 Fuzzy neural networks

Fuzzy neural networks are neural networks of fuzzy neurons [2]. These networks have as the main technical features the synergic collaboration between the fuzzy and neural networks theory generating models that integrate the treatment of the uncertainty and interpretability provided by fuzzy systems and the learning ability provided by neural networks [2]. This type of model has many applications in industry [7] and computer systems, allowing standards to be discovered, series forecasts are made easier and accurate responses are delivered to its users.

The fuzzy logic neurons described in the previous section can be used to construct fuzzy neural networks and solve pattern recognition problems. Figure 1 illustrates the feed forward topology of the fuzzy neural networks presented in [4], which will also be considered in this paper.

Fig.1

Feedforward fuzzy neural network.

In the first model layer neurons are of the type formed by Gaussian membership functions created with the model input data different from the model proposed in [4], which uses triangular functions. They are the activation functions of the corresponding neurons. For each input variable x_ij, K fuzzy sets are defined A_k, k = 1 … K. Thus, the outputs of the first layer are the membership degrees associated with the input values, i.e., a_jk = μ_A^k. for j = 1 …, N and k = 1, …, K, where N is the number of inputs and K is the number of fuzzy sets for each input. The second layer is composed of L unineurons. Each unineuron has the ability to perform a weighted aggregation of some of the outputs of the first layer with the weights w_il (for i = 1 … N and l = 1 … L) of the logical neuron. For each input variable j, only one first layer output a_jk is defined as input of the l-th neuron. Furthermore, in favor of generating sparse topologies, each second layer neuron is associated with only n_l < n input variables, that is, the weight matrix w is sparse [4].

Finally, in the same way as in [4], the output layer uses a classic linear perceptron neuron to compute the network output: $y = sign (\sum_{j = 0}^{l} h_{j} v_{j})$ (6) where h_l for l = 1, …, L are the outputs of second layer neurons, v_l are the output layer weights and h₀ = 1.

Figure 1 illustrates an example of a fuzzy network composed by unineurons. This network has 2 input variables, 3 fuzzy sets for each variable and 3 unineurons, i.e., M = 2, K = 3 and L = 3. The following if-then rules can be extracted from the network structure [4]: $\begin{matrix} Rule 1 : If x_{il} is A_{1}^{1} with certainty w_{11} \dots \\ and x_{i 2} is A_{2}^{1} with certainty w_{21} \dots \\ Then y_{1} is v_{1} \end{matrix}$

$\begin{matrix} Rule 2 : If x_{il} is A_{1}^{2} with certainty w_{12} \dots \\ and x_{22} is A_{2}^{2} with certainty w_{22} \dots \\ Then y_{2} is v_{2} \\ Rule 3 : If x_{i 2} is A_{3}^{1} with certainty w_{13} \dots \\ and x_{i 2} is A_{3}^{2} with certainty w_{23} \dots \\ Then y_{3} is v_{3} \end{matrix}$ (7) where each free parameter v_l for l = 1, …, 3 can be interpreted as a singleton [9].

3 Network training algorithm

3.1 Fuzzy neural network architecture and training

Fuzzy neural network architecture can be done in a number of ways, as has been reported in the literature, but each of these applications can vary greatly with the size of the test sample or even with the complexity of the database. The greater the number of dimensions of a problem, the more complex it is for a neural network algorithm to make adjustments to the internal parameters of the network.

In the model proposed in [4] the adjustment of the parameters occurs over the concepts of Extreme Learning Machine [19]. The algorithm assigns random values for the first layer weights and analytically estimates the output layer weights. When compared with traditional methods for SLFNs learning, this algorithm has good generalization performance and extremely low time complexity, since only the output layer parameters are tuned [19], especially when the database used for the training has unnecessary information discarded, so the use of regularization methods [21] are necessary so that the responses of the model are much closer tothe real.

Input domain partition using a number of equally spaced membership functions that are chosen at the time of network training [4]. This type of approach was first used in [9] and extensively improved in [4]. This approach assigns random values for the neuron parameters and estimates the parameters of the output layer neuron using least squares. In [9] the learning algorithm has a low computational cost, but still generates networks which are not easily interpretable, since the fuzzy sets are still defined using clustering. Already in [4] the regularization by resampling improves the decision space of the model, improving the accuracy of the pattern classification, however in tests with large samples or dimensions the resampling can be slow or require a lot of computingprocessing.

This paper proposes an improvement in the learning algorithm described in [4] in order to improve the interpretability of the resulting networks to maintain the interpretability and transparency of the model, allowing the regularization of fuzzy neural network to happen in a single step, simple configuration and understanding. A variable selection algorithm based on f-scores is used to define the network topology. This algorithm is able to generate a sparse topology which can be interpreted as a compact set of incomplete fuzzy rules like what is done by regularization by resampling, but in a simpler way. The proposal learning algorithm initially defines first layer neurons by dividing each input variable domain interval with K fuzzy sets, where K is usually a small number. All input variables are partitioned using the same number of fuzzy sets [4].

Next, L_c candidate neurons are randomly generated, where L_c < L. For each candidate neuron l = 1, …, L_C, first, a random fraction of the input variables is selected [4].

The value n_l represents the number of input variables associated with the l-th neuron. See more about the n_l variable settings in [4] The learning algorithm assumes that the output of a network composed to all L_c candidate neurons can be writtenas: $f (x_{i}) = \sum_{l = 0}^{Lc} v_{l} h_{l} (x_{i}) = h (x_{i}) v$ (8) where v = [v₀, v₁, v₂, …, v_L] ^T is the output layer weight vector and h (x_i) = [h₀, h₁ (x_i) , h₂ (x_i) , …, h_Lc (x_i)] is the augmented output (row) vector of the second layer, for h₀ = 1. In this context, h (x_i) is non-linear mapping from the input space to a L_c + 1-dimensional fuzzy feature space, performed using the candidate neurons [4].

Finally, once the network structure is defined, the learning algorithm has only to estimate the output layer vector v = [v₀, v₁, v₂ … v_L] ^T which best adjustment the wanted output. In this paper these parameters are computed using the Moore- Penrose pseudo-inverse: $v = (H^{T} H)^{- 1} H^{T} Y = H^{+} Y$ (9) where H is defined as: $H = [\begin{matrix} h_{0} & h_{1} (x_{1}) & h_{2} (x_{1}) & \dots & h_{L} (x_{1}) \\ h_{0} & h_{1} (x_{2}) & h_{2} (x_{2}) & \dots & h_{L} (x_{2}) \\ ⋮ & ⋮ & ⋮ & \dots & ⋮ \\ h_{0} & h_{1} (x_{N}) & h_{2} (x_{N}) & \dots & h_{L} (x_{N}) \end{matrix}]$ (10)

The data projection in (10) of the fuzzy neural network can be interpreted as a characteristic extraction procedure, since it generates a new set of characteristics, represented by the columns of the matrix H, from the combination and transformation of the characteristics of the matrix of input X [28].

3.2 Pruning method in the architecture of fuzzy neural networks using F-scores

In this context, a lot of L neurons can bring the model some information that have high correlation, producing redundant and unnecessary information for the final output of the fuzzy neural network, and may even harm their predictive ability to perform pattern classification. The approach used in this paper to solve the problem of correlation in the fuzzy neural network is to select the characteristics most relevant to the model, those that contain maximum discriminative information about the classes. For this, we will use the F-Scores concept [16] that is capable of evaluating the discriminating power of the variables of a set of characteristics. In this context, we used the concept of F-scores in the second layer of fuzzy neural network seeking to discard neurons that have smaller F-score than a defined threshold value.

Given the output s_j = [s_1j, …, s_n+j, …, S_Nj] ^T of the j-th fuzzy neuron, with number of instances belonging to the positive class equal to n₊ and total number of instances equal to N, the corresponding F-score is defined by: $F_{j} = \frac{{(μ_{j}^{(+)} - μ_{j})}^{2} + {(μ_{j}^{(-)} - μ_{j})}^{2}}{σ_{j}^{2 (+)} + σ_{j}^{2 (-)}}$ (11) $σ_{j}^{2 (+)} = \frac{1}{n_{+} - 1} \sum_{i - 1}^{n_{+}} {(s_{ij} - μ_{j}^{(+)})}^{2}$ (12) $σ_{j}^{2 (-)} = \frac{1}{N - n_{+} - 1} \sum_{i - n_{+} - 1}^{n_{+}} {(s_{ij} - μ_{j}^{(-)})}^{2}$ (13) where μ_j, $μ_{j}^{(+)}$ and $μ_{j}^{(-)}$ are the means of the total, positive and negative instances, respectively and s_ij is the i-th instance of the output of the j-th fuzzy neuron. In this context, the calculation of the F-score will be used in the columns of Equation (10). In order to determine which elements will be selected within this approach, the criterion will be adopted using [29] where the sj variables are smaller than the average of all the f-scores found. This approach will allow a choice of the most significant neurons (Ls) participating in the fuzzy neural network response.

Therefore Equation (8) is now calculated by: $f (x_{i}) = \sum_{l = 0}^{Ls} v_{l} h_{l} (x_{i}) = h (x_{i}) v$ (14)

The new learning procedure is summarized in Algorithm 1. The algorithm has two parameters, half of which is used in [4]:

the number of fuzzy sets for input space partition, K;

the number of candidate neurons, Lc;

Algorithm 1- Learning algorithm for Fuzzy Neural Networks
Define K equally spaced fuzzy sets for each input variable
Create LcCandidates neurons
for all N inputs do
Compute the mapping h (x_i) (8)
end for
Select Ls neurons using F-scores pruning (11)
or all N inputs do
Compute the mapping h (x_i) (14)
end for
Estimate the output layer weights (9)

4 Experiments

The fuzzy neural network learning algorithm described in the previous section is evaluated using pattern classification problems. For all experiments in this section, only networks composed to unineurons are considered. All unineurons use the product t-norm and the probabilistic sum s-norm and only Gaussian fuzzy sets are used. The network used in the experiments was named Pruning UNINET. First, a simple toy binary classification problem is considered to illustrate the transparency of the resulting network. Next, the accuracy of the Pruning UNINET is evaluated for benchmark binary classification problems [22].

4.1 Artificial classification dataset

Initially, the transparency of the Pruning UNINET and its ability to literal interpretation of results was evaluated using a toy problem. The artificial data set consists of 40 samples and was formed by two Gaussians (Fig. 2) with covariance equal to the identity matrix and centered on [0,0] and [6,6, 6,6] respectively. In this context, 70% of the samples for training and 30% for the tests will also be used. The algorithm parameters were set to: K = 2,L_c= 4.

Fig.2

Artificial Classification dataset.

The resulting network is able to classify all test set samples correctly and has only 2 neurons, which can be interpreted as the following fuzzy rules: $\begin{matrix} {Rule}_{1} : If x_{i 1} is Small with certainty 0.96 \dots \\ Then y_{1} is - 0.18 \\ {Rule}_{2} : If x_{il} is Big with certainty 0.54 \dots \\ and x_{i 2} is Big with certainty 0.87 \dots \\ Then y_{2} is 0.49 \end{matrix}$ (15)

Considering this artificial problem, their separation in Cartesian space and the answers obtained, we can consider each of the Gaussian membership functions for each input space as literal features, such as small and large. The two dimensions of this problem may be related to the relationship between people who smoke (x) and on the frequency of consumption of alcoholic beverages (y). The model’s answer is the probability of having cancer (yes (blue) and not (red)). As the model was used with K = 2, the initial interpretation for the problem can be seen in Fig. 3 below where small is the first membership function and large to second.

Fig.3

Artificial Classification dataset and membership functions.

The most representative fuzzy rules for the problem are those that represent the largest set of data. We can verify that for the artificial problem, the neuron formed by the two functions of small and large relevance are the most representative for this problem. The literal interpretation of this hypothetical problem can be given by the following rules:

$\begin{matrix} If a person smokes little \\ and consumes little alcohol \\ Then it is not likely to have cancer . \end{matrix}$ (16) $\begin{matrix} If a person smokes a lot \\ and also consumes a lot of alcohol \\ Then is likely to have cancer . \end{matrix}$ (17)

4.2 Benchmark classification datasets

In this section the proposes learning algorithm is evaluated using benchmark binary classification datasets taken from the UCI Machine Learning repository [26]. In order to verify the performance of the Pruning UNINET, datasets with different sizes and input dimensions were used. The Ionosphere (ion) database, which represents measurements of free electrons in the ionosphere, the Pima Indian Diabetes (pid) that brings information collected regarding the presence of signs of diabetes in patients and finally the database Wisconsin Diagnostic Breast Cancer (wbc), containing information on breast cancer diagnoses. The databases provided by the starlog deal respectively with Australian credit (acr), German (gcr) and probability of heart attacks (hea). The main characteristics of these datasets are summarized in Table 1.

Table 1
Specification of binary classification datasets

Dataset #features #(N) #train #test

acr 14 460 230

gcr 20 666 334

hea 13 180 90

ion 33 234 117

pid 8 512 256

wbc 9 455 228

Dataset	#features #(N)	#train	#test
acr	14	460	230
gcr	20	666	334
hea	13	180	90
ion	33	234	117
pid	8	512	256
wbc	9	455	228

All observations with missing values were removed and the output were normalized to be in [−1, 1]. The inputs were normalized to zero mean and unit variance. Two thirds of the data samples were randomly selected for training and theremaining for performance evaluation. For all experiments, the algorithm parameters were set to: K = 3 and the number of candidate neurons, L_c, was set to half of the training set size of each dataset [20].

The performance of the proposed approach was compared with two alternative classifiers. An algorithm that performs the pruning of neurons in ELM based on affinity matrix (AFP-ELM) described in [30], the Pruning UNINET, the R-ANDNET suggested in [4] and a state-of-the art pruning algorithm for ELMs, the OP-ELM [23]. The number of candidate neurons for the OPL-ELM was also set to half of the training set size and sigmoidal neurons were considered. Since some parameters of these models are selected randomly and may affect the final training results, each approach was run 30 times and the average values of the performance indexes used for comparison. ANDNET uses the same value of K and of L_c that is used by Pruning UNINET. The number of replications of bootstrap is 8 and the established consensus threshold is 60%.

4.3 Results of pattern classification tests

Table 2 details the test set accuracy for each dataset. Table 3 details the average number of neurons after training. The results presented in these tables suggest that the Pruning UNINET performs similar than AFP-ELM, ANDNET and OP-ELM in most of the experiments. Further-more, for all experiments, the resulting network has a very simple topology (small number of neurons), if compared with the other models. The average execution times of the algorithms, measured in seconds, are listed in Table 4. Simulations were performed on a Core (TM) 2 duo CPU, 2.27 GHz with 3-GB RAM.

Table 2
Performance comparison for binary classification datasets

Dataset R-ANDNET AFP-ELM OP-ELM P. UNINET

acr 85,44 (0.89) 85.15 (2.23) 84.87(1.65) 86.18 (1.82)

gcr 74.22 (4.13) 84.78 (2.39) 81.66 (2.43) 84.92 (1.78)

hea 78.93 (2.80) 80.78 (2.80) 81.16 (2.67) 78.62 (5.09)

ion 90,17 (2,45) 87.14 (2.12) 86.44 (3.16) 82.43 (2.57)

pid 76.10 (5,64) 75.49 (3.80) 73.24 (2.51) 76.22 (2.14)

wbc 95.65 (1,54) 96.32(0.93) 95.45 (1.31) 95.10 (1.41)

Dataset	R-ANDNET	AFP-ELM	OP-ELM	P. UNINET
acr	85,44 (0.89)	85.15 (2.23)	84.87(1.65)	86.18 (1.82)
gcr	74.22 (4.13)	84.78 (2.39)	81.66 (2.43)	84.92 (1.78)
hea	78.93 (2.80)	80.78 (2.80)	81.16 (2.67)	78.62 (5.09)
ion	90,17 (2,45)	87.14 (2.12)	86.44 (3.16)	82.43 (2.57)
pid	76.10 (5,64)	75.49 (3.80)	73.24 (2.51)	76.22 (2.14)
wbc	95.65 (1,54)	96.32(0.93)	95.45 (1.31)	95.10 (1.41)

Table 3

Number of neurons for binary classification datasets

Dataset	R-ANDNET	AFP-ELM	OP-ELM	P. UNINET
acr	15,12 (7.28)	41.34 (12.28)	22.43 (12.44)	12.40(3.63)
gcr	19.13 (8.65)	21.80 (24.74)	22.50 (22.12)	18.40(8.10)
hea	9.42 (7.09)	13.76 (19.09)	14.00 (12.52)	10.40(4.35)
ion	19.87 (2.89)	18.60 (17.76)	25.50 (8.96)	16.42 (4.21)
pid	8.98 (2,54)	23.70 (22.39)	24.00 (26.37)	6.10 (3.28)
wbc	18.62 (9.76)	38.40 (20.79)	25.87 (15.78)	9.80 (2.44)

Table 4

Average time in seconds of model training

Dataset	R-ANDNET	AFP-ELM	OP-ELM	P. UNINET
acr	32,06 (2.67)	00.18 (0.02)	2.87 (0.99)	14.18 (1.65)
gcr	29.19 (4.45)	8.80 (0.44)	5.12 (0.43)	2.76 (0.95)
hea	33.76 (6.98)	13.76 (2.43)	1.02 (0.32)	1.80 (1.76)
ion	47.33 (4.54)	00.67 (0.12)	‘ 1.26 (0.76)	6.15 (2.65)
pid	08.99 (2.43)	04.56 (0.24)	3.54 (0.42)	1.85 (1.87)
wbc	36.65 (12.16)	6.32 (1.12)	2.89 (0.07)	2.54 (0.87)

We can verify that in general the accuracy of the methods is presented in a very egalitarian way, be-tween the algorithms tested and the bases submitted to the test. Each model was the best classifier in at least one database, but the highlight in this issue was for Pruning UNINET, which stood out in 3 bases where the number of characteristics vary Between 8 to 20. The other models stood out in one of the essays. Despite the greater prominence of Pruning UNINET, it is important to note that the results are very close in most of the trials, which corroborates that the model proposed in the paper maintained the ability to classify binary data.

When evaluating the amount of retained neurons, we highlight the efficiency of the proposed model in per-forming retention of neurons that are significant to the context. The pruned model and the regularized model of fuzzy neural networks played an efficient role in the classification of patterns, maintaining the accuracy indices close to the models evaluated in the test, performing the task with a smaller number of neurons, which allows us to say that the final model was less complex to carry out classification activities.

Another relevant factor is present in the Table 4, where the execution times of each of the algorithms are highlighted. Because the regularized algorithm uses replication to perform neuron retention, its running time is greater than all others. As the model proposed in the article accomplishes the classification of binary patterns with times similar to the compared models, we can conclude that the proposal has a viable execution time for the resolution of problems related to pattern classification.

4.4 Evaluation of test results

In order to confirm the performance of the proposed model, a statistical analysis is performed on the data collected in the pattern classification.The algorithms will be evaluated considering the accuracy as the only factor, where each of the classification bases can be seen as the blocking factor. To perform the statistical tests, the analysis of variance (ANOVA) [31] on the results of each of the groups (algorithm x block factor) is used in the test. In general it is verified that the test has 24 (4 algorithms and 6 bases) groups. Through this test seeks to establish whether the performance of the proposed algorithms in this paper presents average performance equal to algorithms that perform pruning of neurons.

After performing the analysis of variance it was concluded that the null hypothesis must be accepted (p-value of 0.4712) that the accuracy performance of the four algorithms is equal. After validation of the model [31] where we confirmed the normality of the collected data (p-value of 0.8621), the homoscedasticity (p-value of 0.7687) and the independence of the data with the p-value of 0.0871 can be affirmed with 95% certainty that the four algorithms analyzed have the same behavior when performing the binary classification of the six bases evaluated.

5 Conclusion

This paper presented a pruning method for fuzzy neural network models. In the model, the more well defined the group of fuzzy neurons that perform the classification of binary patterns, the greater is the efficiency of the algorithm.

In the tests carried out, we verified that the proposed use of f-scores retained a significant amount of neurons for the problem and in a smaller amount when compared with state-of-the-art models of neurons based on extreme learning Machine training. Compared to the model of fuzzy neural networks regulated by resampling, the accuracy of results are statistically equal and can be obtained by reducing the complexity of the model, reducing the parameters in half and performing tasks with a run-time in much smaller pattern classification than models that use the bootstrap.

In the future, other models can be added to the test, increasing the base and parameterizing the tests with values that can be changed through cross-validation.

The model can be extended to multi-class tests and other methods that correlate the internal layers of fuzzy neural networks. Can also be explored in order to prune unnecessary neurons. Other hypothesis of improvement in the accuracy of the model can be proposed with new fuzzyfication algorithms, which may be based on other fuzzy elements, such as cluster fuzzy algorithms.

References

Zadeh

L.A.

, Fuzzy sets, Fuzzy Sets, Fuzzy Logic, And Fuzzy Systems: Selected Papers by Lotfi A Zadeh (1996),394–432.

Caminhas

, Tavares

, Gomide

and Pedrycz

, Fuzzy sets based neural networks: Structure, learning and applications, Journal of Advanced Computational Intelligence 3(3) (1999),151–157.

Ballini

and Gomide

, Learning in recurrent, hybrid neurofuzzy networks, in IEEE International Conference on Fuzzy Systems (2002), pp. 785–791.

Souza

P.V.C.

, Regularized fuzzy neural networks for pattern classification problems, International Journal of Applied Engineering Research 13(5) (2018),2985–2991.

Bordignon

and Gomide

, Uninorm based evolving neural networks and approximation capabilities, Neurocomputing 127 (2014),13–20.

Hell

, Costa

and Gomide

, Participatory learning in power transformers thermal modeling, IEEE Transactions on Power Delivery 23(4) (2008),2058–2067.

Gobi

A.E.

and Pedrycz

, Logic minimization as an efficient means of fuzzy structure discovery, IEEE Transactions on Fuzzy Systems 16(3) (2008),553–566.

Lemos

, Caminhas

, Gomide

, New uninorm-based neuron model and fuzzy neural networks, in 2010 Annual Meeting of the North American Fuzzy Information Processing Society (NAFIPS) 2010, pp. 1–6.

Lemos

A.P.

, Caminhas

, Gomide

, A fast learning algorithm for uninorm-based fuzzy neural networks, Fuzzy Information Processing Society (NAFIPS), 2012 Annual Meeting of the North American IEEE, 2012, pp. 1–6.

10.

Gomide

, Pedrycz

, Fuzzy Systems Engineering: Toward Human- Centric Computing. NJ, USA: Wiley Interscience, 2007.

11.

Pedrycz

, Heterogeneous fuzzy logic networks: Fundamentals and development studies, IEEE Transactions on Neural Networks 15(6) (2004),1466–1481.

12.

Pedrycz

, Logic-based fuzzy neurocomputing with unineurons, IEEE Transactions on Fuzzy Systems 14(6) (2006),860–873.

13.

Liang

and Pedrycz

, Logic-based fuzzy networks: A study in system modeling with triangular norms and uninorms, Fuzzy Sets and Systems 160(24) (2009),3475–3502.

14.

Pedrycz

and Aliev

R.A.

, Logic-oriented neural networks for fuzzy neurocomputing, Neurocomputing 73(1-3) (2009), 10–23.

15.

Guillaume

, Designing fuzzy inference systems from data: An interpretability-oriented review, Fuzzy Systems, IEEE Transactions on 9(3) (2001),426–443.

16.

Huang

G.-B.

, Chen

and Siew

C.-K.

, Universal approximation using incremental constructive feedforward networks with random hidden nodes, Neural Networks, IEEE Transactions on 17(4) (2006),879–892.

17.

Girosi

, Jones

and Poggio

, Regularization theory and neural networks architectures, Neural Computation 7(2) (1995),219–269.

18.

Chen

Y.W.

, Lin

C.J.

, Combining SVMs with various feature selection strategies, In Feature Extraction (2006),315–324 Berlin: Springer, Heidelberg.

19.

Huang

G.-B.

, Zhu

Q.-Y.

, Siew

C.-K.

, Extreme learning machine: A new learning scheme of feedforward neural networks, in 2004 IEEE International Joint Conference on Neural Networks (IJCNN) vol. 2, 2004, pp. 985–990.

20.

Hastie

, Tibshirani

, Friedman

J.J.H.

, The elements of statistical learning, Springer New York, vol. 1, 2001.

21.

Efron

, Hastie

, Johnstone

and Tibshirani

, Least angle regression, The Annals of Statistics 32(2) (2004),407–499.

22.

Murphy

K.P.

, Machine learning: A probabilistic perspective, The MIT Press, 2012.

23.

Miche

, Sorjamaa

, Bas

, Simula

, Jutten

and Lendasse

, Op-elm: Optimally pruned extreme learning machine, Neural Networks, IEEE Transactions on 21(1) (2010),158–162.

24.

Martinez-Martiez

J.M.

, Escandell-Montero

, Soria-Olivas

, Martin-Guerrero

J.D.

, Magdalena-Benedito

and Gomez-Sanchis

, Regularized extreme learning machine for regression problems, Neurocomputing 74(17) (2011),3716–3721.

25.

Montesino-Pouzols

, Lendasse

, Evolving fuzzy optimally pruned extreme learning machine for regression problems, Evolving Systems 1(1) (2010),43–58.

26.

Asuncion

, Newman

, UCI machine learning repository, 2007.

27.

Calvo

and De

, Baets and J. Fodor, The functional equations of Frank and Alsina for uninorms and nullnorms, Fuzzy Sets and Systems 120(3) (2001),385–394.

28.

Mao

and Jain

A.K.

, Artificial neural networks for feature extraction and multivariate data projection, IEEE Transactions on Neural Networks 6(2) (1995),296–317.

29.

Güneş

, Polat

, and Yosunkaya

Ş.

, Multi-class f-score feature selection approach to classification of obstructive sleep apnea syndrome, Expert Systems with Applications 37(2) (2010),998–1004.

30.

PintoD.,

et al., An affinity matrix approach for structure selection of extreme learning machines, Proceedings Presses universitaires de Louvain (2015),.

31.

Montgomery

D.C.

, Design and analysis of experiments, John wiley & sons, 2017.

Pruning fuzzy neural networks based on unineuron for problems of classification of patterns

Abstract

Keywords

1 Introduction

2 Fuzzy neural networks

2.1 Fuzzy logic neurons

3.1 Fuzzy neural network architecture and training

4.1 Artificial classification dataset

Table 1 Specification of binary classification datasets Dataset #features #(N) #train #test acr 14 460 230 gcr 20 666 334 hea 13 180 90 ion 33 234 117 pid 8 512 256 wbc 9 455 228

5 Conclusion

References

Table 1
Specification of binary classification datasets

Dataset #features #(N) #train #test

acr 14 460 230

gcr 20 666 334

hea 13 180 90

ion 33 234 117

pid 8 512 256

wbc 9 455 228