Applying deep learning in brain computer interface to classify motor imagery

Abstract

Deep-learning (DL) is a new paradigm in the artificial intelligence field associated with learning structures able to connect directly numeric data with high-level patterns or categories. DL seems to be a suitable technique to deal with computationally challenging Brain Computer Interface (BCI) problems. Following DL strategy, a new modular and self-organized architecture to solve BCI problems is proposed. A pattern recognition system to translate the measured signals in order to establish categories representing thoughts, without previous pre-processing, is developed. To achieve an easy interpretability of the system internal functioning, a neuro-fuzzy module and a learning methodology are carried out. The whole learning process is based on machine learning.

The architecture and the learning method are tested on a representative BCI application to detect and classify motor imagery thoughts. Data is gathered with a low-cost device. Results prove the efficiency and adaptability of the proposed DL architecture where the used classification module (S-dFasArt) exhibits a better behaviour compared with the usual classifiers. Additionally, it employs neuro-fuzzy modules which allow to offer results in a rules format. This improves the interpretability with respect to the black-box description. A DL architecture, going from the raw data to the labels, is proposed.

The proposed architecture, based on Adaptive Resonance Theory (ART) and Fuzzy ART modules, performs data processing in a self-organized way. It follows the DL paradigm, but at the same time, it allows an interpretation of the operation stages. Therefore this approach could be called Transparent Deep Learning.

Keywords

Transparent deep learning brain computer interface neuro-fuzzy modular architecture s-dFasArt motor imagery

1 Introduction

Deep Learning (DL) paradigm has become one of the most cited terms in artificial intelligence in the recent years. It contains a set of techniques historically used to represent and store knowledge in artificial devices and with the ability of learning new information. They have joined and transformed to a more automatic way of implementing the learning process. DL expression was coined in 2006, as Schmidhuber mentioned in [37] and goes beyond the classical neural networks to get closer to the human learning process where an abstract concept can be identified directly from the data sensors (human senses). The common features of the algorithms identified as DL could be [3]:

They are composed of several levels or layers of non-linear operations. From particular to more abstract

They must discover abstractions with as little human effort as possible.

They can transfer human knowledge to machine-interpretable form.

Categories of higher level learn from lower level features. This is the way to learn from sensory data.

Depth of architecture refers to the number of levels.

The Fig. 1 shows, in schematic form, the comparison between Deep Learning algorithms and classical pattern classification algorithms.

Fig. 1

Classical Pattern Recognition versus Deep Learning paradigm.

Applications in image processing, speech recognition and other pattern recognition systems have been developed using Support Vector Machines, Deep Long Short-Term Memory, etc. Great acceptance has had the Deep Belief Networks where a stack of Restricted Boltzmann Machines is disposed to improve learning capability.

1.1 BCI applications

BCI systems capture brain signals and decode them with the purpose of interacting with certain devices without any muscular intervention [30]. Most of the BCI mechanisms used for this purpose are based on electroencephalogram (EEG) techniques, where the electrodes detecting the electric potentials caused by the neurons inside the brain are placed on the scalp of the user [25]. This technology has fundamentally been applied to medical and psychological brain research, rehabilitation treatment of motor dysfunction [44]. However, the decrease in the cost of the headsets is closing the gap between the research or medical environment and the households, where applications such as video gaming are starting to pave the way for the adoption of BCI devices by the marketplace. One of the most difficult problems faced by BCIs is called motor imagery and it consist in properly classifying three mental tasks: left hand movement, right hand movement and generation of words beginning with the same random letter. Solving this problem only with non-invasive EEG data involves processing signals that are highly variable, not only between users, but also within a single user depending on daytime or the fatigue level. Artifacts and outliers usually arise increasing the difficulty of this problem.

Some pattern recognition problems can be directly validated by a human being. A typical example is shown in Fig. 2.

Fig. 2

Handwritten characters recognition application.

However, it must be noted that the EEG signals cannot be read and decoded straightaway by nobody, even the most experienced neurologist Fig. 3.

Fig. 3

EEG signals recognition for BCI application.

It is an emerging area, which fits very well with artificial intelligence and more specifically to the DL way of doing. It is able to extract the underlying information in those difficult problems. The BCI problem is solved following several steps and all of them can be carried out by mean of AI methods, increasing the abstract level of representation. DL must avoid the presence of noise or the high dimensional EEG data that characterized the motor imagery problem, as mentioned in [13]: “the successful implementation of deep learning methods for the classification of EEG signals is quite an achievement”. In that paper, recordings from three electrodes (C3, Cz, and C4) and five electrodes (C3 C1, Cz, C2, and C4) are included, and only two states are taken into account.

DL process is applied in [41] to BCI problems using Convolutional Neural Networks and Recurrent Neural Networks with Long Short-Term Memory architecture. It is applied to a non- motor imagery problem (The stimuli are non-overlapping flickering lights from five magenta boxes with frequencies 6.66 Hz, 7.5 Hz, 8.5 Hz, 10 Hz, and 12 Hz. The objective is to distinguish between the different stimuli frequencies by analysing the Steady State Visually Evoked Potential.

In [45] also Convolutional Neural Networks are used to deal with a motor imagery problem using BCI Competition IV dataset [40] for the experimental work.

A Deep Belief Network of three layers of Restricted Boltzmann Machine is used in [25] to compare Power Spectrum Density extraction methods and classifiers to improve the recognition accuracy in different motor imagery tasks (left hand, right hand and foot movements). A survey of DL technology applications to BCI problems, can be found in [1, 33, 47]. The BCI motor imagery case with EEG signals is examined in this paper.

1.2 The Motor Imagery problem in BCI systems

The motor imagery problem is one of the hardest issues in BCI technology. Here, the mental categories are imaging moving some body part, foot, hand, tongue, etc. The brain activation of this task is linked with the activation of the real movement. Therefore, this kind of problem is suitable to implement rehabilitation systems, exoskeleton control, prosthetic limbs [30], etc. BCI Competition [6, 40] is a forum where researchers can discuss the advance of knowledge in this field. To this aim, a challenge problem is proposed for each edition. Nowadays the databases provided in this competition are still used as testbeds in new researches, i.e. in [14] DL is proposed to deal with Motor Imagery problems using BCI Competition Data. Specifically, BCI Competition III Set V [6] dealt with the prediction of the following tasks: “movement intention of the left hand”, “movement intention of the right hand” and “thinking of a word”.

This Competition provided data from three users with three sessions for each one, to be used to build the prediction model. Data from a fourth session was also provided in order to validate the predictions. Data was gathered with a professional EEG (Biosemi) Fig. 4(a) hardware following a protocol:

Experiments were carried out with the user seated and still.

Experiments lasted 4 minutes and each 15 seconds a new mental category was presented in a random way.

The user was asked to think in the mental task presented and then labelled.

A break time of 5 minutes between two sessions was mandatory.

The sampled frequency of EEG signals was 512 Hz.

A criterion was established to allow a fair comparison among the methods presented.

A window of only the last second registered data can be used to compute the next prediction value.

The model must give a prediction value every half a second.

The quality of the proposed model is determined with the comparison of the global success rate.

Several works have been proposed at this Competition using Neural Networks, Common Spatial Patterns, Support Vector Machines, Linear Fisher Discriminant, Gaussian Classifiers [4, 5, 16, 22, 26, 36, 43].

1.3 Adaptive resonance theory

Deep learning implementations typically use multilayer networks. In this paper we propose an alternative based on Adaptive Resonance Theory (ART) because one of the fundamental characteristics of the ART is its modular nature. It provides a series of building blocks with specific functionality that can be used to build complex architectures [19]. The functioning of the brain is based on basic elements (neurons) that interact generating complex structures. The flexibility that the ART architecture provides to define complex structures from basic elements, and even incorporate elements such as fuzzy logic, has made it possible to apply it to multiple problems [12]. In many cases, this type of architecture aims to process data to reach categories of abstract thought [18] specifically its relationship with thought and the generation of motor commands [20].

To deal with the contribution of different outputs in different time instants, a multidimensional Sustained Temporal Order REcurrent (STORE) [27] type module is proposed. It works as a buffer storing the last model predictions. At first it has a flat characteristic meaning that the unit activations, which represent the predictions, do not depend on the memorized sequence. The units activations of the STORE module are connected to the input by an on-center off-surround multiplicative competitive net [24] through an instar module [17]. The labels associated to the mental tasks are the net outputs.

S-dFasArt model [42] is a supervised version of a dynamical Fuzzy Adaptive System ART based (FasArt) model [7] allowing classifying time signals with some characteristics that make it suitable to BCI applications.

The model is able to deal with temporal data, so not only the instantaneous value is taken into account, but also when the data is registered (time order reception).

The model can be interpreted as a fuzzy-logic system, therefore as a rule set among fuzzy sets. This is very efficient way to manage data with uncertainty because fuzzy logic is introduced as a tool to make decisions under uncertainty conditions.

It has an incremental way of learning where the set size is not preselected. The number of rules can grow depending on the learning process. It can handle very large learning sets (big data) and it can update the model with new learning data without affecting the already stored information (stability-plasticity dilemma).

There are no constraints for the number of rule antecedents, so the input dimension is not a problem.

The dynamical feature of S-dFasArt model allows it to filter outliers in data coming from artifacts usually found in biological signals.

The rules are defined from the learning examples, so there is no problem in generating contradictory ones (same antecedents lead to different labels). In this case the output will activate two labels with the same volume without involving the right-labelled rules that are generated from the close data.

The S-dFastArt performance in BCI applications has been compared against other classification methods using the BCI Competition III Set V Database showing the best success rate [8, 28].

1.4 Low cost platforms

To increase the number of BCI systems applications, there must be a price drop in hardware and an improvement in comfort use. With this goal in mind, some devices that could be named as “low cost”, have appeared to expand the scope of people that could use BCI technologies [11, 32]. These are classified as follows:

End-use devices: they are addressed to a final user, usually with a particular objective, for instance helping a user to relax or helping one to concentrate, as Neurosky [23].

Devices of development: they are designed for research and development purposes, as Open BCI [39].

General use devices: they link features of the previous groups. They can be used with closed software given by the manufacturer, but also tools for research and development are supplied. Emotiv EPOC, Fig. 4(b), and Emotiv Insight can be considered in this group.

Fig. 4

Headset comparison (a) Biosemi. (b) Emotiv EPOC.

In this last set of devices of general use, some features must be highlighted:

Easy to use: It is important that they are easy to place, especially with inexperienced users or with little knowledge of it. The flexible structures seem to be better than the scalp helmets.

Adaptability: It has to be adaptable to cope with the different user’s morphologies, different length and density hair ….

Sensor technology: Sensors that need driver gel are non-friendly, the time of the setup increases and it is not comfortable when they are removed. Better are those, which only need saline, and the best are the dry sensors, both flexible and rigid ones.

Data communication: wireless transmission between the measure device and the process equipment is the ideal way for the user.

Sensor position: it is relevant to be able to locate sensors in different places, that is, depending on the problem that is being addressed.

Emotiv EPOC is one of the main low-cost devices, taking into account the previous ideas. It has a fixed structure with diadem shape and plastic arms allowing some flexibility and achieves scalp contact for most users. Primarily it uses dry sensors, but sometimes saline is used to improve conductivity. This does not have any effect on the user’s hair which remains unaffected. This hardware can be bought both with the manufacturer’s software in a closed version, or with a version for developers. This last one permits access to the raw data coming from the sensors.

The mentioned features and the low cost of the Emotiv EPOC system, has led to researchers to consider it for dealing with engineering problems that need EEG signals acquisition. Some papers have been published making comparisons of data quality between Emotiv and professional equipment [2, 15, 21, 34]. Also papers for BCI applications with this device have appeared [10, 29, 31, 46].

Figure 4 shows a Biosemi and an Emotiv team. The advantages in terms of ease of use and preparation time provided by the Emotiv equipment are evident, especially when considering its use by a user not accustomed to this type of technology.

1.5 Contribution

In the 1 section, the areas of interest considered in the work have been shown: BCI, Motor Imagery, ART Theory and low-cost platforms. Based on this, an architecture to be applicable to different BCI problems and that shares the Deep Learning methodology is proposed. It allows a straightforward application from the raw data to the results without previous steps. On the one hand, the proposed architecture is made up of modules developed in the ART and Fuzzy-ART theory and attempt to be self-organizing and therefore does not require an external expert to determine its topological characteristics or parameters. On the other hand, the use of ART and Fuzzy-ART modules allows an interpretation of the functioning of the architecture, which eliminates the black-box characteristic of many Deep Learning proposals. The presented architecture could be named as: Transparent Deep Learning, because it is self-organizing, learns directly from the data and allows an interpretation of how the processing is performed.

1.6 Paper organization

The 2 section shows the proposed architecture as well as the learning methodology following the Deep Learning philosophy. In the 3 section, the capabilities of the proposed architecture are explored showing its self-organization capacity: in the self-adjustment of the parameters, in the determination of the structure and in the automatic selection of the most relevant characteristics of the learning data. Objective results of the performance of the architecture with a set of original data are also provided. Finally, the most relevant conclusions of the work carried out are collected in the 4 section.

2 Deep-learning architecture for BCI challenges

In this Section, a generic architecture is proposed to deal with problems associated to the BCI paradigm. It will allow to connect EEG sensors measures with mental categories, following the Deep Learning spirit shown in the Fig. 1. Working with BCI application and EEG signals, some drawbacks could be stressed:

Spatial variability: Sensor placement is always imprecise and it is not possible to place them again in the same locations in a new experiment.

Variability among users: The anatomical differences of each user due to hair length, head size, and so on, leads to a great level of variability

Variability in the same user: The characteristics of raw data collected in one experiment followed by a second experiment in the same user could be quite different.

Wrong data due to artifacts and high noise level.

Difficulty to label the categories: The learning process data cannot be labelled even by expert people.

High dimensionality of data: EEG devices work with several channels of sensors and with high sample frequencies. This produces large amount of data to be processed in the learning task.

In order to deal with these challenges, a new architecture is presented as shown in Fig. 5. It is a global approach to the problem that, starting from raw data gathered with sensors, leads to high level mental categories. In this figure, an example with three categories associated to three mental categories are considered: “movement intention of right hand”, “movement intention of left hand”, “thinking of a word”. Obviously, there are a lot of mental activities that could be considered: “thinking singing a song”, “mentally rotating an object”, “movement intention of the left-right foot”,…, where the architecture is also appropriated because it is independent from the categories chosen. Some stages are proposed to process the information in a self-organizing way.

Fig. 5

Proposed Architecture.

The proposed architecture uses different modules proposed in the framework of the Adaptive Resonance Theory and Fuzzy Logic Systems. The architecture takes advantage of the different functionality of the modules to process the data directly, in accordance with the Deep Learning paradigm, but at the same time allows an interpretability of its different stages in a model that could be named as Transparent Deep Learning.

The raw data is collected from the sensors on the scalp. These have a poor spatial resolution, so it is not possible to locate them always in the same position and the regularity in the measurement is not assured. To avoid this difficulty, a first processing stage is proposed mapping from a two-dimensional space, where sensors are located, to an input vector of entries by an instar module [17].

To study biological signals (e.g. cochlear implants) a frequency decomposition is a basic tool. So, a second stage is proposed based on dNSP model [9]. It allows making a frequency representation of time signals while maintaining the timing character. This stage increases the problem dimensionality because the number of frequency bands considered will multiply the number of time signals.

The proposed architecture has a multi-model scheme to facilitate the definition (learning) of different models coming from different sessions data, taking into account the intra-user variability. It is also useful to handle inter-user variability. The multi-model scheme has a drawback because it gives several output alternatives that could be valid.

2.1 Learning methodology

The S-dFasArt stage in the architecture proposed in Fig. 5 requires a learning process to build the rule base for the S-dFasArt model. To this aim, the learning sessions collected from each user are used. The structure is different for each user to manage the inter-user’s variability. Four mechanisms are selected to define the multi-model system, as shown in Fig. 6.

Rule generation: S-dFasArt learning algorithm is used to build the rules set from the data coming from each of the considered sessions. The N rules appear as:

IF x₁ IS $A_{1}^{j}$ AND … AND x_n IS $A_{n}^{j}$ THEN LABEL j

Where:

$A_{j}^{1} \dots A_{j}^{n}$ are the fuzzy sets represented in Fig. 6(a).

Parameter adjustment: The parameters (σ y At) associated with the fuzzy character and with the activation speed of the categories that must be adjusted, Fig. 6(b). A learning algorithm is designed for this task. The adjustment of the parameter σ implies the modification of the fuzzy sets that define the rules, which become the fuzzy sets (in red in Fig. 6(b)).

Rule pruning: Inefficient rules are pruned from the models, Fig. 6(c). The red crosses in the figure mean the rules that are discarded.

Dimensionality reduction: Data dimensionality is adjusted, Fig. 6(d). In the figure the fuzzy sets associated with the dimensions that are selected as relevant for the model are framed.

Note that the last three mechanisms (adjustment, pruning and reduction) allow the model to enhance information coming from other sessions, different to the session used to generate the model. This seeks to improve the model’s generalization ability to handle the intra-user variability.

Fig. 6

Learning methodology for S-FasArt models. (a) Rules generation. (b) Parameter adjustment. (c) Rule pruning. (d) Dimensionality reduction.

3 Results

In this paper, the behaviour of the proposed architecture is tested strictly following the constraints, rules, time specifications, sessions and mental categories of the BCI Competition III (Set V). This is well established and formally described in the literature [6]. For that purpose, a new data set has been collected from four healthy male users with ages between 25 and 50 years old. All of them had no experience with BCI system and they were right-handed. The hardware used was Emotiv EPOC and the sampled frequency was 127 Hz. As the Competition determined, a weight matrix based in Laplacian distribution is used to make the first spatial projection. To compute the Power Spectrum Density for the frequency treatment, the Welch method has been applied.

The problem faced includes two mental categories associated with motor imagery (“movement intention of the left hand”, “movement intention of the right hand”), and a third one without motor significance (“thinking in a word”). So, the proposed model can be employed for two purposes: to detect motor intention in a synchronous way (with no prior knowledge of the event time) and to detect the laterality of the motor intention.

An experimental study is carried out to show the solutions that this learning architecture offers, in order to select the right configuration according to the problem faced. To study the architecture performance and to select the learning methodology, only three data sessions (S1,S2,S3) have been used. The fourth one (S4) has been reserved for the validation task.

In reference [8] a S-dFasArt module is used for the classification of the BCI Competition III (Data set V). There, S-dFasArt method obtained the best results with a success rate of 76.07%, which improves significantly the average of 68.75% of the competition winner. In the rules of BCI Competition III (Data set V) a classical methodology with a data pre-processing stage was used.

3.1 Learning and adjustment method

As mentioned above, three data sessions are used for the learning task. Initially, three different models will be generated, one for each learning session, using S-dFasArt [42] architecture. This architecture generates fuzzy rules set which exhibits the data structure used in the learning process 5.

The activation/membership functions are triangular and are determined from three weights. In the j antecedent of the i rule, these weights are (w_ij, c_ij, v_ij).

The units activation and the weight learning are computed with differential equations according to the assumptions of Adaptive Resonance Theory [17]. Once the learning task has been done, the number of the model parameters will be: P = 3 * N * M + N, where N is the number of rules created and M is the input vector dimension, that is to say, the number of antecedents of each rule.

The prediction of S-dFasArt architecture depends on two design parameters: σ y A_t. The first one modifies the support of the fuzzy sets for the rules antecedents that have triangular activation functions. This support is the interval [w_ij - σ, v_ij + σ] for the j-th antecedent of the i-th rule. It regulates the level of the rule fuzziness [7]. The second parameter, A_t, governs the speed of unit’s activation, and therefore the speed the model responds to input changes [7]. Low values of A_t leads to a slow change in system activation state and so, the input value must be persistent in order to produce a change in network weights.

In the other way, high values of A_t carries to quick activation variations when the input changes. Initially, these parameters must be adjusted in an external way, which would make the model not fully self-organized. To avoid this, an algorithm to parameter adjustment using learning techniques with neuronal inspiration, is proposed. It is a variation of the known Perceptron learning rule [35]. In an incremental way it is given by the algorithm described in equation 1 $\begin{array}{l} A t_{n e w} = A t_{o l d} + Δ A_{t} * f (J (A_{t} + Δ A_{t}, σ) - J (A_{t}, σ)) \\ σ_{n e w} = σ_{o l d} + Δ σ * f (J (A_{t}, σ + Δ σ) - J (A_{t}, σ)) \end{array}$ (1)

where f (x) =1 if x > 0 and f (x) =0 in another case. J (A_t, σ) value is related with the model success for a selected test set. This algorithm is executed until there are no variations in parameters. ΔAt and Δσ will be alternated between positives and negatives values, and they are chosen among a discrete set of possible candidates. In a first step, the same session will be used to do the rule learning and the parameter adjustment. This implies that three different models can be generated.

The activation sequence in the upper layer of units in S-dFasArt is connected with a STORE module set. This set, through an Instar module, also feeds an on-center off-surround net. So, the most activated category results as the unique activated category. Then, the label associated to that one is proposed as the prediction output.

Data collected in Table 1 show the large deviation between the success rate when the session is the same used for the learning task and adjustments, from other sessions. The average of the results obtained for the four users that make up the database are collected.

Table 1

A single session for learning and adjustment. Mean values for the four users

	Learning	Success	Success	Success
SaSbSc	Rules	Adjustment
	Sa	Sa (%)	Sb (%)	Sc (%)
S1S2S3	229.00	97.68	38.60	36.68
S2S1S3	223.25	98.90	25.55	33.76
S3S1S2	229.75	98.39	33.60	39.90

This indicates the intra-user variability. To fix this problem, an information mix coming from different sessions is proposed. At first, a session will be used for model generation and a different one will be used for the adjustment task. The goal is that the built rules acquire enough generality to represent two learning sessions. Now, each model will be created starting from two different learning sessions (Sa,Sb) so that:

Session Sa for the learning task

Session Sb for the adjustment task.

Session Sc is not used to generate the model.

With this method, the three learning sessions collected can create six different models. The third session (Sc) is not employed at all to the model’s development, so the success rate for that session will suggest the model generalization ability.

The results shown in Table 2 have been compared with those obtained using the same session for the learning and adjustment task (Table 1). It can be seen that:

The rate of success of the adjustment session is higher when this session has not previously been used for learning than when the same session was used for both learning and adjustment.

The success rate for a session used neither for learning nor for adjustment, increases when different sessions have previously been used for those tasks.

Table 2

Learning and cross adjustment sessions. Mean values for the four users

	Learning	Success	Success
SaSbSc	Rules	Adjustment
	Sa	Sb (%)	Sc (%)
S1S2S3	229.00	46.09	43.00
S1S3S2	229.00	51.61	37.66
S2S1S3	223.25	45.57	46.61
S2S3S1	223.25	52.99	42.94
S3S1S2	229.75	45.41	43.93
S3S2S1	229.75	50.56	40.53

The improvement of using different sessions for learning and adjustment has been maintained. The increase in the success rate for the adjustment session shows that part of the understood information has been incorporated into the model, so that the model learns and adapts to include information from the new evidence (data).

On the other hand, the increase in the success rate of the session that has not been used neither for the learning task nor for the adjustment, manifest that the model generated with the two sessions presents a better generalization character than the one obtained from a single session. Rule variations coming from the adjustment phase make the model suitable to represent a data set with high variability and so, suitable to respond better to the intra-user variability problem.

3.2 Rule selection method

One of the problems we usually face presents wrong labelled data. These generate rules whose predictions can be inconsistent. Examples where the same situation generates different conclusions, have been presented to the system. This is a handicap to learn properly and produce a non-defined output. To solve this, the S-dFasArt architecture incorporates a match-tracking mechanism that incorporates rules that consider the different outputs even if they come from the same antecedents. The problem could persist because there is not an objective way to choose among all of these rules. The solution proposed is to add a rule pruning stage. Here, the rules are evaluated according to their performance and the ones with the best marks are selected. For the assessment task efficiency criteria have been proposed:

It consists of three phases: To learn, to adjust and to prune. Each of them uses one of the available sessions to make the learning task as follows:

Sa session is used for the S-dFasArt learning phase to generate the model.

Sb session is used for the model parameter adjusting phase.

Sc session is used for the pruning phase of the model rules.

Efficiency criteria for the rule pruning: This method gets one of the learning sessions and computes the predictions of each one of the previously defined models. The rule that has driven to each prediction and if it has been right or wrong, is stored.

At the end, a global calculation of the number of times that each rule has led to a right or a to a wrong prediction is carried out. The model will be composed by the rules set with a positive balance in this evaluation.

The results using the efficiency criteria for the rule pruning task are shown in Table 3

Table 3
Efficiency pruning

Learning Success Success Pruning Success Success

SaSbScSd Rules Adjustment Rules Pruning Pruning

Sa Sb (%) Sd (%) Sc Sb (%) Sd (%)

S1S2S2S3 229.00 46.09 43.00 32.50 57.3075 51.3050

S1S2S3S3 229.00 46.09 43.00 39.75 44.0325 56.5975

S1S3S2S2 229.00 51.61 37.66 19.25 50.5200 51.5375

S1S3S3S2 229.00 51.61 37.66 23.75 56.2875 39.9950

S2S1S1S3 223.25 45.58 46.61 33.25 51.9125 43.9675

S2S1S3S3 223.25 45.58 46.61 44.75 42.3200 60.6400

S2S3S1S1 223.25 52.99 42.94 22.00 49.5000 53.5200

S2S3S3S1 223.25 52.99 42.94 37.50 59.4800 43.6050

S3S1S1S2 229.75 45.41 43.93 31.00 52.5000 43.7900

S3S1S2S2 229.75 45.41 43.93 38.25 45.7175 56.2550

S3S2S1S1 229.75 50.56 40.53 17.00 39.0150 49.3125

S3S2S2S1 229.75 50.56 40.53 21.25 57.9425 42.9050

	Learning	Success	Success	Pruning	Success	Success
S1S2S2S3	229.00	46.09	43.00	32.50	57.3075	51.3050
S1S2S3S3	229.00	46.09	43.00	39.75	44.0325	56.5975
S1S3S2S2	229.00	51.61	37.66	19.25	50.5200	51.5375
S1S3S3S2	229.00	51.61	37.66	23.75	56.2875	39.9950
S2S1S1S3	223.25	45.58	46.61	33.25	51.9125	43.9675
S2S1S3S3	223.25	45.58	46.61	44.75	42.3200	60.6400
S2S3S1S1	223.25	52.99	42.94	22.00	49.5000	53.5200
S2S3S3S1	223.25	52.99	42.94	37.50	59.4800	43.6050
S3S1S1S2	229.75	45.41	43.93	31.00	52.5000	43.7900
S3S1S2S2	229.75	45.41	43.93	38.25	45.7175	56.2550
S3S2S1S1	229.75	50.56	40.53	17.00	39.0150	49.3125
S3S2S2S1	229.75	50.56	40.53	21.25	57.9425	42.9050

Some outcomes can be obtained from Table 3:

The success rate for the session used both for the adjustment and pruning tasks increases.

The success rate for the session used for the pruning task but not for the adjustment increases

Therefore, the effect of the pruning task is to include information from the session used, in the same model that was generated with data coming from another session.

3.3 Dimensionality reduction

This method uses a Group Method Data Handling (GMDH) [38] algorithm to select the rules that will constitute the model. A Regularity Criterium (RC) [28] based in the success rate is defined: $RC = \frac{Wright predictions number}{Data number}$ (2)

Initially the model consists of a single variable. With this assumption, RC value is computed for all the possible models that can be generated with a single variable. The model with the highest score is chosen. In the following steps, the number of variables is enlarged adding a new one. Again, RC value is computed for all the possible models and the model with the highest score is chosen. If this mark is better than the previous one, the new variable that has produced it is incorporated into the model. Now, we will try adding a new variable in a new interaction.

The GMDH algorithm is used to reduce the vector dimensionality associated with the antecedent’s space of S-dFasArt model. So, the number of rules antecedents (at first are 168) is decreased.

From Table 4 the conclusions that can be highlighted for the effect of dimensionality reduction are:

The success rate for the session used both for the adjustment and reduction tasks increases.

The success rate for the session used for the reduction task but not for the adjustment increases.

The success rate for the session used both for the adjustment and reduction-pruning tasks increases.

The success rate for the session used for the reduction-pruning task but not for the adjustment increases.

Table 4

Dimensionality reduction

	Learning	Success	Success	Reduction	RC	Success	Success	Pruning	Success	Success
SaSbScSd	Rules	Adjustment		Dimension	Reduction	Reduced	Reduced	Rules	Pruning	Pruning
	Sa	Sb (%)	Sd (%)	Sc	Sc	Sb (%)	Sd (%)	Sc	Sb (%)	Sd (%)
S1S2S2S3	229.00	46.09	43.00	7.50	54.37	54.38	42.45	64.75	60.18	45.61
S1S2S3S3	229.00	46.09	43.00	12.50	58.41	44.13	58.41	70.00	49.80	65.55
S1S3S2S2	229.00	51.61	37.66	9.25	57.97	40.85	57.95	53.25	42.38	62.78
S1S3S3S2	229.00	51.61	37.69	12.25	62.98	62.98	43.37	53.00	71.52	47.89
S2S1S1S3	223.25	45.57	46.61	11.25	60.23	60.23	48.25	68.00	66.20	45.35
S2S1S3S3	223.25	45.60	46.61	9.75	64.90	39.26	64.90	85.00	43.33	71.89
S2S3S1S1	223.25	52.99	42.94	9.00	58.29	48.79	58.29	56.00	45.23	64.10
S2S3S3S1	223.25	52.99	42.94	10.25	62.51	62.51	43.61	66.50	67.85	46.43
S3S1S1S2	229.75	45.41	43.93	6.00	52.28	52.28	45.81	55.75	57.58	45.31
S3S1S2S2	229.75	45.41	43.93	10.00	61.41	42.92	61.41	82.75	45.17	66.06
S3S2S1S1	229.75	50.56	40.53	9.00	60.59	47.38	59.59	51.50	47.95	66.29
S3S2S2S1	229.75	50.56	40.53	8.25	62.32	62.32	40.98	76.75	69.02	41.11

The model ability to predict properly a data session increases if the dimensionality reduction has been done with the same session. Therefore, the information contained in the session used for the reduction task is added to the model. Also, but with a lower degree of certainty, it can be deduced:

The success rate of the session not used, neither for the adjustment nor for the reduction task, increases.

The success rate of the session not used, neither for the adjustment nor for the reduction-pruning task, increases.

The success rate of the session used only in the adjustment task, increases.

These leads to think that from a generic point of view, the dimensionality reduction has improved the model’s generalization ability, and so the capacity to predict different situations from those that have been used for the learning stage.

It must be noted that, in relation to the number of architecture parameters, for each model the initial number was P_i = 3 * N_i * M + N_i where N_i=number of model rules of i-th model and M = dimension of input vector, 168 for this issue. For instance, using the values of Table 2 we can compute:

$P = \sum_{i = 1}^{6} 3 * N_{i} * M + N_{i} = 688820$

For rule pruning and parameter reduction model using the values of Table 4 and with 6 models, the number of parameters are:

$P = 12515$

In the same way with 12 models:

$P = 23405$

It can be seen that in this last case, the number of parameters is 30 times lower with double of the models. The reduction in computational burden is obvious because the lower parameters used, the lower rules activations and belonging values must be computed. This is remarkable in an application with limited hardware.

3.4 Results with the validation session

As it was described in Section 1.2 the BCI Competition III Set V deal with a problem with three mental categories. This allows us to propose several problems to be solved with our architecture.

The problem faced includes two mental categories associated to motor imagery (“movement intention of left hand”, “movement intention of right hand”), and a third one without motor significance (“thinking of a word”). So, the proposed model can be employing for two purposes: to detect motor intention in a synchronous way (with no prior knowledge of the event time) and to detect the laterality of the motor intention.

The first one is to distinguish between motor and none motor thoughts. With this aim, a success rate can be defined as the number of times that a motor category is predicted properly A_M versus the number of motor categories (N_I + N_D), Motor success= $\frac{A_{M}}{N_{I} + N_{D}}$ and the capacity to detect a non-motor task: Word success= $\frac{A_{W}}{N_{W}}$ The second one is the success in distinguishing the laterality taken into account Left success= $\frac{A_{I}}{N_{I}}$ and Right success= $\frac{A_{D}}{N_{D}}$ . The third one is to achieve a global success rate Global= $\frac{A_{I} + A_{D} + A_{W}}{N_{I} + N_{D}, N_{W}}$

The results for this rate with the 4 users (in %) are stored in Table 5. Confusion matrices are shown in Table 6.

Table 5
Dimensionality reduction influence in success rate for each user

User 1 User 2 User 3 User 4

6 Models 12 Models 6 Models 12 Models 6 Models 12 Models 6 Models 12 Models

(%) (%) (%) (%) (%) (%) (%) (%)

Motor 82.64 79.29 81.21 83.37 76.03 78.19 70.47 75.43

Left 73.74 60.34 55.17 60.69 68.28 76.55 25.29 38.51

Right 24.16 29.53 47.98 50.29 19.97 13.79 68.97 75.17

Word 85.33 90.00 59.31 62.67 77.78 70.83 80.69 80.00

Global 61.92 60.04 53.78 57.45 52.70 51.19 56.25 62.93

	User 1	User 2	User 3	User 4
Motor	82.64	79.29	81.21	83.37	76.03	78.19	70.47	75.43
Left	73.74	60.34	55.17	60.69	68.28	76.55	25.29	38.51
Right	24.16	29.53	47.98	50.29	19.97	13.79	68.97	75.17
Word	85.33	90.00	59.31	62.67	77.78	70.83	80.69	80.00
Global	61.92	60.04	53.78	57.45	52.70	51.19	56.25	62.93

Table 6

Confusion matrices. In brackets the data for 12 models

	User 1
	Prediction	Prediction	Prediction
	Left	Right	Word
Real Left	132 (108)	21 (33)	26 (38)
Real Right	78 (59)	36 (44)	35 (46)
Real Word	15 (8)	7 (7)	128 (135)
	User 2
	Prediction	Prediction	Prediction
	Left	Right	Word
Real Left	80 (88)	54 (49)	11 (8)
Real Right	73 (71)	83 (87)	17 (15)
Real Word	31 (35)	28 (19)	86 (91)
	User 3
	Prediction	Prediction	Prediction
	Left	Right	Word
Real Left	99 (111)	6 (5)	40 (29)
Real Right	102 (120)	33 (24)	39 (30)
Real Word	26 (35)	6 (7)	112 (102)
	User 4
	Prediction	Prediction	Prediction
	Left	Right	Word
Real Left	44 (67)	55 (48)	75 (59)
Real Right	11 (10)	100 (109)	34 (26)
Real Word	8 (11)	20 (18)	117 (116)

The well-known performance Cohen’s Kappa index is computed in Table 7 for all the users with the validation session using 6 and 12 models.

Table 7

Cohen kappa value for each user using 6 or 12 models

	6 models	12 models
User 1	0.4213	0.3985
User 2	0.3052	0.3617
User 3	0.3061	0.2850
User 4	0.3557	0.4514

The results collected in Table 5 indicate that the success rate in the prediction of MOTOR patterns is almost the same using 6 models (77.59%) or 12 models (77.59%). Also, the difference in the success rate for the WORD category is not significant: 75.78% for 6 models and 75.88% for 12 models. This behavior is maintained in the global rate: 56.17 % for 6 models compared to 57.90 % using 12 models. It can be inferred that the architecture can extract the most relevant information from data using only 6 models, which simplifies the necessary calculations and clearly reduces the computational cost.

Likewise, the obtained results reflect the ability of the system to discern between data associated with a motor activity and data associated with a non-motor activity. All this, using low-cost, non-invasive capture equipment, and using a generic and self-organizing architecture that does not require specific previous knowledge of the problem.

4. Conclusions

In this paper, a self-organized DL architecture, going from the raw data to the labels, is presented. It has a neurofuzzy scheme and a learning methodology allowing it to deal with different kinds of issues. One of them is the BCI-EEG paradigm, which has some characteristics that make it especially difficult to manage. The proposed system is suitable to face the EEG challenges. On the one hand, the fuzzy character of the showed architecture provides a natural way to processing data with great uncertainty, because the model is able to decode the prediction models as a rule set that joins fuzzy sets with labels associated to mental states. On the other hand, the high dimensionality and the presence of noise are avoided pruning the inefficient rules and adjusting the problem dimensionality.

The proposed learning methodology is arranged in several stages. Firstly, the signals are processed with a dNSP architecture to obtain an initial frequency composition. Secondly, neurofuzzy dFastArt algorithms with a mechanism of parameters adjustment are used to generate multiple models from several learning sets in an autonomous way. Thirdly, a learning method that prunes existing rules is used to erase the relationships generated from incoherent learning data. At last, there is a STORE module and an on-center off-surround network, which constitute a method to gather the evidences and to make the final decisions.

The architecture shares the paradigms of multilayer modularity and self-organization that characterizes the DL. It allows to predict high level mental categories directly from EEG sensor measurements. The modular attribute of the proposed architecture allows to understand how it works, solving the dark “black-box” nature of the DL systems. The activation of the output units of dNSP module fits with the frequency composition of the EEG signals. The work of the neurofuzzy S-dFasArt unit can be interpreted from the If THEM rules applied on fuzzy sets. This feature is even relevant to the rule generation because for each decision the module makes it is possible to determine which rule is using and which examples, taken from the learning set, have carried to the generation of each rule.

To test the proposed DL architecture, a particular BCI problem has been tested. It is the motor imagery issue. Some processing alternatives have been studied. The results prove the efficiency to detect the movement intention thoughts. Although the sensor’s positions on the scalp have not been the most appropriate to predict the laterality problem, and the difficulty of the problem faced, the success rates are very acceptable: Motor 77.59%, Word 75.78%.

A relevant contribution of the proposed architecture is that it allows an interpretation of the data processing, called Transparent-Deep Learning (T-DL). Which means, that it maintains the principles of DL but adding the possibility to interpretate the intermediate steps. Therefore, the results can be seen as the application of a set of rules which are easy to understand.

Footnotes

Acknowledgments

The authors belong to the Neuro-Technology, Control and Robotics (NEUROCOR) research group placed in the Technical University of Cartagena. They want to thank Dr. Juan Antonio Martínez León’s work. We would also like to thank the reviewers for their comments.

References

Aggarwal

and Chugh

, Review of machine learning techniques for eeg based brain computer interface, Archives of Computational Methods in Engineering (2022), 1–20.

Badcock

N.A.

, Preece

K.A.

, de Wit

, Glenn

, Fieder

, Thie

and McArthur

, Validation of the Emotiv EPOC EEG system for research quality auditory event-related potentials in children, PeerJ 3 (2015), e907.

Bengio

, Learning deep architectures for AI,^®, in Machine Learning 2(1) (2009), 1–127.

Bhattacharya

, Bhimraj

, Haddad

R.J.

and Ahad

, Optimization of EEG-based imaginary motion classification using majority-voting. In SoutheastCon 2017, pages 1–5. IEEE, 2017.

Bhattacharya

, Haddad

R.J.

and Ahad

, A multiuser EEG based imaginary motion classification using neural networks. In SoutheastCon 2016, pages 1–5. IEEE, 2016.

Blankertz

, Muller

K.-R.

, Krusienski

D.J.

, Schalk

, Wolpaw

J.R.

, Schlogl

, Pfurtscheller

, Millan

J.R.

, Schroder

and Birbaumer

, The BCI Competition III: Validating alternative approaches to actual BCI problems, IEEE Transactions on Neural Systems and Rehabilitation Engineering 14(2) (2006), 153–159.

Cano-Izquierdo

, Almonacid

, Pinzolas

and Ibarrola

, dFasArt: Dynamic neural processing in FasArt model, Neural Networks. 22 (2009), 479–487.

Cano-Izquierdo

, Ibarrola

and Almonacid

, Improving motor imagery classification with a new BCI design using neuro-fuzzy S-dFasArt, IEEE Transactions on Neural Systems and Rehabilitation Engineering 20(1) (2012), 2–7.

Cano-Izquierdo

J.M.

, Ibarrola

, Pinzolas

and Almonacid

, dNSP: A biologically inspired dynamic neural network approach to signal processing, Neural Networks 21(7) (2008), 1006–1019.

10.

Chen

, James

, Bao

F.S.

, Ling

and Fan

, Relationship between video game events and player emotion based on EEG. In International Conference on Human-Computer Interaction, pages 377–384. Springer, 2016.

11.

Chu

N.N.

, Brain-computer interface technology and development: The emergence of imprecise brainwave headsets in the commercial world, IEEE Consumer Electronics Magazine 4(3) (2015), 34–41.

12.

da Silva

L.E.B.

, Elnabarawy

and Wunsch

D.C.

II , A survey of adaptive resonance theory neural network models for engineering applications, Neural Networks. 120 (2019), 167–203.

13.

Dai

, Zheng

, Na

, Wang

and Zhang

, EEG classification of motor imagery using a novel deep learning framework, Sensors 19(3) (2019), 551.

14.

Dai

, Zheng

, Na

, Wang

and Zhang

, Eeg classification of motor imagery using a novel deep learning framework, Sensors 19(3) (2019), 551.

15.

de Lissa

, Sorensen

, Badcock

, Thie

and McArthur

, Measuring the face-sensitive N170 with a gaming EEG system: a validation study, Journal of Neuroscience Methods. 253 (2015), 47–54.

16.

Galan

, Oliva

and Guardia

, Using mental tasks transitions detection to improve spontaneous mental activity classification, Medical & Biological Engineering & Computing 45(6) (2007), 603–609.

17.

Grossberg

, Nonlinear neural networks: Principles, mechanisms, and architectures, Neural Networks 1(1) (1988), 17–61.

18.

Grossberg

, Attention: Multiple types, brain resonances, psychological functions, and conscious states, Journal of Integrative Neuroscience 20(1) (2021), 197–232.

19.

Grossberg

, Conscious mind, resonant brain: How each brain makes a mind. Oxford University Press, 2021.

20.

Grossberg

, A neural model of intrinsic and extrinsic hippocampal theta rhythms: Anatomy, neurophysiology, and function, Frontiers in Systems Neuroscience 15 (2021), 665052.

21.

Grummett

, Leibbrandt

, Lewis

, DeLosAngeles

, Powers

, Willoughby

, Pope

and Fitzgibbon

, Measurement of neural signals from inexpensive, wireless and dry EEG systems, Physiological Measurement 36(7) (2015), 1469.

22.

Irtiza

and Farooq

, Classification of brain states using subject-specific trained classifiers. University of Engineering and Technology Taxila, Technical Journal 21(2) (2016), 96.

23.

Katona

, Farkas

, Ujbanyi

, Dukan

and Kovari

, Evaluation of the neurosky mindflex eeg headset brain waves data. In 2014 IEEE 12th international symposium on applied machine intelligence and informatics (SAMI), pages 91–94. IEEE, 2014.

24.

Kohonen

, Self-organization and associative memory volume 8. Springer Science & Business Media, 2012.

25.

Lebedev

M.A.

and Nicolelis

M.A.

, Brain–machine interfaces: past, present and future, TRENDS in Neurosciences 29(9) (2006), 536–546.

26.

Lin

C.-J.

and Hsieh

M.-H.

, Classification of mental task from EEG data using neural networks based on particle swarm optimization, Neurocomputing 72(4–6) (2009), 1121–1130.

27.

Maroto

J.P.

, Dimitriadis

Y.A.

, Izquierdo

J.M.C.

and Coronado

J.L.

, Analysis and application of the store neural model in recognizing handwritted symbols. In International Workshop on Artificial Neural Networks, pages 934–941. Springer, 1995.

28.

Martinez-Leon

J.-A.

, Cano-Izquierdo

J.-M.

and Ibarrola

, Feature selection applying statistical and neurofuzzy methods to EEG-based BCI, Computational Intelligence and Neuroscience 54 (2015), 2015.

29.

Mazzoleni

and Previdi

, A comparison of classification algorithms for brain computer interface in drug craving treatment, IFAC-PapersOnLine 48(20) (2015), 487–492.

30.

Millan

J.D.R.

, Rupp

, Muller-Putz

, Murray-Smith

, Giugliemma

, Tangermann

, Vidaurre

, Cincotti

, Kubler

, Leeb

et al. Combining brain-computer interfaces and assistive technologies: state-of-the-art and challenges, Frontiers in Neuroscience 4 (2010), 161.

31.

Parameshwaran

and Thiagarajan

T.C.

, Complexity of EEG reflects socioeconomic context and geofootprint. bioRxiv, page 125872, 2017.

32.

Ramadan

R.A.

and Vasilakos

A.V.

, Brain computer interface: control signals review, Neurocomputing. 223 (2017), 26–44.

33.

Rasheed

, A review of the role of machine learning techniques towards brain–computer interface applications, Machine Learning and Knowledge Extraction 3(4) (2021), 835–862.

34.

Ries

A.J.

, Touryan

, Vettel

, McDowell

and Hairston

W.D.

, A comparison of electroencephalography signals acquired from conventional and mobile systems, Journal of Neuroscience and Neuroengineering 3(1) (2014), 10–20.

35.

Rumelhart

D.E.

, McClelland

J.L.

, Group

P.R.

et al. Parallel distributed processing, volume 1. IEEE New York, 1988.

36.

Saa

J.F.D.

and Cetin

, Discriminative methods for classification of asynchronous imaginary motor tasks from EEG data, IEEE Transactions on Neural Systems and Rehabilitation Engineering 21(5) (2013), 716–724.

37.

Schmidhuber

, Deep learning in neural networks: An overview, Neural Networks. 61 (2015), 85–117.

38.

Sugeno

and Yasukawa

, A fuzzy-logic-based approach to qualitative modeling, IEEE Transactions on Fuzzy Systems 1(1) (1993), 7.

39.

Suryotrisongko

and Samopa

, Evaluating openbci spiderclaw v1 headwear’s electrodes placements for brain-computer interface (bci) motor imagery application, Procedia Computer Science. 72 (2015), 398–405.

40.

Tangermann

, Muller

K.-R.

, Aertsen

, Birbaumer

, Braun

, Brunner

, Leeb

, Mehring

, Miller

K.J.

, Mueller-Putz

et al. Review of the BCI Competition IV, Frontiers in Neuroscience 6 (2012), 55.

41.

Thomas

, Maszczyk

, Sinha

, Kluge

and Dauwels

, Deep learning-based classification for brain-computer interfaces. In 2017 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pages 234–239. IEEE, 2017.

42.

Toledo-Moreo

, Pinzolas-Prado

and Cano-Izquierdo

J.M.

, Maneuver prediction for road vehicles based on a neuro-fuzzy architecture with a low-cost navigation unit, IEEE Transactions on Intelligent Transportation Systems 11(2) (2010), 498–504.

43.

Velayutham

, Non-invasive electroencephalography signals classification using rough neural network, International Journal of Computational Biology and Drug Design 8(3) (2015), 212–225.

44.

Wang

, Su

, Yan

, Lu

, Zhao

, Liu

and Zhou

, Rehabilitation treatment of motor dysfunction patients based on deep learning brain–computer interface technology, Frontiers in Neuroscience 14 (2020), 595084.

45.

Yang

, Sakhavi

, Ang

K.K.

and Guan

, On the use of convolutional neural networks and augmented CSP features for multi-class motor imagery of eeg signals classification. In 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pages 2620–2623. IEEE, 2015.

46.

Yau

S.H.

, McArthur

, Badcock

N.A.

and Brock

, Case study: auditory brain responses in a minimally verbal child with autism and cerebral palsy, Frontiers in Neuroscience 9 (2015), 208.

47.

Zhang

, Yao

, Wang

, Monaghan

, Mcalpine

and Zhang

, A survey on deep learning based brain computer interface: Recent advances and new frontiers. arXiv preprint arXiv:1905.04149, 66, 2019.