Applying deep learning in brain computer interface to classify motor imagery

BCI systems capture brain signals and decode them with the purpose of interacting with certain devices without any muscular intervention [30]. Most of the BCI mechanisms used for this purpose are based on electroencephalogram (EEG) techniques, where the electrodes detecting the electric potentials caused by the neurons inside the brain are placed on the scalp of the user [25]. This technology has fundamentally been applied to medical and psychological brain research, rehabilitation treatment of motor dysfunction [44]. However, the decrease in the cost of the headsets is closing the gap between the research or medical environment and the households, where applications such as video gaming are starting to pave the way for the adoption of BCI devices by the marketplace. One of the most difficult problems faced by BCIs is called motor imagery and it consist in properly classifying three mental tasks: left hand movement, right hand movement and generation of words beginning with the same random letter. Solving this problem only with non-invasive EEG data involves processing signals that are highly variable, not only between users, but also within a single user depending on daytime or the fatigue level. Artifacts and outliers usually arise increasing the difficulty of this problem.

Some pattern recognition problems can be directly validated by a human being. A typical example is shown in Fig. 2.

OPEN IN VIEWER

Fig. 2

Handwritten characters recognition application.

However, it must be noted that the EEG signals cannot be read and decoded straightaway by nobody, even the most experienced neurologist Fig. 3.

OPEN IN VIEWER

Fig. 3

EEG signals recognition for BCI application.

It is an emerging area, which fits very well with artificial intelligence and more specifically to the DL way of doing. It is able to extract the underlying information in those difficult problems. The BCI problem is solved following several steps and all of them can be carried out by mean of AI methods, increasing the abstract level of representation. DL must avoid the presence of noise or the high dimensional EEG data that characterized the motor imagery problem, as mentioned in [13]: “the successful implementation of deep learning methods for the classification of EEG signals is quite an achievement”. In that paper, recordings from three electrodes (C3, Cz, and C4) and five electrodes (C3 C1, Cz, C2, and C4) are included, and only two states are taken into account.

DL process is applied in [41] to BCI problems using Convolutional Neural Networks and Recurrent Neural Networks with Long Short-Term Memory architecture. It is applied to a non- motor imagery problem (The stimuli are non-overlapping flickering lights from five magenta boxes with frequencies 6.66 Hz, 7.5 Hz, 8.5 Hz, 10 Hz, and 12 Hz. The objective is to distinguish between the different stimuli frequencies by analysing the Steady State Visually Evoked Potential.

In [45] also Convolutional Neural Networks are used to deal with a motor imagery problem using BCI Competition IV dataset [40] for the experimental work.

A Deep Belief Network of three layers of Restricted Boltzmann Machine is used in [25] to compare Power Spectrum Density extraction methods and classifiers to improve the recognition accuracy in different motor imagery tasks (left hand, right hand and foot movements). A survey of DL technology applications to BCI problems, can be found in [1, 33, 47]. The BCI motor imagery case with EEG signals is examined in this paper.

The motor imagery problem is one of the hardest issues in BCI technology. Here, the mental categories are imaging moving some body part, foot, hand, tongue, etc. The brain activation of this task is linked with the activation of the real movement. Therefore, this kind of problem is suitable to implement rehabilitation systems, exoskeleton control, prosthetic limbs [30], etc. BCI Competition [6, 40] is a forum where researchers can discuss the advance of knowledge in this field. To this aim, a challenge problem is proposed for each edition. Nowadays the databases provided in this competition are still used as testbeds in new researches, i.e. in [14] DL is proposed to deal with Motor Imagery problems using BCI Competition Data. Specifically, BCI Competition III Set V [6] dealt with the prediction of the following tasks: “movement intention of the left hand”, “movement intention of the right hand” and “thinking of a word”.

This Competition provided data from three users with three sessions for each one, to be used to build the prediction model. Data from a fourth session was also provided in order to validate the predictions. Data was gathered with a professional EEG (Biosemi) Fig. 4(a) hardware following a protocol:

Experiments were carried out with the user seated and still.

A criterion was established to allow a fair comparison among the methods presented.

A window of only the last second registered data can be used to compute the next prediction value.

Several works have been proposed at this Competition using Neural Networks, Common Spatial Patterns, Support Vector Machines, Linear Fisher Discriminant, Gaussian Classifiers [4, 5, 16, 22, 26, 36, 43].

1.3 Adaptive resonance theory

Deep learning implementations typically use multilayer networks. In this paper we propose an alternative based on Adaptive Resonance Theory (ART) because one of the fundamental characteristics of the ART is its modular nature. It provides a series of building blocks with specific functionality that can be used to build complex architectures [19]. The functioning of the brain is based on basic elements (neurons) that interact generating complex structures. The flexibility that the ART architecture provides to define complex structures from basic elements, and even incorporate elements such as fuzzy logic, has made it possible to apply it to multiple problems [12]. In many cases, this type of architecture aims to process data to reach categories of abstract thought [18] specifically its relationship with thought and the generation of motor commands [20].

To deal with the contribution of different outputs in different time instants, a multidimensional Sustained Temporal Order REcurrent (STORE) [27] type module is proposed. It works as a buffer storing the last model predictions. At first it has a flat characteristic meaning that the unit activations, which represent the predictions, do not depend on the memorized sequence. The units activations of the STORE module are connected to the input by an on-center off-surround multiplicative competitive net [24] through an instar module [17]. The labels associated to the mental tasks are the net outputs.

S-dFasArt model [42] is a supervised version of a dynamical Fuzzy Adaptive System ART based (FasArt) model [7] allowing classifying time signals with some characteristics that make it suitable to BCI applications.

The model is able to deal with temporal data, so not only the instantaneous value is taken into account, but also when the data is registered (time order reception).

The S-dFastArt performance in BCI applications has been compared against other classification methods using the BCI Competition III Set V Database showing the best success rate [8, 28].

1.4 Low cost platforms

To increase the number of BCI systems applications, there must be a price drop in hardware and an improvement in comfort use. With this goal in mind, some devices that could be named as “low cost”, have appeared to expand the scope of people that could use BCI technologies [11, 32]. These are classified as follows:

End-use devices: they are addressed to a final user, usually with a particular objective, for instance helping a user to relax or helping one to concentrate, as Neurosky [23].

OPEN IN VIEWER

Fig. 4

Headset comparison (a) Biosemi. (b) Emotiv EPOC.

In this last set of devices of general use, some features must be highlighted:

Emotiv EPOC is one of the main low-cost devices, taking into account the previous ideas. It has a fixed structure with diadem shape and plastic arms allowing some flexibility and achieves scalp contact for most users. Primarily it uses dry sensors, but sometimes saline is used to improve conductivity. This does not have any effect on the user’s hair which remains unaffected. This hardware can be bought both with the manufacturer’s software in a closed version, or with a version for developers. This last one permits access to the raw data coming from the sensors.

The mentioned features and the low cost of the Emotiv EPOC system, has led to researchers to consider it for dealing with engineering problems that need EEG signals acquisition. Some papers have been published making comparisons of data quality between Emotiv and professional equipment [2, 15, 21, 34]. Also papers for BCI applications with this device have appeared [10, 29, 31, 46].

Figure 4 shows a Biosemi and an Emotiv team. The advantages in terms of ease of use and preparation time provided by the Emotiv equipment are evident, especially when considering its use by a user not accustomed to this type of technology.

In the 1 section, the areas of interest considered in the work have been shown: BCI, Motor Imagery, ART Theory and low-cost platforms. Based on this, an architecture to be applicable to different BCI problems and that shares the Deep Learning methodology is proposed. It allows a straightforward application from the raw data to the results without previous steps. On the one hand, the proposed architecture is made up of modules developed in the ART and Fuzzy-ART theory and attempt to be self-organizing and therefore does not require an external expert to determine its topological characteristics or parameters. On the other hand, the use of ART and Fuzzy-ART modules allows an interpretation of the functioning of the architecture, which eliminates the black-box characteristic of many Deep Learning proposals. The presented architecture could be named as: Transparent Deep Learning, because it is self-organizing, learns directly from the data and allows an interpretation of how the processing is performed.

1.6 Paper organization

The 2 section shows the proposed architecture as well as the learning methodology following the Deep Learning philosophy. In the 3 section, the capabilities of the proposed architecture are explored showing its self-organization capacity: in the self-adjustment of the parameters, in the determination of the structure and in the automatic selection of the most relevant characteristics of the learning data. Objective results of the performance of the architecture with a set of original data are also provided. Finally, the most relevant conclusions of the work carried out are collected in the 4 section.

The S-dFasArt stage in the architecture proposed in Fig. 5 requires a learning process to build the rule base for the S-dFasArt model. To this aim, the learning sessions collected from each user are used. The structure is different for each user to manage the inter-user’s variability. Four mechanisms are selected to define the multi-model system, as shown in Fig. 6.

Rule generation: S-dFasArt learning algorithm is used to build the rules set from the data coming from each of the considered sessions. The N rules appear as:

IF x₁ IS A 1 j AND … AND x _n IS A n j THEN LABEL j

Where:

A j 1 … A j n are the fuzzy sets represented in Fig. 6(a).

Note that the last three mechanisms (adjustment, pruning and reduction) allow the model to enhance information coming from other sessions, different to the session used to generate the model. This seeks to improve the model’s generalization ability to handle the intra-user variability.

OPEN IN VIEWER

Fig. 6

Learning methodology for S-FasArt models. (a) Rules generation. (b) Parameter adjustment. (c) Rule pruning. (d) Dimensionality reduction.

3.1 Learning and adjustment method

As mentioned above, three data sessions are used for the learning task. Initially, three different models will be generated, one for each learning session, using S-dFasArt [42] architecture. This architecture generates fuzzy rules set which exhibits the data structure used in the learning process 5.

The activation/membership functions are triangular and are determined from three weights. In the j antecedent of the i rule, these weights are (w _ij , c _ij , v _ij ).

The units activation and the weight learning are computed with differential equations according to the assumptions of Adaptive Resonance Theory [17]. Once the learning task has been done, the number of the model parameters will be: P = 3 * N * M + N, where N is the number of rules created and M is the input vector dimension, that is to say, the number of antecedents of each rule.

The prediction of S-dFasArt architecture depends on two design parameters: σ y A _t . The first one modifies the support of the fuzzy sets for the rules antecedents that have triangular activation functions. This support is the interval [w _ij  - σ, v _ij  + σ] for the j-th antecedent of the i-th rule. It regulates the level of the rule fuzziness [7]. The second parameter, A _t , governs the speed of unit’s activation, and therefore the speed the model responds to input changes [7]. Low values of A _t leads to a slow change in system activation state and so, the input value must be persistent in order to produce a change in network weights.

In the other way, high values of A _t carries to quick activation variations when the input changes. Initially, these parameters must be adjusted in an external way, which would make the model not fully self-organized. To avoid this, an algorithm to parameter adjustment using learning techniques with neuronal inspiration, is proposed. It is a variation of the known Perceptron learning rule [35]. In an incremental way it is given by the algorithm described in equation 1 A t n e w = A t o l d + Δ A t * f ( J ( A t + Δ A t , σ ) − J ( A t , σ ) ) σ n e w = σ o l d + Δ σ * f ( J ( A t , σ + Δ σ ) − J ( A t , σ ) )(1)

where f (x) =1 if x > 0 and f (x) =0 in another case. J (A _t , σ) value is related with the model success for a selected test set. This algorithm is executed until there are no variations in parameters. ΔAt and Δσ will be alternated between positives and negatives values, and they are chosen among a discrete set of possible candidates. In a first step, the same session will be used to do the rule learning and the parameter adjustment. This implies that three different models can be generated.

The activation sequence in the upper layer of units in S-dFasArt is connected with a STORE module set. This set, through an Instar module, also feeds an on-center off-surround net. So, the most activated category results as the unique activated category. Then, the label associated to that one is proposed as the prediction output.

Data collected in Table 1 show the large deviation between the success rate when the session is the same used for the learning task and adjustments, from other sessions. The average of the results obtained for the four users that make up the database are collected.

This indicates the intra-user variability. To fix this problem, an information mix coming from different sessions is proposed. At first, a session will be used for model generation and a different one will be used for the adjustment task. The goal is that the built rules acquire enough generality to represent two learning sessions. Now, each model will be created starting from two different learning sessions (Sa,Sb) so that:

Session Sa for the learning task

With this method, the three learning sessions collected can create six different models. The third session (Sc) is not employed at all to the model’s development, so the success rate for that session will suggest the model generalization ability.

The results shown in Table 2 have been compared with those obtained using the same session for the learning and adjustment task (Table 1). It can be seen that:

The rate of success of the adjustment session is higher when this session has not previously been used for learning than when the same session was used for both learning and adjustment.

The improvement of using different sessions for learning and adjustment has been maintained. The increase in the success rate for the adjustment session shows that part of the understood information has been incorporated into the model, so that the model learns and adapts to include information from the new evidence (data).

On the other hand, the increase in the success rate of the session that has not been used neither for the learning task nor for the adjustment, manifest that the model generated with the two sessions presents a better generalization character than the one obtained from a single session. Rule variations coming from the adjustment phase make the model suitable to represent a data set with high variability and so, suitable to respond better to the intra-user variability problem.

One of the problems we usually face presents wrong labelled data. These generate rules whose predictions can be inconsistent. Examples where the same situation generates different conclusions, have been presented to the system. This is a handicap to learn properly and produce a non-defined output. To solve this, the S-dFasArt architecture incorporates a match-tracking mechanism that incorporates rules that consider the different outputs even if they come from the same antecedents. The problem could persist because there is not an objective way to choose among all of these rules. The solution proposed is to add a rule pruning stage. Here, the rules are evaluated according to their performance and the ones with the best marks are selected. For the assessment task efficiency criteria have been proposed:

It consists of three phases: To learn, to adjust and to prune. Each of them uses one of the available sessions to make the learning task as follows:

Sa session is used for the S-dFasArt learning phase to generate the model.

Efficiency criteria for the rule pruning: This method gets one of the learning sessions and computes the predictions of each one of the previously defined models. The rule that has driven to each prediction and if it has been right or wrong, is stored.

At the end, a global calculation of the number of times that each rule has led to a right or a to a wrong prediction is carried out. The model will be composed by the rules set with a positive balance in this evaluation.

The results using the efficiency criteria for the rule pruning task are shown in Table 3

Some outcomes can be obtained from Table 3:

The success rate for the session used both for the adjustment and pruning tasks increases.

Therefore, the effect of the pruning task is to include information from the session used, in the same model that was generated with data coming from another session.

This method uses a Group Method Data Handling (GMDH) [38] algorithm to select the rules that will constitute the model. A Regularity Criterium (RC) [28] based in the success rate is defined: RC = Wright predictions number Data number(2)

Initially the model consists of a single variable. With this assumption, RC value is computed for all the possible models that can be generated with a single variable. The model with the highest score is chosen. In the following steps, the number of variables is enlarged adding a new one. Again, RC value is computed for all the possible models and the model with the highest score is chosen. If this mark is better than the previous one, the new variable that has produced it is incorporated into the model. Now, we will try adding a new variable in a new interaction.

The GMDH algorithm is used to reduce the vector dimensionality associated with the antecedent’s space of S-dFasArt model. So, the number of rules antecedents (at first are 168) is decreased.

From Table 4 the conclusions that can be highlighted for the effect of dimensionality reduction are:

The success rate for the session used both for the adjustment and reduction tasks increases.

The model ability to predict properly a data session increases if the dimensionality reduction has been done with the same session. Therefore, the information contained in the session used for the reduction task is added to the model. Also, but with a lower degree of certainty, it can be deduced:

The success rate of the session not used, neither for the adjustment nor for the reduction task, increases.

These leads to think that from a generic point of view, the dimensionality reduction has improved the model’s generalization ability, and so the capacity to predict different situations from those that have been used for the learning stage.

It must be noted that, in relation to the number of architecture parameters, for each model the initial number was P _i  = 3 * N _i  * M + N _i where N _i =number of model rules of i-th model and M = dimension of input vector, 168 for this issue. For instance, using the values of Table 2 we can compute:

P = ∑ i = 1 6 3 * N i * M + N i = 688820

For rule pruning and parameter reduction model using the values of Table 4 and with 6 models, the number of parameters are:

P = 12515

In the same way with 12 models:

P = 23405

It can be seen that in this last case, the number of parameters is 30 times lower with double of the models. The reduction in computational burden is obvious because the lower parameters used, the lower rules activations and belonging values must be computed. This is remarkable in an application with limited hardware.

As it was described in Section 1.2 the BCI Competition III Set V deal with a problem with three mental categories. This allows us to propose several problems to be solved with our architecture.

The problem faced includes two mental categories associated to motor imagery (“movement intention of left hand”, “movement intention of right hand”), and a third one without motor significance (“thinking of a word”). So, the proposed model can be employing for two purposes: to detect motor intention in a synchronous way (with no prior knowledge of the event time) and to detect the laterality of the motor intention.

The first one is to distinguish between motor and none motor thoughts. With this aim, a success rate can be defined as the number of times that a motor category is predicted properly A _M versus the number of motor categories (N _I  + N _D ), Motor success= A M N I + N D and the capacity to detect a non-motor task: Word success= A W N W The second one is the success in distinguishing the laterality taken into account Left success= A I N I and Right success= A D N D . The third one is to achieve a global success rate Global= A I + A D + A W N I + N D , N W

The results for this rate with the 4 users (in %) are stored in Table 5. Confusion matrices are shown in Table 6.

The well-known performance Cohen’s Kappa index is computed in Table 7 for all the users with the validation session using 6 and 12 models.

The results collected in Table 5 indicate that the success rate in the prediction of MOTOR patterns is almost the same using 6 models (77.59%) or 12 models (77.59%). Also, the difference in the success rate for the WORD category is not significant: 75.78% for 6 models and 75.88% for 12 models. This behavior is maintained in the global rate: 56.17 % for 6 models compared to 57.90 % using 12 models. It can be inferred that the architecture can extract the most relevant information from data using only 6 models, which simplifies the necessary calculations and clearly reduces the computational cost.

Likewise, the obtained results reflect the ability of the system to discern between data associated with a motor activity and data associated with a non-motor activity. All this, using low-cost, non-invasive capture equipment, and using a generic and self-organizing architecture that does not require specific previous knowledge of the problem.