A review of artificial intelligence for EEG‐based brain−computer interfaces and applications

2.1 Overview

Machine learning primarily aims to automatically discover the brain patterns in a specific task without using traditional statistical methods. The ML technologies in EEG‐based BCI applications are usually classified into two major categories: classification, and individual adaptive tasks.

Before applying the ML technologies to EEG signals, we require to preprocess the EEG signals and then extract features, such as power spectral and functional connectivity [7] from the preprocessed EEG data. Fig. 3 shows a genetic pipeline of data processing in the EEG‐based BCI in a compact and relevant way, which includes data acquisition, data preprocessing, pattern extraction, and classification techniques.

OPEN IN VIEWER

Fig. 3.

The genetic pipeline of data processing in the EEG‐based BCI.

In terms of classification tasks, we usually apply the following two conventional types of ML: supervised learning and unsupervised learning [8]. Specifically, in supervised learning, the data is usually divided into two subsets: the training and testing sets. The testing set is used to judge the type of event by using the knowledge learned in the training phase with label sets. To classify or predict future events, supervised learning can be used for regression or classification tasks. The data used for training here is neither classified nor labeled; instead, we apply unsupervised learning.

In terms of individual adaptive tasks, we generally refer to transfer learning, exhibiting distinctive classification problems of consistency. The above ML strategy, such as supervised learning, assumes that the training and the testing datasets used to evaluate the classifier geared with the same feature space and probability distribution. However, the above assumption is usually violated in practical applications because the human body is different [9] in most cases. Therefore, transfer learning can be used to overcome the purpose of dealing with data that violates the above hypothesis by using the knowledge obtained while learning a given task to solve another but related tasks.

Furthermore, reinforcement learning (RL) features rewards through the interaction between learning and the environment. It uses feedback signals to build adaptability that is suitable for EEG‐based BCI. The RL learning method helps us to understand better, and then use brain activities to interact with computers and other devices to improve the performance of several EEG‐based BCI applications.

2.2 Supervised and unsupervised learning

The traditional supervised learning models include linear models, neural networks (NNs), Bayesian classifiers, neighbor‐distance classifiers, and classifier combinations [10]. For linear models, such as linear discriminant analysis (LDA), regularized LDA, and linear support vector machine (SVM), the discriminative EEG patterns are classified using the linear decision boundary between the feature vectors of each class. Furthermore, for NNs, they assemble artificial neuron layers to border on any non‐linear decision boundary. A common type in EEG‐based BCI applications is a multi‐layer perceptron, using only one or two hidden layers. In the next section, we will discuss deep NNs for DL. Then, in terms of non‐linear Bayesian classifiers, such as Bayesian quadratic classifiers and Hidden Markov Models, it measures the probability distribution of each category and uses Bayesian rules to select the EEG patterns. Besides, the nearest‐neighbor classifier, such as the k‐nearest‐neighbor (kNN) algorithm, considering the physical distance of each class, proposes to assign classes to the EEG patterns according to its nearest‐neighbor. Finally, a combination strategy, such as voting or stacked‐based combination method, is applied in classification tasks by mixing the predicted classes from multi‐classifiers.

Based on the EEG‐based BCI classification algorithms in the past ten years [11], the classifier options that go with different kinds of EEG patterns/features were reviewed as follows:

EEG features with noise and/or outliers: we suggest choosing regularized classifiers, such as the SVM, which is suitable for handling outliers. The discriminative classifiers [12] may be a back‐up choice, as it showed better performance compared to generative classifiers in the presence of noise or outliers.

2.3 Transfer learning

The EEG patterns are the potential to be passed between different domains. For example, selecting one type of human activity is transmitted based on the similarity of the EEG pattern between the target and source domains. If both domains are very similar, and the EEG pattern is semblable, the discriminatory information should be transmitted. However, for the cases that domains are diverse, and there exists some standard information between the target and source domains, where some unchanged or persistent information can be transferred via a constant system.

In EEG‐based BCI applications, the covariate shift is the most common case, which input distributions of the training and test samples differently [13]. The BCI‐based transfer learning is divided into three categories: (1) transfer from tasks to tasks, (2) transfer from subjects to subjects, and (3) transfer from sessions to sessions. For example, given a set of training data sets, such as source tasks, subjects, or sessions, the covariate shift tries to find a transformation space where training the model could facilitate the classification or prediction of new samples.

Especially, subject to subject transfer is the leading category in the EEG‐based BCI transfer learning. The BCI may need to collect a small amount of data, called preliminary data, to overcome differences between subjects, before applying individual EEG features to a training model. But transfer learning can overcome the above matter. For example, the study of driving drowsiness detection [14] evaluated inter‐subject and intra‐subject variability. The transfer was verified by performing hierarchical clustering on large‐scale EEG data sets collected from a large number of subjects.

Furthermore, interdisciplinary transfer learning can also be used to reduce the training data collection time, which is the least square transformation (LST) approach [15]. The experiments conducted to confirm the performance of the LST approach using interdisciplinary visual stimulation data demonstrate that the number of training templates can be reduced. Besides, when labeling data is not possible, the inter‐subject and inter‐subject transfer learning can be applied under unsupervised conditions. For instance, a recent study [16] is used to directly align the EEG trajectories between different objects in Euclidean space with increasing similarity. Experimental results confirm the potential for learners to transfer learning using unsupervised ECI‐based BCI.

2.4 Reinforcement learning

RL aims to learn and optimize the agent’s actions and behaviors based on reward signals during task execution. These reward signals are usually designed by human experience. Still, these rewards are often difficult to encode in complex situations, making the EEG activities from a BCI system a suitable substitution. Furthermore, it is a novel approach to capture subjective aspects and criteria unique to each user. For example, the EEG signals are considered a reward reference to accelerated robot movement and learning [17] and applied in neurofeedback mindfulness system optimization [18].

3.1 Overview

Deep learning is a series of specific ML algorithms, which refers to the system of deep NNs, and it is used to learn the EEG patterns [19]. The typical characteristic of the DL systems is based on a large number of neurons in multiple cascades, allowing them to learn features highly related to the human cognitive level. The typical systems for DL include convolutional neural network (CNN), recurrent neural network (RNN), and generative adversarial network (GAN). For the EEG‐based BCI applications, the DL is widely used to extract EEG features and then combine with the traditional ML technology to achieve a stateof‐the‐art classification or regression task. In this section, spontaneous EEG‐based BCI applications with CNN system, the use of GAN, the process and application of RNN, especially long short‐term memory (LSTM), and the spatiotemporal combination of CNN and LSTM are introduced.

3.2 Convolutional neural networks

Convolutional neural network is a model of feedforward deep NNs, where input information flows the EEG patterns through the essential three stacked layers: a convolutional layer, pooling layer, and fully connected layer [20]. The convolutional layer convolves the tensor with the shape, and then pooling layer simplifies the necessary calculations to reduce the data dimension. Finally, the fully connected layer connects each neuron in the previous layer to a new layer for predicting or classifying the target goals. This system of DL with stacked layers aims to reduce the input data and capture unique spatial dependence of the EEG patterns.

The significant progress of CNN in classification tasks is vital to the development of the BCI, such as improvement of EEG‐based fatigue assessment. For example, an EEG‐based system of the spatial‐temporal convolutional neural network was proposed [21]. The advance of this system extracts temporal dependencies. It combines with dense layers for spatial‐temporal EEG information process and classification, which achieved 97% accuracy in the classification of EEG driving fatigue levels. Another recent study [22] also assessed diverse fatigue levels in a visual EEG experiment. It proposed a system of multi‐scale CNN that uses a space‐time-frequency combined features extraction strategy to extract raw EEG signals from an inception structure.

3.3 Recurrent neural networks and long short‐term memory

The RNN, featuring with a temporal sequence to record the memory, can address human reasoning by forming a directed graph between neurons. However, the generic network of RNN could only look at the new dependencies to avoid retaining some long‐term information. A particular type of RNN, LSTM networks, was proposed to handle learning long‐term dependencies capability.

The studies of the RNN or LSTM framework in EEG‐based BCIs increased substantially in the past years with findings proposing that the results of RNN or LSTM‐based methods outperform traditional ML. The reason is that they can extract hidden EEG feature representations, and RNN combined with CNN can optimize feature representations by the integration of both temporal and spatial information of EEG signals. The framework is helpful for motion identification, emotion recognition, and sleep staging assessment [23]. For example, Supratak et al. [24] investigated automatic sleep stage scoring by employing an LSTM bidirectional algorithm for transitional rules learning of hidden time‐series EEG features and CNN to extract time-invariable EEG features.

3.4 Generative adversarial networks

In some cases, the real‐world EEG data is limited, which could be vulnerability when we train some ML or DL models. To overcome this problem, the GAN, including two synchronously trained NNs, called “generator networks” and “discriminator networks”, was recently proposed primarily for data augmentation purposes. It claims GAN generates new samples to mimic real‐world data from implying generative models [25].

For example, a novel GAN model [26] was recently proposed, which learned the statistical characteristics of the EEG and increased the size of datasets by generating synthesis samples to improve classification performance. Also, a new EEG‐based framework [27], named Conditional Wasserstein GAN, was proposed to enhance emotion recognition by generating high‐quality synthetic EEG data based using the SEED and DEAP datasets.

4.1 Computer vision

Computer vision explores digital images from cameras and videos and uses the ML to identify and classify objects to represent what a human sees accurately.

The EEG signals were recently investigated to see if they could be used as effective ways to transform human visual abilities into the CV. For example, an experiment of previous work [28] used 128‐channel electrodes to record the EEG signals, while six subjects were meant to view 2000 images from 40 types of object classes. Then, the authors proposed a visual object classifier driven by the EEG signals, which use an RNN to learn discriminative brain activities manifolds of visual categories. Also, they trained a CNN‐based regression model to project the image onto the learned manifold for classifications with 83% accuracy. Furthermore, a follow‐up study [29] used the learned EEG manifold to adjust the GAN’s image generation. During the inference process, the GAN reads the EEG signals and converts it into an image, while the generated image is real and very similar to the image of GAN’s EEG signals.

Another recent study [30] introduced a brain sourcing framework which measured EEG responses to perform a recognition task where visual features of faces (gender, hair color, age, and smile) are recognized better from a group of subjects instead of the individual subject. It may open opportunities for more general applications using the BCI in a crowdsourced setting.

4.2 Natural language processing

The NLP is a theory‐motivated range of computation methods for automatic analysis and representation of human language, where distributional semantics describe the meaning of words and sentences using vectorized representations.

We assume that our brain processes language and generates cognitive processing data, such as the EEG signals, when we read words and sentences. A recent work [31] evaluated word representations from reflecting the semantic representations in the EEG and associated physiological signals. It proposed a multi‐modal framework, called CogniVal, for evaluating English word representations based on cognitive lexical semantics. Also, six types of word embeddings were evaluated by fitting them to 4 EEG datasets during language processing.

Machine translation is a critical element of NLP, which aims to build a reliable encoder-decoder system. Recently, Chang’s team [32] released an EEG‐based text translation study showing how to decode EEG signals with high accuracy and natural speech rate. They trained an RNN to encode each sentence length sequence of neural activity into an abstract representation, and then decode this representation verbatim into English sentences, and achieved a 3% average word error rate.

4.3 Robotic controls

For a human who cannot control such robots through peripheral nerves and muscles, the EEG‐based BCIs are instead proven suitable means to communicate with the robots. The early stage of the EEG‐based robotic controls was driving a wheelchair from one destination to the other by using visual simulations or motor imagery. Some recent studies are motivated to address the safety risks in controlling robots. Li et al. [33] proposed a shared control strategy between a brain–machine and autonomous control modes. In the autonomous control mode, the synthesis of angle‐based potential field and vision‐based simultaneous localization and mapping was used to guide the robot for obstacle avoidance and keep the robot safe. Then, Tonin et al. [34] proposed a control method of motor‐imagery BCI to ensure robot safety by investigating inherent predictive nature and constraint handling capabilities to reduce the false‐positive rate. Besides, the RL, especially reinforce information from hierarchical critics [35], can also be another suitable method to propose a shared control approach to address the safety and performance issues of brain‐controlled robots. This work provides an enabling platform to enhance the future research and development of brain‐controlled robots.