Review of brain–computer interface based on steady‐state visual evoked potential

2.1 Visual information‐processing mechanism of SSVEP

All human senses are derived from activities in the sensory system. Different receptors generate sensory signals after receiving specific types of stimuli. These signals are transmitted to the center through complex processing in the sensory pathway and finally result in perception [29]. These senses include visual, auditory, and sensory stimulation, among others. Specific thinking tasks or specific external stimuli encode the electrophysiological activities in the brain, and subjects can generate different EEG signal patterns. SSVEP can be considered as a specifically encoded external stimulus. For perception of visual information, the external visual stimuli fall to the retina through the optical imaging system of the eye. After preliminary information processing on the retina, the stimuli reach the external knee body along the optic intersection of the nerve fibers. After a nerve alternation, they enter the secondary and advanced visual centers of the brain through the primary visual cortex in the occipital lobe and subsequently enter a higher level joint area located in the occipital and parietal cortices for higher level information processing [30].

When a human body is stimulated by a fixed frequency flickering vision, the potential activity of the cerebral cortex is modulated. Thus, a continuous frequency is generated, which is related to the stimulation frequency (at the fundamental frequency or frequency doubling of the stimulation frequency), which contains a periodic rhythm similar to that of visual stimulation. An SSVEP signal is expressed as an EEG signal, and a spectral peak can appear on the stimulation frequency or harmonic in the power spectrum. By analyzing and detecting the corresponding frequency at the spectral peak, the stimulus source of the visual fixation of the subject can be detected to identify the intention of the subject. When a visual stimulation device is used to obtain an SSVEP signal, six EEG channels are available that are related to the occipital visual area in the cerebral cortex. (The six channels are PO3, POZ, PO4, O1, Oz, and O2.)

In 1966, Regan induced a periodic EEG‐corresponding SSVEP with the same fundamental machine‐harmonic frequency as the visual stimulation frequency in the occipital part of the brain using a fixed frequency‐modulated scintillation stimulation on human eyes [31]. Therefore, an SSVEP signal is generally believed to be a physical response of the primary visual cortex to external periodic stimuli and does not involve advanced visual information processing and cognitive psychological processes [32]. This signal mainly appears in the occipital region where the primary visual cortex is located.

2.2 Transient VEP

In early BCI research, the transient VEP (TVEP) was considered as the input of a BCI system. In the published literature, the first article on TVEP was published by Vidal et al. in the BCI Laboratory of University of California Los Angeles [33]. Vidal et al. first applied TVEP to the study of BCI. In 1977, they developed an online traditional reactive TVEP‐BCI system based on TVEP. Fig. 3 shows that the user generates TVEP by looking at one of the four fixed points around the maze and checkerboard stimulation. The four vertices of the visual stimulation interface used by the BCI system were four flashing stimuli (corresponding to “up”, “down”, “left”, and “right” in four spatial directions). The user could generate TVEP signals using the corresponding directional characteristics by looking at the flashing stimuli in different directions to follow the direction of the cursor. The user must stepwise move the indicating arrow from the beginning of the chessboard maze to getting out of the maze.

OPEN IN VIEWER

Fig. 3.

Stimulus target in the TVEP experiments. The target consists of a fixed diamond‐shaped red checkerboard illuminated with a xenon flash to provide visual stimulation.

In the present study, the original EEG signals are first filtered and denoized, and they are then classified using a pattern‐recognition algorithm. The accuracy of the system target recognition reaches 80%–90%, and real‐time control of the BCI system is realized. This study proves the possibility of using brain electrical signals to control peripherals for the first time.

2.3 SSVEP

SSVEP exhibits significant periodicity and is not easily disturbed by facial expression, blinking, and other movements. In 1989, Regan et al. first proposed the experimental paradigm of using SSVEP as a closed‐loop feedback‐control BCI system [34].

Dechwechprasit et al. studied the time–frequency coherent response of SSVEP red–green‐scintillation stimulation paradigm to EEG signals [35]. They found that in the monochromatic‐contrast experiment, green light had higher accuracy than red light with average accuracies of 59% and 41%, respectively. In the alternating‐scintillation experiment under the stimulation of two lights, the accuracy of red light was higher than that of the green light, and the average accuracies of the red and green lights were 62% and 38%, respectively.

The teams of Shangkai Gao and Xiaorong Gao from Tsinghua University have made considerable achievements. They implemented a BCI cursor‐control system based on SSVEP in 1999 [36]. The system was designed to help patients with severe limb paralysis control a cursor. Then, they developed a set of BCI telephone‐dialing system in 2002, and the information‐transmission rate reached 27.15 bits/min [37]. The SSVEP‐inducing platform of the experimental paradigm of the system was realized using virtual keys that flashed at different frequencies on the display. The signal processing scheme was a spectrum analysis based on fast Fourier transform. By comparing the spectrum peak of the SSVEP signal online, the flashing block watched by the subject could be identified to realize the brain‐idea selection button. Since then, they have designed a series of SSVEP‐BCI systems with a high transmission rate.

2.4 Improved SSVEP paradigm

The SSVEP paradigm has achieved reliable results in some fields. However, new application scenarios require more enhanced powerful performance of the experimental paradigm. Some new experimental paradigms (such as phase‐coded SSVEP, motion SSVEP, and hybrid SSVEP) have been successively proposed to optimize the user experience, expand the instruction set, and improve the information‐transmission rates.

2.5 Problems in experimental paradigm

During the development of SSVEP‐BCI from the traditional SSVEP paradigm to various improved SSVEP paradigms, the limitations of the traditional SSVEP paradigm have been remarkably minimized, and the SSVEP‐BCI performance has significantly improved. The performance evaluation of the different experimental SSVEP paradigms is listed in Table 1. However, we must note that two main reasons exist why the equipment (brain‐controlled exoskeleton and brain‐controlled typing) based on SSVEP‐BCI was not publicly applied.

OPEN IN VIEWER

Table 1.

Performance evaluation of different SSVEP experimental paradigms.

Paradigm categories	Research team	Year	Performance evaluation
TVEP	Vidal et al. [33]	1977	SSVEP was later proposed based on the TVEP
SSVEP	Regan et al. [31]	1989	With significant periodic spectral characteristics and stable amplitude and less susceptible to external interference
Phase‐coded SSVEP	Manfred et al. [38]	2007	Significantly expands the number of instruction set of SSVEP
Motion SSVEP	Xie et al. [40]	2012	Provides some relief from visual fatigue problems
Hybrid SSVEP	Panicker et al. [41]	2011	Increases the number of classifiable flicker stimulus targets

3.1 CCA algorithm

CCA has been validated to be a more efficient detection technique for SSVEP recognition. Lin et al. first applied the CCA algorithm to a BCI system based on SSVEP in 2006 [44]. Fig. 4 shows the flowchart of CCA applied to a BCI system based on SSVEP. The CCA algorithm exploits the fact that the autocorrelation of muscle activity is weaker than that of brain activity. CCA is a statistical method for estimating the degree of correlation of two multivariate signals. The largest coefficient between the recorded EEG and reference signals is used to identify a flicking‐target stimulus. The reference signals comprise a series of sinusoidal‐like waveforms at the stimulation frequency and its harmonic frequencies. The CCA‐based frequency detection method has been widely used owing to its high efficiency, robustness, and high accuracy [45, 46]. In a previous study, a SSVEP‐BCI system that used the CCA‐based method achieved high ITRs (approximately 100 bits/min) [47]. CCA has been widely used to recognize flicking targets in SSVEP‐BCI systems owing to its efficiency and training‐free properties.

OPEN IN VIEWER

Fig. 4.

Flowchart of the CCA algorithm for EEG data classification

CCA analyzes SSVEP signals by calculating the typical correlation coefficients of two groups of signals. One group is recorded EEG signal X = [x ₁, x ₂,…, x_n ], where n is the number of channels for collecting the EEG signals. The other group is reference signal Y _i that corresponds to the visual stimulation frequency.

Y i = ( sin ( 2 π f i t ) cos ( 2 π f i t ) … sin ( 2 π k f i t ) sin ( 2 π k f i t ) ) , t = 1 F S , 2 F S , … , N S F S

1

where i is the number of stimulation targets, F _S is the stimulation frequency, k is the number of harmonics in the reference signal, and N _S is the number of sampling points.

The linear combination of X and Y can be expressed as

x = X T W X

2

and

y = Y T W Y

3

where W _X and W _Y are weight matrixes. Thus, the correlation coefficient of X that corresponds to the reference signal that matches the f_i stimulus is

ρ ( f i ) = E ( x T y i ) E ( x T x ) E ( y T y ) = E ( W X T X Y i W Y i ) E ( W X T X X T W X ) E ( W Y i T Y Y T W Y i ) )

4

If K represents the number of stimulation frequencies, the final identified target frequency is

f s = max f i ρ ( f i ) , i = 1 , 2 , … , K

5

3.2 Improved CCA algorithm

Some problems are encountered in the performance of traditional CCA, such as subject differences and easy overfitting. To solve these problems, many scholars have proposed several improved algorithms in which the most typical is the filter bank CCA (FBCCA) algorithm. The FBCCA method was developed to improve the frequency detection of SSVEP‐based BCI systems. The FBCCA algorithm includes the following three procedures: (1) filter bank analysis, (2) CCA between SSVEP subband components and sinusoidal reference signals, and (3) target identification. In a recent study, an SSVEP‐based BCI system that used the FBCCA‐based method has achieved higher ITRs (approximately 151 bits/min) [48].

The typical correlation analysis method of a filter bank aims to improve the accuracy of frequency identification using the harmonic signal in an SSVEP signal. The basic idea of the FBCCA algorithm is that in the SSVEP potential, the amplitude of the fundamental frequency signal is often larger than that of the harmonic signal. In addition, the harmonic signal weakens with the increase in the number of harmonics, but the corresponding SNR only very slightly weakens, which indicates that the harmonic signal has strong identifiability relative to the surrounding noise signal. In a typical correlation analysis algorithm based on CCA filter bank, different filters are designed to fully extract useful frequency‐band information, maximize the correlation coefficient among the SSVEP signal, training‐set template, and preset reference signal, effectively improve the feature‐recognition accuracy of SSVEP, and verify the effectiveness of the algorithm through an online system [49].

3.3 Riemannian geometric classification algorithm

Riemannian geometry was discovered by German mathematician Georg Friedrich Bernhard Riemann, who wrote a paper on a hypothesis based on geometry in 1851 that clearly put forward the existence of another geometry. In EEG signal processing, the matrix is generally run under smooth constraints. Thus, the constrained space can be understood as a smooth curved space. The symmetric positive definite‐matrix space where the commonly used sample covariance matrix is located is a Riemannian manifold. The traditional linear discriminant analysis (LDA), CCA, and other methods use the sample covariance matrix in Euclidean space without considering the curvature of a symmetric positive definite‐matrix space, which is not conducive to accurate establishment of a model [50]. In Riemannian geometry, the Riemannian distance exhibits the invariance of inversion and congruence [51]. The algorithm based on Riemannian geometry features good generalization ability and robustness [52, 53]. Riemannian geometric classification algorithm has been applied to SSVEP‐BCI systems and is generally believed to offer great application prospects [54].

3.4 Machine‐learning algorithm

In addition to the previously mentioned algorithms, machine‐learning algorithms are widely used in the SSVEP‐BCI system. The commonly used classification algorithms in the SSVEP‐BCI system include the following categories: naive Bayesian model [55, 56], decision‐tree model [57 –59], support‐vector machine (SVM) [60 –63], k‐nearest neighbor (KNN) [64 –66], logistic regression [67 –70], and ensemble learning [71 –73]. The Bayesian model is based on Bayesian principle and uses the knowledge of probability and statistics to classify a sample data set. Because of its solid mathematical foundation, the misjudgment rate of the Bayesian classification algorithm is very low. SVM is a type of supervised learning, i.e. a generalized linear classifier, for binary classification of data. Its decision boundary for learning samples is represented by a maximum‐margin hyperplane. The idea of the KNN classification algorithm is that if most of the k most similar (i.e., the nearest) samples of a sample in the feature space belong to a certain category, the sample also belongs to this category. Logistic regression is a generalized linear model; thus, it possesses many similarities with the multiple linear‐regression analysis. Ensemble learning, which is sometimes called a multiclassifier system, completes a learning task by constructing and combining multiple learners. By combining multiple learners, ensemble learning can often achieve more significant generalization performance than a single learner.

In 2020, Rashid et al. [74] used common spatial pattern to extract the features of an SSVEP response and classified these features using SVM. The classification accuracy of the architecture reached up to 88.3%. The experimental results demonstrated that the architecture could be used for classification of an SSVEP‐BCI system and real‐time application of BCI. In 2017, Chatzilari et al. [75] proposed a brain electrolytic‐code algorithm that combined the multichannel CCA decoding and SVM algorithms. The SVM algorithm provided a confidence score, which allowed dynamic adjustment of the length of analysis data and accuracy of the classifier. Through this dynamic adjustment strategy, the algorithm could maximize the system ITR. In 2019, Acampora et al. [70] applied SVM to classify single‐channel SSVEP signals. They used an evolutionary‐algorithm‐based method to select an SVM model from the EEG signals of 11 volunteers, which improved the generalization performance of the SVM algorithm.

3.5 Deep‐learning algorithm

In recent years, deep‐learning algorithms have achieved rapid progress in many fields such as image classification, speech recognition, and recommendation system. Deep‐learning algorithms have also been applied in the field of SSVEP‐BCI system classification algorithm such as convolutional neural network (CNN) [76], recycle neural network [77], and long short‐term memory (LSTM) [78]. However, the deep‐learning algorithm has not shown significant advantages from a comprehensive view of these articles on SSVEP‐BCI, which may be due to the difficulty for scholars to obtain a large number of EEG data and the high SNR of EEG data. Similarly, the current classic deep‐learning networks are algorithms applied to the image and voice fields, and no deep‐learning network has been dedicated to EEG data classification.

3.6 Problems in classification algorithms

The research on decoding algorithms in the BCI field based on VEP has undergone many years of development. The defects and shortcomings of the original decoding algorithm have been greatly minimized, the system performance has significantly improved, and many extensions in application scenarios have been achieved. The innovation of the algorithms promotes the development and practicability of BCI. In recent years, BCI decoding algorithm has achieved great progress in overcoming the high variability and low SNR of EEG signals. The algorithm features of the different SSVEP decoding algorithms are listed in Table 2.

OPEN IN VIEWER

Table 2.

Algorithm features of different SSVEP decoding algorithms.

Algorithm categories	Research team	Year	Algorithm features
CCA algorithm	Lin et al. [44]	2006	Extracts spatial feature information
Improved CCA algorithm	Chen et al. [48]	2015	Analyzes both spatial feature information and frequency feature information
Riemannian geometric classification algorithm	Kalunga et al. [54]	2016	Maps EEG data to Riemannian geometric space
Machine‐learning algorithm	Chatzilari et al. [75]	2017	Has high generalization ability but requires training
Deep‐learning algorithm	Saidutta et al. [76]	2018	Can explore deep‐seated features of EEG data, but the model does not migrate well between different users

Developing a robust BCI decoding algorithm remains necessary to further solve the problems of large variability of EEG signals among individuals, few training samples, and low SNR to realize a universal, noncalibrated, and high‐performance BCI technology. At present, most studies on SSVEP‐BCI still use the CCA algorithm and its improved algorithms. The current CCA and its improved algorithms consider the spatial and frequency feature information but ignore the temporal feature information. The advantage of the EEG data lies in higher temporal resolution, which also contains a large amount of feature information in the temporal dimension. Therefore, in the future, SSVEP decoding algorithms that combine the temporal, spatial, and frequency dimensions can be developed, which are expected to significantly improve the decoding performance. In addition, the application potential of deep‐learning algorithms cannot be ignored. Because of its limitations, the deep‐learning algorithm has not been widely used in the BCI field. On the one hand, most of the existing deep‐learning algorithms that are applied to SSVEP decoding transform EEG signals into the frequency domain and input them into neural networks (e.g., CNN‐structured network structures) for training and finally obtaining classification results. The existing deep‐learning network structures are not good at processing EEG data. Therefore, the focus should be on developing neural network structures that are dedicated to EEG classification. On the other hand, because deep‐learning is a big‐data‐driven algorithm, obtaining a large data set for a single EEG experiment is often difficult. Research on deep‐learning algorithms should focus more on developing transferable models across subjects.

In the evaluation of SSVEP‐BCI decoding algorithms, the classification accuracy, response time, information‐transmission rate, and SNR are mainly used to evaluate the system performance. The aforementioned indexes fail to consider the needs of the actual tasks of a controlled system and cannot be used as the basis for effectively verifying and evaluating the overall performance of a system. Research on BCI evaluation methods from the perspective of “human (user)” and physiological evaluation based on interactive experience are lacking. Designing the evaluation criteria that can evaluate the user‐interaction experience and serve patients with movement disorders is a key problem that needs to be solved in the future.

The BCI decoding algorithm has achieved many breakthroughs and great progress in overcoming EEG signal variability and low SNR. At present, most studies on SSVEP‐BCI still use the CCA algorithm and its improved algorithms. Because of its limitations, deep‐learning algorithms have not been widely used in the BCI field. However, the application potential of the deep‐learning algorithm cannot be ignored. The innovation of the algorithm has promoted the development and practical process of BCI. At present, developing a robust BCI decoding algorithm is necessary to further solve the problems of large EEG signal variability, few training samples, and low SNR among individuals to realize a universal, calibration‐free, and high‐performance BCI technology.