EEG signal recognition algorithm with sample entropy and pattern recognition

Abstract

Brain-computer interface (BCI) is an emerging paradigm to achieve communication between external devices and the human brain. Due to the low signal-to-noise ratio of the original electroencephalograph (EEG) signals, it is different to achieve feature extraction and feature selection, and further high classification accuracy cannot be obtained. To address the above problems, this paper proposes a pattern recognition method that takes into account sample entropy combined with a batch-normalized convolutional neural network. In addition, the sample entropy is used to extract features from the EEG signal data processed by wavelet transform and independent component analysis, and then the extracted data are fed into the convolutional neural network structure to recognize the EEG signal. Based on the comparison of experimental results, it is found that the method proposed in this paper has a high recognition rate.

Keywords

Sample entropy EEG signal batch normalization pattern recognition

1. Introduction

With the development of artificial intelligence (AI), how to make machines with human-like perception, thinking ability and behavioral functions has become a hot issue in current research, and emotional computing technology plays a crucial role in realizing intelligent human-computer interaction, and emotional recognition, as a branch of emotional computing, is the basis and core of realizing human-computer emotional interaction [1]. Compared with non-physiological signals such as speech and expressions, electroencephalograph (EEG) signals can reflect the inner emotional state of a person without human subjective control and have the advantages of non-fake ability, real-time variability, and easy acquisition.

EEG signal-based emotion recognition has potential application prospects in the fields of entertainment, education, criminal investigation, and medical rehabilitation [2]. For example, in the aerospace field, monitoring the emotional state of astronauts can grasp the physiological state of astronauts in real time and help astronauts complete space missions [3, 4]. In clinical medical care, by observing the emotional state of patients, different care measures can be taken to improve the quality of care.

Currently, many researchers have investigated EEG signals. Pandey et al. proposed a subject-independent emotion recognition technique based on EEG signals [5], which extracted features using a variational pattern decomposition technique and extracted subject-independent EEG emotion features from EEG signals using deep neural networks as classifiers, and the experimental results showed that the method improved the classification accuracy by about 6.4%. Kwon et al. used wavelet transforms to obtain a two-dimensional time-frequency mapping of EEG signals, and adapted an adaptive convolutional neural network model to achieve average recognition rates of 76.56%, 80.46%, and 73.43% on the validity and arousal dimensional classification and validity-arousal dimensional quadruple classification of the DEAP dataset, respectively [6]. Li et al. transformed the original physiological signals of each channel into spectrograms to obtain temporal and frequency features, and then used a multi-modal attention-based BiLSTM to automatically learn the best temporal features from them and input them into a deep neural network to predict the emotional output probability of each channel with high accuracy [7]. Rabby et al. divided the EEG signal into multiple non-overlapping signal segments and extracted multiple time-domain, frequency-domain, and nonlinear dynamics features from each segment [8], which were connected into feature sequences for a long time and used to train the long short-term memory (LSTM) classification model, and the experimental results showed that the model obtained 73.87%, 73.50% and 72.80% of classification accuracy. Manish et al. proposed a two-layer two-way gated cyclic unit model with an attention mechanism, which extracted salient features of EEG signals by assigning different weights to local and global EEG features, and achieved 67.9% and 66.5% in validity and arousal dimension binary classification recognition rates, respectively, which were 4.2% and 4.6% higher than the traditional LSTM model [9]. Kevinvric et al. obtained the EEG signal using wavelet packet transform time-frequency decomposition reconstruction, input the instantaneous power signal to DBN for unsupervised training pre-training, and then fine-tuned it by supervised training to achieve automatic feature extraction and pattern classification using a softmax classifier [3].

Although there are numerous methods for EEG emotion recognition, there are still two important problems that need to be studied in depth. First, because the EEG signal is a non-smooth and non-linear random signal, when using the time-frequency domain feature extraction method, although it can improve the correct recognition rate of the EEG signal and shorten the recognition time to a certain extent, it still cannot meet the requirement of control system stability. Secondly, how to construct a more effective deep feature learning and emotion classification model.

In this paper, we study the EEG signal of imaginative movement, firstly we use the sample entropy algorithm to achieve the feature value extraction of the signal after noise reduction and reconstruction and select the sample entropy value of each channel in the imaginary motion period data as the feature value to establish the EEG emotion classification model. And then the feature vector is used for pattern recognition using a deep learning classification algorithm, which improves the recognition rate of EEG signal classification and lays the foundation for the controlling future realization of external devices.

2. EEG signal acquisition and pre-processing

The EEG signals were first generated by a 32-channel EEG acquisition device with a sampling frequency of 51 Hz, and the electrode positions were referenced to the international 10–20 electrode method. In EEG signal acquisition experiments, the 10–20 electrode method is an internationally accepted method of placing electrode positions on the scalp, and the system is designed based on the relationship between the electrode positions and potential areas of the cerebral cortex. Before the acquisition, the parameters of the EEG signal acquisition equipment need to be set, the subject wears an EEG cap to prepare for the acquisition, and then the tester clicks the start acquisition button on the signal acquisition interface, enters the EEG signal acquisition state and then clicks the start button on the stimulation signal generation interface to give the evoked signal and record the subject’s EEG signal. The original signal is depicted in Fig. 1.

Figure 1.

The original signal of EEG.

After the EEG signal is successfully acquired, it needs to be pre-processed to improve the signal-to-noise ratio of the EEG signal due to its very weak amplitude, large background noise, and strong randomness, which is susceptible to external interference. In this paper, wavelet transform and independent component analysis are combined to pre-process the acquired EEG signals. Firstly, the EEG containing various noises is wavelet decomposed by using the multi-resolution property of wavelet transform [8, 9]. In this paper, 3-layer wavelet decomposition is used to obtain sub-band signals of different frequency bands as shown in Fig. 2.

Figure 2.

3-layer wavelet decomposition tree diagram.

In Fig. 2, A indicates the low-pass approximation component (low-frequency component), D indicates the detail component at different scales (high-frequency component), and the end serial number is the index number of decomposition layers. To imagine the left hand as an example of motion, db4 is selected as the wavelet basis function to carry out three layers of small wave decomposition, as shown in Fig. 3.

Figure 3.

The wavelet 3-layer decomposition diagram of imagining the left-hand motion C4 channel.

Then, the sub-band signals of each EEG at the wavelet scale are combined into the input of independent component analysis according to the need, and the signals of each frequency band of EEG are separated by using the blind source separation property of independent component analysis [10]. The advantage of this method is that the spike/spike and slow-wave component signals in the EEG signal can be enhanced while removing artifacts, and then separated and extracted by independent component analysis without involving other parameters and options, which is very concise and effective. Moreover, no signal loss will occur if it is chosen to decompose the signal using a continuous wavelet transform. In this paper, the continuous wavelet transforms soft threshold method is used to analyze the original EEG signal of the C4 channel on the left-hand motion as shown in Fig. 4.

Figure 4.

C4 channel primitive EEG of imagining left-hand motion.

The principle of the algorithm is as follows: assuming that there are $N$ leads of EEG data $x(t)=[{x_{1}(t),x_{2}(t),\cdots x_{N}(t)}]^{T}$ to be processed, the wavelet decomposition is first performed on each lead $x_{i}$ with a decomposition layer $L$ , so that the EEG data of each lead is decomposed into the subband signals of: Then the sub-band signals of each lead EEG at the same scale or multiple scales are selected as needed to form the input $\hat{x}=[{\hat{x}_{1},\hat{x}_{2},\cdots\hat{x}_{N}}]^{T}$ for independent component analysis. $\hat{x}_{i}$ is defined as follows.

$\displaystyle\hat{x}_{i}=\hat{x}_{in}+\cdots\hat{x}_{im}=\sum\limits_{j=n}^{m}% {\hat{x}_{ij}}$ (1)

where $m$ and $n$ denote the $m$ -th and $n$ -th subband signals of the $i$ -th lead data, respectively, and their specific values depend on the EEG signal to be analyzed. After noise reduction processing, the reconstructed signal after soft threshold processing is obtained as shown in Fig. 5. The signal ratio after noise reduction can be seen in Fig. 5. The original signal waveform is smooth.

Figure 5.

C4 channel reconstructed EEG after soft threshold noise reduction of imagining left-hand motion.

3. Feature extraction of EEG signals

EEG signal is a non-smooth and non-linear random signal, and sample entropy is a non-linear method that is sensitive to small fluctuations of the signal so that the complexity of the signal time series can be further measured. Therefore, this paper selects the sample entropy method to extract the eigenvalues of EEG signals [11, 12].

The sample entropy is a statistic that does not count the matches proposed by Pincus based on approximate entropy. The sample entropy has all the advantages of approximate entropy while avoiding the problem of inconsistent statistics in approximate entropy [13]. Therefore, the sample entropy is used to quantize the probability of incoming patterns in the time series. The larger the sample entropy value is, the greater the probability of incoming patterns in the time series and the more complex the series is. The detailed procedures to calculate the sample entropy are shown as follows.

(1) The raw EEG signal sequences were sequentially composed into a $m$ dimensional vector.

$\displaystyle K_{m}(i)=[k(i),k(i+1),\cdots k(N+m-1)],1\leqslant i\leqslant N-m+1$ (2)

(2) Define the distance between the vector $K_{m}(i)$ and $K_{m}(s)$ as the largest of the differences between the corresponding elements of the two vectors.

$\displaystyle d[{K_{m}(i),K_{m}(s)}]=\mathop{\max}\limits_{l=0,1,\cdots,m-1}|k% (i+l)-k(s+l)|$ (3)

In Eq. (3), $i,s=1,2,\cdots,N-m;i\neq s$ .

(3) Given that the similarity capacity $r({r>0})$ , $N-m$ and $d[{K_{m}(i),K_{m}(s)}]$ are compared with the similarity capacity, respectively, and when the calculated distance is less than $r$ , the 2 subsequences corresponding to the current distance are considered similar to each other, after which the ratio of the statistically derived number to the total number of vectors $N-m-1$ is noted as.

$\displaystyle B_{i}^{m}(r)=\frac{\textit{number}\{{[{d(K_{m}(i),K_{m}(s))}]{r}% }\}}{N-m-1}$ (4)

(4) The average of the calculation $B_{i}^{m}(r)(i=1,2,\cdots,N-m)$ can be noted as $B^{m}(r)$ .

$\displaystyle B^{m}(r)=\frac{1}{N-m}\sum\limits_{i=1}^{N-m}{B_{i}^{m}(r)}$ (5)

(5) Repeat steps (1) to (4) to transform the $m$ dimension into $m+1$ dimensions, $B_{i}^{m+1}(r)$ and $B^{m+1}(r)$ can be further achieved.

$\displaystyle B_{i}^{m+1}(r)=\frac{\textit{number}\{{[{d(K_{m+1}(i),K_{m+1}(s)% )}]{r}}\}}{N-m-1}$ (6) $\displaystyle B^{m+1}(r)=\frac{1}{N-m}\sum\limits_{i=1}^{N-m}{B_{i}^{m}+1(r)}$ (7)

(6) The sample entropy of the sequence is defined as follows.

$\displaystyle\textit{SampEn}(m,r)=-\mathop{\lim}\limits_{N\to\infty}\ln\frac{B% ^{m+1}(r)}{B^{m}(r)}$ (8)

(7) In practice, due to the limited length of the sequence, the estimation value of the sample entropy of the sequence with several points $N$ is used.

$\displaystyle\textit{SampEn}(m,r,N)=-\ln\frac{B^{m+1}(r)}{B^{m}(r)}$ (9)

where SampEn is determined by the values of the parameters $m$ , $r$ , and $N$ . Pincus’ study concluded that $m=2$ makes $r$ taking values of 0.1 $\sim$ 0.25SD (where SD is the standard deviation of the original series) yielded valid statistical features [14]. Based on the above principles, $m=$ 2, $r=$ 0.2SD, $N$ is chosen as the input data point in this paper.

The ERD/ERS phenomenon of imaginary hand movements is mainly concentrated at around 10 Hz and around 20 to 24 Hz. When imagining the left hand moving, it can be seen from Fig. 6a that C4 (ERD) is near 10 Hz. The amplitude of the region is smaller than that of the C3 (ERS) region. In contrast, it can be seen from Fig. 6b that the C3 (ERD) amplitude is around 10 Hz during right-hand movement in the middle image. The value is smaller than the amplitude of the C4 (ERS) region.

Figure 6.

Spectrogram of imagining left-band motion (a) and right-hand motion (b).

4. Pattern recognition of EEG signals

Pattern recognition is a crucial step in brain-computer interface technology, and the key to controlling external devices lies in the accuracy of classifier recognition results [15, 16, 17]. Only by accurately classifying EEG signals can human-computer interaction be perfectly realized. In pattern recognition of EEG signals, a convolutional neural network algorithm is used to accomplish EEG signal recognition and use batch normalization to make the parameter search problem easy and suppress model overfitting.

Because the network is trained with the data in the input layer (because the samples in the input layer have been artificially normalized), the parameters of each subsequent layer will change with the input layer, and batch normalization can address the situation where the distribution of the data in the middle changes [18]. When training the model, batch normalization mainly uses the mean and variance on small batches and maps them to a region with a variance of 1 and a mean of 0. The intermediate output of the convolutional neural network is continuously adjusted so that the output of the whole network is more stable in each layer [19, 20, 21]. The batch normalization algorithm proceeds as follows.

(1) Calculate the mean $\mu_{B}$ and the variance $\sigma_{B}^{2}$ of the neurons.

$\displaystyle\mu_{B}=\frac{1}{C}\sum\limits_{i=1}^{c}{x_{i}}$

(10) $\displaystyle\sigma_{B}^{2}=\frac{1}{C}\sum\limits_{i=1}^{c}{(x_{i}}-\mu_{B})^% {2}$

(2) Batch normalization operation for the current batch of input data.

$\displaystyle\hat{x}_{i}=\frac{x_{i}-\mu_{B}(x_{i})}{\sqrt{\sigma_{B}^{2}+% \varepsilon}}$ (11)

(3) Reconstructive transformation of the batch normalized data by the learnable parameters $\gamma$ and $\beta$ to output the final data.

$\displaystyle y_{i}=\gamma\hat{x}_{i}+\beta$ (12)

where $x_{i}$ is the neuron of the current layer, subscript $i=1,2,\cdots,c$ , and $\varepsilon$ is a positive constant close to 0, which is used to indicate the stability of the value in the transform reconstruction.

Convolutional neural networks are a class of feedforward neural networks that include convolutional computation and have a deep structure [22]. They automatically extract various features of the input signal through multi-layer convolution and pooling and then obtain the expected classification results through the fully connected layers and classifiers deployed at the end. The pattern recognition architecture proposed in this paper is shown in Fig. 7.

Figure 7.
Architecture of the proposed pattern recognition.

The convolutional neural network structure for feature classification consists of four convolutional layers, four maximum pooling layers, two fully connected layers, and one SoftMax layer. For the convolutional layers, the ReLU function is used as the activation function, the convolutional kernel is 3, and the padding is “same”, which represents the padding of the input so that the output has the same length as the original input.

In this paper, the sample entropy value of all EEG data within 3 $\sim$ 6 s of each channel was obtained, and the relationship between each channel in the three kinds of imaginary actions was obtained. As shown in Fig. 8a, when left-hand movement is imagined, the ERD phenomenon occurs in the C4 channel. The complexity of the C4 brain region is higher than that of the C3 brain region, and the sample entropy value is higher than that of the C3 brain region. As shown in Fig. 8b, when the right-hand movement is imagined, ERD occurs in the C3 channel, the complexity of the C3 brain region is higher than C4 brain region, and the sample entropy value is higher than that of the C4 brain region.

Figure 8.
Sample entropy diagram of imagining left-band motion (a) and right-hand motion (b).

In addition, the foot movement is imagined as shown in Fig. 9, the ERS phenomenon occurs in C3 and C4 channels, the EEG complexity of the foot movement is low, and the sample entropy becomes low. The ERD phenomenon occurs in Cz channels due to both the increasing EEG complexity and the high sample entropy.

Figure 9.
Sample entropy diagram of imagining foot motion.

5. Results and discussion

The classification accuracies of each subject at each period were recorded and averaged to obtain the classification accuracy of each subject. The classification accuracy of each subject was averaged to obtain the classification accuracy of each subject. Finally, the classification accuracy of the 15 subjects was averaged to obtain the overall average classification accuracy. The classification accuracy of each subject during the experiment is recorded in Table 1.

As can be seen from Table 1, the different subjects have differences in classification accuracy, where for learning vector quantization (LVQ) neural network, the average classification accuracy over imagining motion is 87% (lowest) and the accuracy of recognition for left-hand, right-hand, and foot is 90%, 83%, and 87%, respectively. For BP neural network, the average classification accuracy over imagining motion is 92% (middle), which reflects individual variability. However, for the same subject, there were fluctuations in classification accuracy at different periods. Moreover, for the convolution neural network, the average classification accuracy over imagining motion was 93% (highest).

Table 1
Classification results of three motor imageries for different neural networks

EEG signal classifier	Imagining motion (40 $+$ 30)	Number of correctly identified samples	Total sample number of test sets	Accuracy of recognition (%)	Overall recognition rate (%)
LVQ neural network	Left-band	27	30	90	87
	Right-hand	25	30	83
	Foot	26	30	87
BP neural network	Left-band	28	30	93	92
	Right-hand	27	30	90
	Foot	28	30	93
Convolution neural network	Left-band	29	30	97	93
(proposed)	Right-hand	27	30	90
	Foot	28	30	93

For comparison, the performance of the proposed method in this paper is compared with other better models. It can be seen from Table 1 that the proposed method combining sample entropy and batch-normalized convolutional neural network demonstrates better performance. The convolution neural network, whether for a single limb imagine movement EEG classification result or effect on the overall classification is superior to the former two kinds of classifier, and the characteristics of three kinds of imagine movement EEG signals vector classification has improved significantly, the overall recognition rate reached 93%. Considering the portability and high recognition rate of the convolution neural network, the convolution neural network is used as a pattern recognition classifier of the sample entropy feature vector to provide control signals for prosthesis control.

6. Conclusion

Emotion plays a crucial role in human life, and in recent years, the field of artificial intelligence has increasingly focused on the research of emotion recognition. Benefiting from the successful application of deep neural networks in feature extraction and classification recognition, various new methods for the emotion recognition of EEG signals have emerged. However, due to the diversity and complexity of EEG signals, different brain regions do not experience emotions to the same degree, and these factors greatly increase the difficulty of EEG emotion recognition, and how to improve the recognition rate of EEG signal-based emotion classification is still a challenge to be solved.

In this paper, we propose an EEG signal emotion recognition method that combines sample entropy and batches normalized convolutional neural network. The EEG signals collected by EEG acquisition devices are processed by noise reduction using wavelet transform and independent component analysis, and the effective EEG feature signals are extracted using the sample entropy method, while the extracted EEG feature signals are recognized later, an average classification accuracy of over 93% is achieved. The effect of increasing the number of convolutional layers to tap deeper channel features on the accuracy of emotion classification can be further studied and analyzed later. In addition, combining human physiological signals and external emotion representation carriers such as expressions, speech and behavioral postures for multimodal emotion recognition research is also of great practical importance to further improve the accuracy of emotion recognition.

Footnotes

Funding

This work was supported by the General Program of Chongqing Natural Science Foundation (No. cstc2021jcyj-msxm2774).

References

Zheng

Liu

Cichocki

. Emotionmeter: A multimodal framework for recognizing human emotions. IEEE T Cybernetics. 2018; 49(3): 1110-1122.

Congedo

Barachant

Bhatia

. Riemannian geometry for EEG-based brain-computer interfaces; a primer and a review. Brain-Comput Interfac. 2017; 4(3): 155-174.

Kevinvric

Subasi

. Comparison of signal decomposition methods in classification of EEG signals for motor-imagery BCI system. Biomed Signal Proces. 2017; 31: 398-406.

Zhang

Yin

Chen

Nichele

. Emotion recognition using multi-modal data and machine learning techniques: A tutorial and review. Inform Fusion. 2020; 59: 103-126.

Pandey

Seja

. Subject independent emotion recognition from EEG using VMD and deep learning. J King Saud Univ-Com. 2019; 34(5): 1730-1738.

Kwon

Shin

Kim

. Electroencephalography based fusion two-dimensional (2D)-convolution neural networks (CNN) model for emotion recognition system. Sensors. 2018; 18(5): 1383.

Bao

Zhao

. Exploring temporal representations by leveraging attention-based bidirectional LSTM-RNNs for multi-modal emotion recognition. Inform Process Manag. 2020; 57(3): 102185.

Rabby

MKM

Islam

AKMK

Belkasim

Bikdash

. Wavelet transform-based feature extraction approach for epileptic seizure classification. In: Proceedings of the 2021 ACM Southeast Conference. 2021 April; Virtual Event, USA. New York: Association for Computing Machinery; 2021. pp. 164-169.

Manish

Ram

Acharya

. A new approach to characterize epileptic seizures using analytic time-frequency flexible wavelet transform and fractal dimension. Pattern Recogn Lett. 2017; 94: 172-179.

10.

Meng

Chen

Jiang

Liu

Fan

Dai

, et al. Evaluation of decomposition parameters for high-density surface electromyogram using fast independent component analysis algorithm. Biomed Signal Proces. 2022; 75: 103615.

11.

Rizal

Hadiyoso

. Sample entropy on multidistance signal level difference for epileptic EEG classification. Sci World J. 2018; 8463256.

12.

Liu

Fan

Abbod

Shieh

. Sample entropy analysis for the estimating depth of anesthesia through human EEG signal at different levels of unconsciousness during surgeries. PeerJ. 2018; 6: e4817.

13.

Thomas

Vinod

. Biometric identification of persons using sample entropy features of EEG during rest state. In: 2016 IEEE International Conference on Systems, Man, and Cybernetics (SMC). 2016 October 09–12; Budapest, Hungary. New York: IEEE; 2016. pp. 003487-003492.

14.

Pincus

. Approximate entropy (ApEn) as a complexity measure. Chaos: Interdiscip J Nonlinear Sci. 1995; 5(1): 110-117.

15.

Chen

Zhang

Ouyang

Liu

Shen

Wang

. “BNN-BN=?”: Training binary neural networks without batch normalization. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021 June 19–25; Electr Network. New York: IEEE; 2021. pp. 4619-4629.

16.

Ergen

Sahiner

Ozturkler

Pauly

Mardani

Pilanci

. Demystifying batch normalization in relu networks: Equivalent convex optimization models and implicit regularization. arXiv preprint arXiv: 2103.01499, 2021.

17.

Merlin

Angelin

Thomas

. Deep learning techniques for EEG signal applications – A review. IETE J Res. 2022; 68(4): 3030-3037.

18.

Kobler

Hirayama

Kawanabe

. Controlling The Fréchet variance improves batch normalization on the symmetric positive definite manifold. In: ICASSP 2022–2022 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP). 2022 May 23–27; Singapore. New York: IEEE; 2022. pp. 3863-3867.

19.

Mukhtar

Qaisar

Zaguia

. Deep convolutional neural network regularization for alcoholism detection using EEG signals. Sensors. 2021; 21(16): 5456.

20.

Maheshwari

Ghosh

Tripathy

Sharma

Acharya

. Automated accurate emotion recognition system using rhythm-specific deep convolutional neural network technique with multi-channel EEG signals. Comput Biol Med. 2021; 134: 104428.

21.

Ouyang

Yuan

Guo

, et al. An EEG data processing approach for emotion recognition. IEEE Sens J. 2022; 22(11): 10751-10763.

22.

Kobler

Hirayama

Zhao

Kawanabe

. SPD domain-specific batch normalization to crack interpretable unsupervised domain adaptation in EEG. arXiv preprint arXiv: 2206.01323, 2022.

EEG signal recognition algorithm with sample entropy and pattern recognition

Abstract

Keywords

1. Introduction

2. EEG signal acquisition and pre-processing

Table 1 Classification results of three motor imageries for different neural networks

Footnotes

Funding

References

Table 1
Classification results of three motor imageries for different neural networks