An active and passive upper limb rehabilitation training system based on a hybrid brain

Abstract

Movement function rehabilitation of patients with craniocerebral injuries is an important issue facing neurorehabilitation science. The use of brain–computer interface technology in rehabilitation training systems can allow patients to actively participate in the rehabilitation training process and use the brain’s neuroplasticity to enhance the effects from rehabilitation training. At present, the brain–computer interface-based rehabilitation training system still has problems such as insufficient active participation of patients, resulting in slowed motor neural circuit repair or low action execution accuracy. In response to the above problems, this paper designed an active and passive upper limb rehabilitation training system based on a hybrid brain–computer interface of steady-state visual evoked potentials (SSVEP) and movement-related cortical potentials (MRCPs). The system includes six parts: task setting and training guidance module, EEG signal acquisition module, EEG signal preprocessing and recognition module, rehabilitation training module, training completion evaluation module, and communication module. The system drives the rehabilitation robot to complete the training actions by identifying the participant’s SSVEP and evaluates the completion of the rehabilitation training based on the patient’s movement intention recognition results. In this study, 12 participants were recruited. In the online test, the system achieved an average action execution accuracy of 99.3%. The movement intention detection based on MRCPs reached an average accuracy of 82.7%. The participants’ average completion rate was 0.91. The experimental results show that the system can achieve a high rate of execution accuracy. In addition, it can evaluate the active participation level of patients in rehabilitation training based on the movement intention detection results, accelerate the reconstruction of motor neural circuits, improve the effects of training, and provide more effective ways of thinking for the study of upper limb rehabilitation training systems for patients with craniocerebral injuries.

Keywords

EEG Hybrid-BCI motor dysfunction upper limb rehabilitation training training evaluation

1 Introduction

Motor dysfunction is an important manifestation of neurological impairment after craniocerebral injury. Stroke is the main cause of brain neurological impairment. For surviving stroke patients, 80–90% have a loss of motor function (Langhorne, Coupar, & Pollock, 2009; Stinear, 2010; Stinear, 2017), and among them, the incidence of upper limb motor dysfunction is about 77% (Lawrence et al., 2001). The lack of motor function in patients has placed a heavy burden on families and society and become a global public issue (Katan & Luft, 2018; Yang et al., 2019). The results of neuroplasticity research show that, in addition to some interventional treatments (Zhao et al., 2018), a large number of target-based repetitive active training can also repair or reorganize brain networks and functions so that patients can regain motor functions (Nudo, 2006; Wolpaw, 2012). Moreover, timely and effective feedback can also improve the effect of rehabilitation training (Cirstea & Levin, 2007; Cramer et al., 2011; Parker, Mountain, & Hammerton, 2011; Rong et al., 2021). However, because the affected limbs of some patients with craniocerebral injury have no motor function, understanding how to use effective methods to regain the upper limb function of the patient has become the focus of research in the field of sports rehabilitation.

Traditional training based on manual training and simple rehabilitation equipment has brought a greater workload to medical staff. The rapid development of rehabilitation robots provides a convenient way to rehabilitate patients and reduces the work pressure on medical staff (Hesse, Mehrholz, & Werner, 2008; Kwakkel, Kollen, & Krebs, 2008; Nordin, Xie, & Wünsche, 2014). However, because rehabilitation robots perform repetitive mechanical training on the patient’s limbs in accordance with established procedures, and because the patient’s initiative to train is poor, the rehabilitation effect of the patient’s brain nerve function is limited (Kakuda, 2020). Recent studies have found that motor imagery, (MI) and actual exercise can activate the same brain area. By simulating limb movement in the heart, it can enhance the patient’s brain motor network and improve the patient’s motor function, and is thereby suitable for stroke patients at any stage. However, because the motor imagery therapy is not actual limb movement, the patients themselves cannot get the corresponding feedback, and the clinician cannot correctly measure the patient’s compliance with the MI task, so optimal treatment cannot be achieved (Johnson, Sprehn, & Saykin, 2002; Machado, Carregosa, Santos, Ribeiro, & Melo, 2019; Sharma, Pomeroy, & Baron, 2006). Therefore, there is an urgent need for a rehabilitation training system that can train the brain and limbs of the patient at the same time, and also provide timely feedback to the patient, so as to improve the rehabilitation efficiency of the patient’s brain.

The brain–computer interface (BCI) system for motor imaging recognizes the participants’ motor intentions by analyzing the event-related synchronization (ERS) and event-related desynchronization (ERD) phenomena of mu rhythm (8–12 Hz) and beta rhythm (13–30 Hz) in EEG signals (Cantillo-Negrete et al., 2021; Chaudhary, Birbaumer, & Ramos-Murguialday, 2016; Cheng et al., 2020; Choi, Kwon, Lee, & Nam, 2020; Lazarou, Nikolopoulos, Petrantonakis, Kompatsiaris, & Tsolaki, 2018; Sebastián-Romagosa et al., 2020; Silvoni et al., 2011; Wolpaw, Birbaumer, McFarland, Pfurtscheller, & Vaughan, 2002). It provides nerve control feedback for motor imagery therapy, and has also become a control method of rehabilitation equipment (Buch et al., 2008; Chew et al., 2020; Hashimoto et al., 2021; Monge-Pereira et al., 2017; Jonathan R. Wolpaw, 2007). However, the rehabilitation training system based on the BCI technology of motor imagery has the following problems which limit its clinical application: (1) There is the phenomenon of BCI illiteracy, and the recognition rate of EEG signals of some patients with motor imagery is very low. The method of rehabilitation training of the motor imagery BCI will dampen their confidence in training; (2) The single recognition rate of the motor imagery EEG signal is relatively low, which also makes the application of the BCI based on motor imagery subject to some restrictions (Machado, Carregosa, Santos, Ribeiro, & Melo, 2019; Molinari & Masciullo, 2020; Simon, Bolton, Kennedy, Soekadar, & Ruddy, 2021).

To improve the accuracy of brain electrical signal recognition and the number of actions in rehabilitation training, some researchers have used SSVEP-based BCIs for rehabilitation training of stroke patients (Horki, Neuper, Pfurtscheller, & Muller-Putz, 2010; Lim & Ku, 2018). The BCI system based on SSVEP has a large number of commands and high recognition and information transmission rates. It has good prospective applications such as assistance for the disabled, military, and entertainment (Chen, Chen, Gao, & Gao, 2013; Han et al., 2020). Horki et al. used the SSVEP signal to control the two-dimensional motion of a neural prosthesis, achieving the highest true positive rate of 83% and the smallest false negative of 1% (Horki et al., 2010); Gui et al. (Gui, Ren, & Zhang, 2015; McGeady, Vuckovic, & Puthusserypady, 2019) designed a rehabilitation training system based on the SSVEP BCI to control the exoskeleton rehabilitation robot. They used a steady-state, visually evoked potential to identify four intent controls related to the walking mode. The recognition rate of BCI to complete the corresponding mode and switch is over 90%, and the delay is about 1.52 seconds. Although the BCI system based on SSVEP is not limited by the patient’s state during rehabilitation training, it also has high applicability. However, because it is an evoked potential, the patient can control the rehabilitation robot without performing motor imagery or thinking. The participation and initiative of the patient are not as high as the rehabilitation training system based on the BCI of motor imagery.

Recent research results have shown that movement-related cortical potentials (MRCPs) can be used to detect the participant’s motor intention. MRCPs are a type of slow potential that are related to the movement process that starts two seconds before motor execution and motor imagery. It can reflect the dynamic processing of motor preparation, motor execution, and the end of the movement (Giesche et al., 2021; Niazi et al., 2011; Riaz et al., 2014). Due to the high stability of MRCPs, the difference between participants is relatively small, and the recognition rate is relatively high. Motor intention recognition based on MRCPs has also attracted increasing attention, but there are few reports on its use in rehabilitation training for stroke patients. Eduardo et al. used MRCPs to decode the gait intentions of healthy participants and paralyzed patients and triggered the exoskeleton. The results showed that the correct decoding rate of motor intention of healthy participants and paralyzed patients were 84.44±14.56% and 77.6 1±14.72%, respectively (Bhagat et al., 2016). However, there has been no report on the use of MRCPs for upper limb motor function rehabilitation training.

In summary, the upper limb rehabilitation training system based on BCI control has the problems of low control accuracy and insufficient patient participation. To improve the performance of the brain-controlled upper limb rehabilitation training system and the patient’s rehabilitation, this paper designs an active and passive upper limb rehabilitation training system based on the SSVEP and MRCP hybrid BCI. It uses a stable SSVEP to drive the upper limb rehabilitation robot to improve brain control accuracy and improves the experience of patients. It also uses a recognition rate of MRCPs to evaluate the degree of active participation in the training of patients. Finally, combining the accuracy of brain control and the detection rate of motor intention, an evaluation index for the effect of active and passive rehabilitation training is proposed.

2 System design and algorithm

Neuroscience research shows that a large number of target-based repetitive training and timely neurofeedback can promote the repair or reconstruction of motor neural circuits. In particular, active rehabilitation training can improve the rehabilitation effect of patients with motor dysfunction. However, the low recognition rate based on the MI-BCI system and BCI illiteracy will reduce a patient’s sense of experience and increase the patient’s frustration. Although the rehabilitation training system based on SSVEP-BCI has a higher recognition rate, the patient participation level is insufficient, which affects rehabilitation. Therefore, there is an urgent need for a brain-controlled rehabilitation training system that not only has high brain control accuracy, but also enables patients to actively participate, so that the upper-limb rehabilitation training system based on brain control can exit the laboratory and be used in clinical practice. In view of the above problems, combining the advantages of different BCI systems, this article considers introducing a hybrid active and passive BCI system to improve the rehabilitation training effect of patients with motor dysfunction, making full use of the high recognition rate of SSVEP and the high active participation of motor intentions. It uses the recognition results of the SSVEP signal to control the rehabilitation robot and improves the patient’s control and experience using the rehabilitation robot as well as increasing the neurofeedback effect. The detection rate of the MRCPs signal is used to evaluate the patient’s active participation in rehabilitation training. Finally, a training completion index based on manipulation accuracy and active participation is defined to evaluate the therapeutic effect of a patient’s rehabilitation training.

2.1 System design

The active and passive upper limb rehabilitation training system based on hybrid BCI includes six parts as shown in Fig. 1: task setting and training guidance module, EEG signal acquisition module, EEG signal preprocessing and recognition module, rehabilitation training module, training completion evaluation module, and communication module. The system drives the rehabilitation robot to complete the training actions by recognizing the patient’s SSVEP characteristics and combines the patient’s motor intention detection rate to evaluate the patient’s training completion. This is done to achieve direct control of the patient’s brain on the rehabilitation arm and training effect feedback, as well as to promote rehabilitation of the patient’s limbs and cranial nerve circuits.

Fig. 1

System design flowchart.

2.2 Main modules and functions of the system

2.2.1 Task setting and training guidance module

To improve the effect of rehabilitation training, an experimental paradigm of SSVEP plus pictures to guide motor intentions were designed. The pictures of rehabilitation training actions are used as the flashing blocks of SSVEP. During rehabilitation training, patients are required to try to respond while looking at the flashing blocks. To improve the recognition rate of the patient’s brain’s electrical signal, this study focuses on the characteristics of the patient’s repeated training of a certain action, sets the same SSVEP frequency for different actions, and only decodes the brain’s electrical signal for different training actions, instead of classifying actions.

Before the task starts, one needs to choose a rehabilitation action based on the patient’s situation. The system has six actions to choose from, namely raising the arm, lowering the arm, turning the wrist to the left, turning the wrist to the right, moving the arm to the left, moving the arm to the right, and curling and extending the fingers. In addition, one can also set the number of training actions and the duration of each trial. After this part is set up, the rehabilitation training process begins, as shown in Fig. 2. The experimental part of this article is mainly based on the action of raising the arm. At the beginning of the training, there is a preparation time of 2 s, and then one enters the task interface. In this interface, there is a virtual arm picture flashing at a frequency of 7.5 Hz in the center of the screen, and the duration can be set. This article takes 3 s as an example for research. During the flashing process participants were asked to concentrate on the flashing of the flashing block and perform the corresponding movement tasks as best they could. After the flashing, a black screen emerged, allowing the patient to rest for 2–4 s before proceeding to the next task. The number of tasks was the same as for the previous setting.

Fig. 2

Experimental stimulation pattern.

2.2.2 EEG signal acquisition module

Considering the operability and comfort in the rehabilitation training process, the EEG collection function in this article uses Wearable Sensing’s DSI-24 dry electrode EEG system. The system is wearable and portable, works for over 8 hours, and supports Bluetooth wireless transmission. The DSI-24 system has a total of 24 electrodes, and the electrode setting adopts the 10–20 international standard system, as shown in Fig. 3. Among them, 21 channels are used for the synchronous acquisition of EEG signals, and the remaining three channels are extended, which can be used to collect EMG, ECG, and other electrophysiological signals. In this paper, 21 EEG signal channels are used in the process of EEG data acquisition, and the frequency is set to 300 Hz. The channel in the red area on the upper part of Fig. 3b is used to identify the movement intention. The movement-related cortical potentials (MRCPs) and event-related desynchronization (ERD) occur mainly in the frontal and parietal lobes (Di Russo et al., 2017; Edelman, Baxter, & He, 2016), so electrodes FP1, FP2, F3, F4, Fz, C3, C4, and Cz near these two brain regions were selected. The channel in the red area at the bottom of Fig. 3b is used to identify steady-state visually evoked potentials. SSVEP mainly appears in the occipital lobe (Chen, Wang, Zhang, & Gao, 2018; Islam, Molla, Nakanishi, & Tanaka, 2017), so the Occipital lobe electrodes O1 and O2 were selected.

Fig. 3

EEG collection equipment (a) DSI-24 EEG collection device; (b) Spatial distribution of lead positions.

2.2.3 EEG signal preprocessing and recognition module

(1) EEG signal preprocessing

In this study, dry electrodes were used to collect EEG signals. EEG signals are susceptible to interference and produce artifacts. Common artifacts include EOG artifacts, EMG artifacts, and baseline drift. These interferences will affect the analysis of the signal, so that after the signal is collected, some preprocessing should be done on the original signal. The EEG preprocessing in this study includes two parts: steady-state visually evoked potentials and motor intention recognition. The stimulation frequency of steady-state visually evoked potentials is 7.5 Hz, and most of the interference can be removed by high-pass filtering. Therefore, the preprocessing is mainly 5 Hz high-pass filtering. The motor intention recognition method based on MRCPs is susceptible to the interference of the EOG artifacts. This study mainly used the automatic EOG removal method based on wavelet coefficients to remove the EOG (Betta et al., 2013), and then performed a 0.1–1 Hz band-pass filter. Movement intention recognition based on mu and beta rhythms is mainly 8–30 Hz band-pass filtering. Through the above preprocessing process, EEG data that meets the requirements can be obtained.

(2) Feature extraction and recognition module

The EEG signal processing flow of this system is shown in Fig. 4. The identification of SSVEP features is mainly done through canonical correlation analysis (CCA); the motor intention recognition method based on MRCPs is performed through the features and support vector machine (SVM) of MRCPs; the motor intention recognition method based on mu and beta rhythms uses common spatial pattern (CSP) and SVM.

Fig. 4

EEG signal processing flowchart.

1) Visually evoked potential recognition

CCA is a statistical method (Lin, Zhang, Wu, & Gao, 2006; Zhang, Zhou, Jin, Wang, & Cichocki, 2014), used to measure potential correlation between two multidimensional variables. The CCA method solves the correlation coefficient by referring to the two sets of multi-dimensional variables X and Y and their linear combination x = X^TW_X and y = Y^TW_Y to find the maximum weight vectors W_X and W_Y of the vectors x and y. $max_{W_{X}, W_{Y}} ρ (x, y) = \frac{E [W_{X}^{T} {XY}^{T} W_{Y}]}{\sqrt{E [W_{X}^{T} {XX}^{T} W_{X}] E [W_{Y}^{T} {YY}^{T} W_{Y}]}}$ (1)

In the frequency detection of SSVEP, X represents multiple SSVEP channels and Y represents reference signals. To detect the frequency of SSVEP in an unsupervised manner, a sinusoidal signal is used as a reference signal Y_f: $Y_{f} = [\begin{matrix} \sin (2 π ft) \\ \cos (2 π ft) \\ \begin{matrix} ⋮ \\ \sin (2 π N_{h} ft) \\ \cos (2 π N_{h} ft) \end{matrix} \end{matrix}]$ (2)

where f is the stimulation frequency, and N_h is the number of harmonics. To identify the frequency of SSVEP, CCA calculates the correlation between the multi-channel SSVEP and the reference signal corresponding to each stimulation frequency. The frequency of the reference signal with the greatest correlation is considered the frequency of SSVEP.

2) Motor intention recognition method based on MRCPs

The motor intention recognition method based on MRCPs refers to the pre-movement preparation potential, not the usual motor imagery. The recognition of motor intention includes the following steps:

Convert reference electrode, convert reference electrode to CAR or RESR reference (Zhang et al., 2020);

Filter the data by 0.1–1 Hz;

Downsampling of the data, the sampling rate is reduced to 10 Hz;

Select the data of the forehead area 1.5 s after the stimulus starts to flash as the feature;

Use the support vector machine to classify (Note, the model here is the model trained for the participant, if there is a new participant, the model needs to be retrained).

3) Movement intention recognition method based on mu and beta rhythms

The EEG features in the motor imagery process detected by the mu and beta rhythm-based movement intention recognition method use the traditional CSP (co-space mode) (Müller-Gerking, Pfurtscheller, & Flyvbjerg, 1999) and support vector machine algorithms, which specifically includes the following steps:

Convert reference electrode to CAR or RESR reference;

Filter the data by 8–30 Hz;

Select the data of the forehead 1.5 s after the stimulus starts to flicker, and use the multi-type CSP algorithm to perform spatial filtering;

Extract the energy of each channel after spatial filtering as a feature;

Use the support vector machine to classify (Note: the model here is the model trained for the participant, if there is a new participant, the model needs to be retrained).

The classification features used in this paper are shown in Table 1.

Table 1

Classification features used in the paper

	SSVEP	MRCPs	ERD
Feature	Two-channel SSVEP data	Time domain features after down-sampling	Energy features after CSP

2.2.4 Rehabilitation training module

The upper limb rehabilitation robot is provided by Huibo Shenfang Rehabilitation Robot Company, which can be used for secondary development.

2.2.5 Communication module

The EEG signal processing module and the rehabilitation robot use Bluetooth to communicate, and the control commands output by the online system are converted to Bluetooth through the serial port and sent to the rehabilitation robot for execution. Using Bluetooth can reduce the wired connection between devices.

2.2.6 Training completion evaluation module

This study combines the results of motor intention recognition and SSVEP recognition to evaluate the participants’ training completion. The specific calculation formula is as follows: $SOC = (ACCSSVEP + ACCMI) / 2$ (3)

Here, SOC represents the degree of training completion, which is between 0 and 1; ACCSSVEP represents the recognition rate of steady-state visually evoked potentials, and ACCMI represents the recognition rate of movement intention.

3 Experimental test and results

3.1 Online test

In this experiment, a total of 12 healthy participants aged 18–30 years old were selected, including eight boys and four girls, who were right-handed and had no neurological diseases. The experimental environment was in a room with normal light and good sound insulation. Participants were sitting in front of the prompt screen, as shown in Fig. 5. To prevent the artifact signals generated by EMG, they were required to keep their body relaxed and avoid unnecessary swallowing movements. The experimental design complied with the “Declaration of Helsinki of the World Medical Congress” and was approved by the Ethics Committee of Zhengzhou University. Before the experiment started, the participants were informed of the experimental tasks and signed an informed consent form. After the test, the participants received the corresponding remuneration.

Fig. 5

Online test scenario diagram.

This experiment uses the visual stimulus interface shown in Fig. 2. Before the experiment, 12 participants performed an offline experiment to collect EEG signals and calculate the SVM classification model for motor intention recognition. The specific test process was as follows: the participant first put on the EEG cap and sat in front of the screen. They then performed the task setting module and started the rehabilitation training after the task setting was completed. After the training started, the display dimmed, and the participant had 2 s of preparation time. Then, the stroboscopic block appeared on the screen and began to flicker. The data analysis system intercepted the data 3 s after flashing for identification; for the steady-state visually evoked potentials, the data was first filtered by 5–45 Hz, and then identified by the CCA algorithm. After identifying the SSVEP feature, instructions were sent to the rehabilitation robot, and the rehabilitation robot performed the corresponding action tasks. For the motor intention recognition method based on MRCPs and mu and beta rhythms, we intercept the data 1.5 s after flashing for preprocessing, and then extract the features for classification and recognition. The result is indirect control of the movement of the rehabilitation robot. Then, the screen goes dark, and the cycle continues for the next trial. Each participant needed to execute 50 commands per round of the experiment and perform three rounds of the experiment.

3.2 Test results

The online test results of 12 participants are shown in Tables 2 –4. Among them, Table 2 is the control action execution accuracy of the upper limb rehabilitation robot based on SSVEP; the 12 participants achieved an average control accuracy of 99.3%. Table 3 is the MRCP-based motor intention detection accuracy rate; the 12 participants achieved an average recognition rate of 82.7%. Table 4 shows the mu and beta rhythm-based motor intention recognition method; the 12 participants achieved an average detection accuracy of 77.2%, but the recognition results of the three participants were particularly poor, that is, the three participants are BCI-illiterates (Vidaurre & Blankertz, 2010). The mu and beta rhythm-based motor intention detection method is not as stable as the MRCPs-based motor intention detection method, and there will be BCI-illiterates. Therefore, this study uses a combination of the MRCP-based motor intention detection results and the SSVEP recognition control accuracy method to evaluate the completion of training. Table 5 shows the evaluation results of the active and passive training degree of completion of the active and passive upper limb rehabilitation training system based on SSVEP and MRCPs. The 12 participants achieved an average completion degree of 0.91. In addition, the system had almost no delay when collecting EEG signals in real time.

Table 2
Execution accuracy of upper limb rehabilitation robots based on SSVEP

Participant First session Second session Third session Average accuracy

LB 100.0% 100.0% 100.0% 100.0%

WP 100.0% 100.0% 100.0% 100.0%

LXP 98.0% 100.0% 98.0% 98.7%

YYF 100.0% 98.0% 100.0% 99.3%

WYW 98.0% 100.0% 100.0% 99.3%

LJJ 100.0% 98.0% 100.0% 99.3%

TGD 98.0% 98.0% 100.0% 98.7%

WJT 100.0% 98.0% 100.0% 99.3%

HSM 100.0% 98.0% 100.0% 99.3%

ZLP 98.0% 100.0% 98.0% 98.7%

LRQ 98.0% 100.0% 100.0% 99.3%

YDR 100.0% 100.0% 98.0% 99.3%

Average accuracy 99.3%

Participant	First session	Second session	Third session	Average accuracy
LB	100.0%	100.0%	100.0%	100.0%
WP	100.0%	100.0%	100.0%	100.0%
LXP	98.0%	100.0%	98.0%	98.7%
YYF	100.0%	98.0%	100.0%	99.3%
WYW	98.0%	100.0%	100.0%	99.3%
LJJ	100.0%	98.0%	100.0%	99.3%
TGD	98.0%	98.0%	100.0%	98.7%
WJT	100.0%	98.0%	100.0%	99.3%
HSM	100.0%	98.0%	100.0%	99.3%
ZLP	98.0%	100.0%	98.0%	98.7%
LRQ	98.0%	100.0%	100.0%	99.3%
YDR	100.0%	100.0%	98.0%	99.3%
Average accuracy				99.3%

Table 3

MRCP-based motor intention detection accuracy

Participant	First time	Second time	Third time	Average accuracy
LB	76.0%	94.0%	88.0%	86.0%
WP	72.0%	70.0%	92.0%	78.0%
LXP	90.0%	76.0%	88.0%	84.7%
YYF	88.0%	80.0%	86.0%	84.7%
WYW	88.0%	90.0%	80.0%	86.0%
LJJ	74.0%	74.0%	82.0%	76.7%
TGD	92.0%	70.0%	82.0%	81.3%
WJT	82.0%	94.0%	84.0%	86.7%
HSM	84.0%	80.0%	88.0%	84.0%
ZLP	86.0%	72.0%	82.0%	80.0%
LRQ	86.0%	90.0%	76.0%	84.0%
YDR	70.0%	92.0%	80.0%	80.7%
Average accuracy				82.7%

Table 4

Accuracy of participants’ motor intention detection based on mu and beta rhythms

Participant	First time	Second time	Third time	Average accuracy
LB	58.0%	62.0%	68.0%	62.7%
WP	84.0%	78.0%	86.0%	82.7%
LXP	92.0%	80.0%	84.0%	85.3%
YYF	58.0%	60.0%	56.0%	58.0%
WYW	86.0%	92.0%	80.0%	86.0%
LJJ	84.0%	78.0%	86.0%	82.7%
TGD	82.0%	80.0%	82.0%	81.3%
WJT	78.0%	88.0%	80.0%	82.0%
HSM	74.0%	82.0%	78.0%	78.0%
ZLP	88.0%	78.0%	86.0%	84.0%
LRQ	52.0%	60.0%	56.0%	56.0%
YDR	84.0%	88.0%	90.0%	87.3%
Average accuracy				77.2%

Table 5

Evaluation results of the degree of training completion of active and passive upper limb rehabilitation training systems based on SSVEP and MRCPs

Participant	First time	Second time	Third time	Average completion
LB	0.88	0.97	0.94	0.93
WP	0.86	0.85	0.96	0.89
LXP	0.94	0.88	0.93	0.917
YYF	0.94	0.89	0.93	0.92
WYW	0.93	0.95	0.90	0.927
LJJ	0.87	0.86	0.91	0.88
TGD	0.95	0.84	0.91	0.90
WJT	0.91	0.96	0.92	0.93
HSM	0.92	0.89	0.94	0.917
ZLP	0.92	0.86	0.90	0.893
LRQ	0.92	0.95	0.88	0.917
YDR	0.85	0.96	0.89	0.90
Average completion				0.91

4 Discussion

In this paper, the control method of the upper limb rehabilitation robot based on SSVEP signal was used, and the control accuracy rate was 99.3%. Compared with the commonly used motor imagery EEG signals in rehabilitation training (Khan, Das, Iversen, & Puthusserypady, 2020; Xu et al., 2020), the features of the SSVEP signal were more stable, and the recognition rate was also higher. Compared with the current SSVEP BCI technology, because this research uses only one command, the recognition rate is much higher than that reported in the relevant literature (Karunasena et al., 2021; Zhang, 2020). Given that the lower recognition rate will increase the frustration of the participants, which will lead to the resistance of the participants to the rehabilitation training, this study can avoid this problem by using the high recognition rate of SSVEP.

In addition, this article compares two kinds of motor intention recognition results based on MRCPs and mu and beta rhythms. The movement intention recognition method based on MRCPs achieved a higher average detection rate. The main reason for this result is that 3 out of 12 participants were BCI-illiterates, and it was difficult for these three participants to obtain a high recognition accuracy through the features of mu and beta rhythms. This result is consistent with our expectations, that is, the MRCPs motor intention recognition method has higher stability and can reduce the BCI-illiteracy phenomenon. Therefore, the participant completion evaluation method in this study adopted the MRCP-based motor intention recognition method.

Compared with the current upper extremity rehabilitation training system that only uses SSVEP control, the upper extremity rehabilitation training system proposed in this article requires the participants to complete passive SSVEP tasks and perform active movement or imagery tasks at the same time. The combination of active and passive styles can be used to improve the accuracy of the system’s motion control, guide the participants to actively participate in the training process, accelerate the reconstruction of motor neural circuits, and enhance the effects of training. In addition, the system can combine the action execution of the upper limb rehabilitation training system and the accuracy of the movement intention detection to calculate the patient’s level of completion evaluation index, which can evaluate the concentration of the participants in the training process and provide an objective index for the patient’s rehabilitation effects of training.

The experimental results prove that the system has high accuracy in analyzing the user’s EEG signals. The participants can control the upper limb rehabilitation training robot system based on the BCI according to their own rhythm and evaluate their own completion. The next steps for this research includes the following two aspects: 1) The preliminary test of the system has been completed, hence a large number of application tests are needed to continuously improve the function of the system and apply the system to the clinical rehabilitation training of patients. 2) Although the brain structure is similar in general, there are significant individual differences in EEG signals, and the difference between normal people and patients is even more significant, which will have a certain impact on the stability of the system. Follow-up measures are still needed to improve the robustness of the algorithm, making the system more stable.

5 Conclusion

This article first introduced the background and current state of upper limb rehabilitation technology for patients with craniocerebral injury. Aimed at the current problems of rehabilitation technology, an active and passive upper limb rehabilitation training system based on a hybrid BCI was proposed. The system combines the stability of the SSVEP signal as a driving signal and the advantage of no BCI illiteracy when the MRCPs signal is used to detect the participant’s active movement intention. Through the combination of active and passive control methods, the control stability of the system was improved, and the participants were guided to actively participate in rehabilitation training to accelerate the reconstruction of motor neural circuits and enhance the effects of training.

Footnotes

Author Biographies

Tongda Shen is studying for a Bachelor’s degree in the Tandon School of Engineering, New York University. From July 2018 to September 2021, he was a research intern at the Henan Key Laboratory of Brain Science and Brain-Computer Interface Technology.

Lipeng Zhang is studying for a PhD in the School of Electrical Engineering, Zhengzhou University. In 2013, he received the B.S degree in Automation major from the Henan University of Urban Construction. In 2016, he received the M.S degree in Navigation, guidance and control major from the Zhengzhou University. From 2016 to 2018, he was a Research Assistant in the Department of Automation, Tsinghua University. Since 2018, he is in Zhengzhou University pursuing his Ph.D degree. He is the author of three articles. His research interests in Brain Computer Interface and the mechanism of people’s brain during motor.

Shaoting Yan is studying for a PhD in the School of Electrical Engineering, Zhengzhou University, and he is also a member of Henan Key Laboratory of Brain Science and Brain Computer Interface Technology. In 2018, he received a bachelor’s degree in automation from Henan Polytechnic University. In 2021, he obtained a master’s degree in control engineering from Qingdao University of Technology. His research focuses on Brain Computer Interface.

Yuxia Hu is Associate Professor at Zhengzhou University, China. She received her PhD from Zhengzhou University for research on Control Science and Engineering. Her current research interests are Brain-computer interface technology, neural engineering and Cognitive Neuroscience.

References

Betta,

, Gemignani,

, Landi,

, Laurino,

, Piaggi,

, & Menicucci,

(2013). Detection and removal of ocular artifacts from EEG signals for an automated REM sleep analysis. Annu Int Conf IEEE Eng Med Biol Soc, 2013, 5079–5082.

Bhagat,

N.A.

, Venkatakrishnan,

, Abibullaev,

, Artz,

E.J.

, Yozbatiran,

, Blank,

A.A.

, ... Contreras-Vidal,

J.L.

(2016). Design and Optimization of an EEG-Based Brain Machine Interface (BMI) to an Upper-Limb Exoskeleton for Stroke Survivors. Front Neurosci, 10, 122.

Buch,

, Weber,

, Cohen,

L.G.

, Braun,

, Dimyan,

M.A.

, Ard,

, ... Birbaumer,

(2008). Think to Move: a Neuromagnetic Brain-Computer Interface (BCI) System for Chronic Stroke. Stroke, 39(3), 910–917.

Cantillo-Negrete,

, Carino-Escobar,

R.I.

, Carrillo-Mora,

, Rodriguez-Barragan,

M.A.

, Hernandez-Arenas,

, Quinzaños-Fresnedo,

, ... Arias-Carrion,

(2021). Brain-Computer Interface Coupled to a Robotic Hand Orthosis for Stroke Patients’ Neurorehabilitation: A Crossover Feasibility Study. Frontiers in Human Neuroscience, 15, 656975.

Chaudhary,

, Birbaumer,

, & Ramos-Murguialday,

(2016). Brain–computer interfaces for communication and rehabilitation. Nature Reviews Neurology, 12(9), 513–525.

Chen,

, Chen,

, Gao,

, & Gao,

(2013). Brain-computer interface based on intermodulation frequency. J Neural Eng, 10(6), 066009.

Chen,

, Wang,

, Zhang,

, & Gao,

(2018). Enhancing Detection of SSVEPs with Intermodulation Frequencies Using Individual Calibration Data. Conf Proc IEEE Eng Med Biol Soc, 2018, 2531-2534. Retrieved from https://www.ncbi.nlm.nih.gov/pubmed/30440923. doi:10.1109/EMBC.2018.8512783

Cheng,

, Phua,

K.S.

, Lai,

H.S.

, Tam,

P.K.

, Tang,

K.Y.

, Cheng,

K.K.

, ... Lim,

J.H.

(2020). Brain-Computer Interface-Based Soft Robotic Glove Rehabilitation for Stroke. IEEE Transactions on Biomedical Engineering, 67(12), 3339–3351.

Chew,

, Teo,

W.-P.

, Tang,

, Ang,

K.K.

, Ng,

Y.S.

, Zhou,

J.H.

, ... Guan,

(2020). Using Transcranial Direct Current Stimulation to Augment the Effect of Motor Imagery-Assisted Brain-Computer Interface Training in Chronic Stroke Patients—Cortical Reorganization Considerations. Frontiers in Neurology, 11, 948.

10.

Choi,

, Kwon,

G.H.

, Lee,

, & Nam,

C.S.

(2020). Functional Electrical Stimulation Controlled by Motor Imagery Brain-Computer Interface for Rehabilitation. Brain Sciences, 10(8), 512.

11.

Cirstea,

M.C.

, & Levin,

M.F.

(2007). Improvement of arm movement patterns and endpoint control depends on type of feedback during practice in stroke survivors. Neurorehabil Neural Repair, 21(5), 398–411.

12.

Cramer,

S.C.

, Sur,

, Dobkin,

B.H.

, O’Brien,

, Sanger,

T.D.

, Trojanowski,

J.Q.

, ... Vinogradov,

(2011). Harnessing neuroplasticity for clinical applications. Brain, 134(Pt 6), 1591–1609. Retrieved from https://www.ncbi.nlm.nih.gov/pubmed/21482550. doi:10.1093/brain/awr039

13.

Di Russo,

, Berchicci,

, Bozzacchi,

, Perri,

R.L.

, Pitzalis,

, & Spinelli,

(2017). Beyond the “Bereitschaftspotential”: Action preparation behind cognitive functions. Neurosci Biobehav Rev, 78, 57–81. Retrieved from https://www.ncbi.nlm.nih.gov/pubmed/28445742. doi:10.1016/j.neubiorev.2017.04.019

14.

Edelman,

B.J.

, Baxter,

, & He,

(2016). EEG Source Imaging Enhances the Decoding of Complex Right-Hand Motor Imagery Tasks. IEEE Trans Biomed Eng, 63(1), 4–14. Retrieved from https://www.ncbi.nlm.nih.gov/pubmed/26276986. doi:10.1109/TBME.2015.2467312

15.

Giesche,

, Vieluf,

, Wilke,

, Engeroff,

, Niederer,

, & Banzer,

(2021). Cortical Motor Planning and Biomechanical Stability During Unplanned Jump-Landings in Males With ACL-Reconstruction. J Athl Train.

16.

Gui,

, Ren,

, & Zhang,

(2015). Online brain-computer interface controlling robotic exoskeleton for gait rehabilitation, 50 Nanyang Avenue, Singapore, Singapore.

17.

Han,

, Xu,

, Wang,

, Tang,

, Liu,

, An,

, ... Ming,

(2020). Write’ but not ‘spell’ Chinese characters with a BCI-controlled robot. Annu Int Conf IEEE Eng Med Biol Soc, 2020, 4741–4744.

18.

Hashimoto,

, Kakui,

, Ushiba,

, Liu,

, Kamada,

, & Ota,

(2021). Portable rehabilitation system with brain-computer interface for inpatients with acute and subacute stroke: A feasibility study. Assistive Technology, 1-9.

19.

Hesse,

, Mehrholz,

, & Werner,

(2008). Robot-Assisted Upper and Lower Limb Rehabilitation After Stroke. Deutsches Aerzteblatt Online, 105(18), 330–336.

20.

Horki,

, Neuper,

, Pfurtscheller,

, & Muller-Putz,

(2010). Asynchronous steady-state visual evoked potential based BCI control of a -DoF artificial upper limb. Biomed Tech (Berl), 55(6), 367–374.

21.

Islam,

M.R.

, Molla,

M.K.I.

, Nakanishi,

, & Tanaka,

(2017). Unsupervised frequency-recognition method of SSVEPs using a filter bank implementation of binary subband CCA. Journal of Neural Engineering, 14(2), Retrieved from <Go to ISI>://WOS:000412758600001.

22.

Johnson,

S.H.

, Sprehn,

, & Saykin,

A.J.

(2002). Intact Motor Imagery in Chronic Upper Limb Hemiplegics: Evidence for Activity-Independent Action Representations. Journal of Cognitive Neuroscience, 14(6), 841–852.

23.

Kakuda,

(2020). Future directions of stroke rehabilitation. Rinsho Shinkeigaku, 60(3), 181–186.

24.

Karunasena,

S.P.

, Ariyarathna,

D.C.

, Ranaweera,

, Wijayakulasooriya,

, Kim,

, & Dassanayake,

(2021). Single-Channel EEG SSVEP-based BCI for Robot Arm Control.

25.

Katan,

, & Luft,

(2018). Global Burden of Stroke. Seminars in Neurology, 38(02), 208–211.

26.

Khan,

M.A.

, Das,

, Iversen,

H.K.

, & Puthusserypady,

(2020). Review on motor imagery based BCI systems for upper limb post-stroke neurorehabilitation: From designing to application. Computers in Biology and Medicine, 123, 103843.

27.

Kwakkel,

, Kollen,

B.J.

, & Krebs,

H.I.

(2008). Effects of Robot-Assisted Therapy on Upper Limb Recovery After Stroke: A Systematic Review. Neurorehabilitation and Neural Repair, 22(2), 111–121.

28.

Langhorne,

, Coupar,

, & Pollock,

(2009). Motor recovery after stroke: a systematic review. The Lancet Neurology, 8(8), 741–754.

29.

Lawrence,

E.S.

, Coshall,

, Dundas,

, Stewart,

, Rudd,

A.G.

, Howard,

, & Wolfe,

C.D.

(2001). Estimates of the prevalence of acute stroke impairments and disability in a multiethnic population. Stroke, 32(6), 1279–1284.

30.

Lazarou,

, Nikolopoulos,

, Petrantonakis,

P.C.

, Kompatsiaris,

, & Tsolaki,

(2018). EEG-Based Brain–Computer Interfaces for Communication and Rehabilitation of People with Motor Impairment: A Novel Approach of the st Century. Frontiers in Human Neuroscience, 12, 14.

31.

Lim,

, & Ku,

(2018). A Brain–Computer Interface-Based Action Observation Game That Enhances Mu Suppression. IEEE Transactions on Neural Systems and Rehabilitation Engineering, PP(12), 1–1.

32.

Lin,

, Zhang,

, Wu,

, & Gao,

(2006). Frequency recognition based on canonical correlation analysis for SSVEP-based BCIs. IEEE Transactions on Biomedical Engineering, 53(12), 2610–2614.

33.

Machado,

T.C.

, Carregosa,

A.A.

, Santos,

M.S.

, Ribeiro,

, & Melo,

(2019). Efficacy of motor imagery additional to motor-based therapy in the recovery of motor function of the upper limb in post-stroke individuals: a systematic review. Top Stroke Rehabil, 26(7), 548–553.

34.

McGeady,

, Vuckovic,

, & Puthusserypady,

(2019). A Hybrid MI-SSVEP based Brain Computer Interface for Potential Upper Limb Neurorehabilitation: A Pilot Study, Gangwon, Korea, Republic of.

35.

Molinari,

, & Masciullo,

(2020). Stroke and potential benefits of brain-computer interface. Handb Clin Neurol, 168, 25–32.

36.

Monge-Pereira,

, Ibañez-Pereda,

, Alguacil-Diego,

I.M.

, Serrano,

J.I.

, Spottorno-Rubio,

M.P.

, & Molina-Rueda,

(2017). Use of Electroencephalography Brain-Computer Interface Systems as a Rehabilitative Approach for Upper Limb Function After a Stroke: A Systematic Review. PM&R, 9(9), 918–932.

37.

Müller-Gerking,

, Pfurtscheller,

, & Flyvbjerg,

(1999). Designing optimal spatial filters for single-trial EEG classification in a movement task. Clinical Neurophysiology, 110(5), 787–798.

38.

Niazi,

I.K.

, Jiang,

, Tiberghien,

, Nielsen,

J.F.

, Dremstrup,

, & Farina,

(2011). Detection of movement intention from single-trial movement-related cortical potentials. J Neural Eng, 8(6), 066009.

39.

Nordin,

, Xie,

, & Wünsche,

(2014). Assessment of movement quality in robot-assisted upper limb rehabilitation after stroke: a review. Journal of Neuro Engineering and Rehabilitation, 11(1), 137.

40.

Nudo,

R.J.

(2006). Mechanisms for recovery of motor function following cortical damage. Current Opinion in Neurobiology, 16(6), 638–644.

41.

Parker,

, Mountain,

, & Hammerton,

(2011). A review of the evidence underpinning the use of visual and auditory feedback for computer technology in post-stroke upper-limb rehabilitation. Disabil Rehabil Assist Technol, 6(6), 465–472.

42.

Riaz,

, Hassan,

, Rehman,

, Niazi,

, Jochumsen,

, & Dremstrup,

(2014). Processing movement related cortical potentials in EEG signals for identification of slow and fast movements. Annu Int Conf IEEE Eng Med Biol Soc, 2014, 4908–4911.

43.

Rong,

, Ding,

, Xiong,

, Zhang,

, Wang,

, Deng,

, ... Jia,

(2021). Mirror Visual Feedback Prior to Robot-Assisted Training Facilitates Rehabilitation After Stroke: A Randomized Controlled Study. Frontiers in Neurology, 12, 683703. Retrieved from https://www.ncbi.nlm.nih.gov/pubmed/34305792. doi:10.3389/fneur.2021.683703

44.

Sebastián-Romagosa,

, Cho,

, Ortner,

, Murovec,

, Von Oertzen,

, Kamada,

, ... Guger,

(2020). Brain Computer Interface Treatment for Motor Rehabilitation of Upper Extremity of Stroke Patients—A Feasibility Study. Frontiers in Neuroscience, 14, 591435.

45.

Sharma,

, Pomeroy,

V.M.

, & Baron,

J.-C.

(2006). Motor Imagery. Stroke, 37(7), 1941–1952.

46.

Silvoni,

, Ramos-Murguialday,

, Cavinato,

, Volpato,

, Cisotto,

, Turolla,

, ... Birbaumer,

(2011). Brain-Computer Interface in Stroke: A Review of Progress. Clinical EEG and Neuroscience, 42(4), 245–252.

47.

Simon,

, Bolton,

, Kennedy,

N.C.

, Soekadar,

S.R.

, & Ruddy,

K.L.

(2021). Challenges and Opportunities for the Future of Brain-Computer Interface in Neurorehabilitation. Front Neurosci, 15, 699428.

48.

Stinear,

(2010). Prediction of recovery of motor function after stroke. The Lancet Neurology, 9(12), 1228–1232.

49.

Stinear,

C.M.

(2017). Prediction of motor recovery after stroke: advances in biomarkers. The Lancet Neurology, 16(10), 826–836.

50.

Vidaurre,

, & Blankertz,

(2010). Towards a cure for BCI illiteracy. Brain topography, 23(2), 194–198.

51.

Wolpaw,

J.R.

(2007). Brain-computer interfaces as new brain output pathways. The Journal of Physiology, 579(3), 613–619.

52.

Wolpaw,

J.R.

(2012). Harnessing neuroplasticity for clinical applications. e; author reply . Retrieved from. Brain, 135(Pt 4), e216. Retrieved from https://www.ncbi.nlm.nih.gov/pubmed/22374936. doi:10.1093/brain/aws017

53.

Wolpaw,

J.R.

, Birbaumer,

, McFarland,

D.J.

, Pfurtscheller,

, & Vaughan,

T.M.

(2002). Brain-computer interfaces for communication and control. Clin Neurophysiol, 113(6), 767–791.

54.

Xu,

, Li,

, He,

, Wei,

, Zhang,

, Wu,

, & Song,

(2020). Motor imagery based continuous teleoperation robot control with tactile feedback. Electronics, 9(1), 174.

55.

Yang,

, Liu,

, Hu,

, Qiao,

, Wang,

, Yang,

, ... Lu,

(2019). Research progress on the prevalence and influencing factors of stroke at home and abroad. Chinese Journal of Preventive Medicine, v.20(02), 65–70.

56.

Zhang,

, Dong,

, & Zhu,

(2020). Brain-controlled wheelchair system based on SSVEP.

57.

Zhang,

, Wang,

, Zhang,

, Chen,

, Shi,

, Gao,

, & Hu,

(2020). The Influence of Different EEG References on Scalp EEG Functional Network Analysis During Hand Movement Tasks. Frontiers in Human Neuroscience, 14, 367.

58.

Zhang,

Y.U.

, Zhou,

, Jin,

, Wang,

, & Cichocki,

(2014). Frequency recognition in SSVEP-based BCI using multiset canonical correlation analysis. International Journal of Neural Systems, 24(04), 1450013.

59.

Zhao,

, Zhang,

, Qiu,

, Wang,

, Xiang,

, Wang,

, ... Wu,

(2018). Scalp acupuncture plus low-frequency rTMS promotes repair of brain white matter tracts in stroke patients: A DTI study. Journal of Integrative Neuroscience, 125-139.

An active and passive upper limb rehabilitation training system based on a hybrid brain–computer interface

Abstract

Keywords

1 Introduction

2 System design and algorithm

2.1 System design

2.2.1 Task setting and training guidance module

2.2.5 Communication module

2.2.6 Training completion evaluation module

3.1 Online test

5 Conclusion

Footnotes

Author Biographies

References