Implementation of an SSVEP-based intelligent home service robot system

Abstract

BACKGROUND:

People with severe neuromuscular disorders caused by an accident or congenital disease cannot normally interact with the physical environment. The intelligent robot technology offers the possibility to solve this problem. However, the robot can hardly carry out the task without understanding the subject’s intention as it relays on speech or gestures. Brain-computer interface (BCI), a communication system that operates external devices by directly converting brain activity into digital signals, provides a solution for this.

OBJECTIVE:

In this study, a noninvasive BCI-based humanoid robotic system was designed and implemented for home service.

METHODS:

A humanoid robot that is equipped with multi-sensors navigates to the object placement area under the guidance of a specific symbol “Naomark”, which has a unique ID, and then sends the information of the scanned object back to the user interface. Based on this information, the subject gives commands to the robot to grab the wanted object and give it to the subject. To identify the subject’s intention, the channel projection-based canonical correlation analysis (CP-CCA) method was utilized for the steady state visual evoked potential-based BCI system.

RESULTS:

The offline results showed that the average classification accuracy of all subjects reached 90%, and the online task completion rate was over 95%.

CONCLUSION:

Users can complete the grab task with minimum commands, avoiding the control burden caused by complex commands. This would provide a useful assistance means for people with severe motor impairment in their daily life.

Keywords

Brain-computer interface Steady-state visual evoked potential Humanoid robot Home service Channel projection-based canonical correlation analysis (CP-CCA)

1. Introduction

Spinal cord injury caused by motor neuron disease (MND), amyotrophic lateral sclerosis (ALS), or accidents leads to a reduction in the quality of life for many people. It is difficult for these people to perform some basic daily tasks, such as grabbing, lifting objects and walking. Many studies attempted to create a high-tech assistive device to enhance their quality of their life [1, 2, 3]. In the past few decades, numerous attempts had been made to design and manufacture full-bodied humanoid robots. The development of mechanics, electronics and computer science technology has promoted the development of humanoid robots, such as ASIMO, HUBO and HOAP-2 [4, 5, 6]. Advances in robotics allow individuals with disabilities to use robots to perform daily tasks more independently [7, 8]. Brain–computer interface (BCI) is a technology that can be used to help these people perform certain daily tasks with the help of a robot, such as reaching and grabbing objects.

Vidal proposed the concept of BCI in 1973 [9]. BCI is an advanced communication and control system that operates external devices by directly converting brain activity into digital signals. Therefore, BCI can enable disabled individuals to communicate with other people or control their surroundings without using any muscle activities [10, 11]. This system obtains neural responses from the human brain invasively or non-invasively, and explains human intentions by dividing the neural responses into several mental states [12]. This mind-reading technique can convey human intentions as commands to the machine. Several studies have successfully proven that the invasive BCI technique can be used to control a robotic arm to perform a series of actions [13, 14]. These invasive methods can obtain signals with high signal-to-noise ratio (SNR), which can control the peripheral devices better. However, this technology is invasive and requires electrodes to be surgically implanted, and users face the risk of post-operative complications and infections that may cause serious harm. Furthermore, long-term stability of recorded signals may be another issue that should be addressed. Therefore, non-invasive BCI methods are more suitable for humans because they can avoid health risks and related ethical issues, and the method is easy to apply and harmless to patients [15]. Different methods such as sensorimotor rhythm, moving image (MI), P300 potential and steady state visual evoked potential (SSVEP) are used in BCI technology [16, 17]. Compared to other BCI methods, SSVEP has the advantages of high SNR, higher accuracy, higher information transmission rate (ITR), and shorter training time [18, 19, 20], so it is more suitable to control peripheral efficiently.

SSVEP is the brain’s periodic response to periodic visual stimuli modulated at frequencies above 6 Hz [21]. When the subject focuses on the visual stimulation, the visual pathways will be affected and the frequency of the visual stimulation will induce the subject’s brain and generate a signal with the same frequency as the stimulation frequency or its harmonic. Multiple light sources can be used to provide visual stimulation in SSVEP applications, such as light emitting diode (LED), cathode ray tube (CRT) monitors, or liquid crystal display (LCD) monitors. Most studies verified that the strongest SSVEP response can be observed in the visual cortex [22, 23].

With the development of robotics and neural engineering, a BCI control system for robots based on EEG has been proposed, so some elderly or disabled people can control the robot naturally and intuitively merely by thinking when using the system. The ultimate goal of this BCI-based robotic control system is to generate and transmit stable, sophisticated, or even emotional, intention into robots and let them perform various complex tasks according to human intentions. BCI-based robotic control systems using EEG have been applied to mobile robots [24], manipulators [25], wheelchairs [26, 27], and humanoid robots [28]. These previous studies have effectively proven the possibility of EEG-based BCI systems for robot control.

For practical human–robot interaction applications, proposed brain-controlled robot system using EEG-based BCI employed different types of electrophysiological brain signals, such as SSVEP and sensorimotor rhythms. According to the properties of brain signals, the system can be categorized as either a reactive BCI or an active BCI [29]. The reactive BCI enables users to control equipment by detecting indirectly modulated brain signals related to specific external stimuli. SSVEP is a reactive signal and is commonly used in BCI-based robotics applications. These signals are generated when the target object visually stimulates the brain in some methods such as a sudden flash of light [30].

Figure 1.

Schematic of system.

One of the main goals of EEG-based BCIs for human-robot interaction is to be able to directly control a robot by low-frequency visual stimulation without thinking. Therefore, our study adopts a reactive and no training BCI approach to control a new brain-actuated humanoid robot system for home service. The robot automatically navigates to the designated placement area to grab the required object and send it back to the user. With the help of machine intelligence of multi-sensors fusion, it can avoid collisions between robot and obstacles on the ground during navigation. A channel projection-based canonical correlation analysis (CP-CCA) target recognition method is adopted for signal analysis to send commands [31]. Through user interface visual feedback, when the robot recognizes the wrong target, subjects can cancel the wrong command and resend the required command. Furthermore, to further increase the practicality of the proposed system, a portable and low-cost wireless EEG device is utilized to measure SSVEP signals.

The contribution of this work is to develop an efficient and feasible brain-robot system to help users grab the needed object remotely in an indoor complex environment without moving their body, thereby improving the independence and quality of their daily life.

2. Materials and methods

2.1 Subjects

Ten healthy subjects (7 males and 3 females) participated in the offline and online experiments, respectively. All 10 subjects ranged in age from 21 to 26 (average age 24). These fully BCI-naive subjects had normal or corrected vision. All subjects provided written informed consent and were clearly instructed about the purpose of the experiment and possible results. Subjects received a small monetary compensation for their participation.

2.2 System description

Figure 1 shows the overall control architecture of the proposed SSVEP BCI robot control smart home system. The wirelessly transmitted raw EEG data are recorded by the headset with dry electrodes and then transmitted to the PC for preprocessing to increase the SNR. For target recognition, the EEG analysis algorithm performs feature extraction and classification on signals. Finally, the control commands are generated by the computer according to the classification results. The humanoid will conduct the control commands to move and grab the target object. In order to get real-time feedback on the status of the robot, the data (i.e., the visual images from the humanoid monocular camera) are transmitted using wireless TCP/IP communication protocol between the humanoid and other systems.

After receiving the start command (teeth clenching), the robot Nao will automatically navigate to the object placement area with the help of Naomark. During navigation, the multi-sensor detects obstacles and prevents the robot from colliding with objects. Naomark is a special landmark that can identify location and is described in more detail in Section 2.4. When the robot reaches the object placement area, it will scan the surrounding objects from left to right. The object is placed in a specific box that the humanoid robot can grab and pasted a Naomark on its outside for rapid recognition by the robot. The different IDs of the Naomark represent each of the different objects. The robot sends the scanned ID number to the user interface and converts it into a corresponding object, for example ID “84” stands for an apple. As shown in Fig. 2a, the subject can choose the object of interest. When an object is selected, the user interface will switch to the second layer (see Fig. 2b). The selected object will appear in the center of the interface and will stay there for 4 seconds. The countdown time will appear in the upper right corner of the screen. If the selected object is wrong, the subject can send the “teeth clenching” signal to cancel the command and return to the first layer interface.

Figure 2.

The user interface of the BCI system. (a) First layer of the interface, (b) Second layer of the interface.

The stimulus frequencies normally used in the SSVEP can be divided into three frequency bands: low (1–12 Hz), medium (12–30 Hz), and high (30–60 Hz). A study shows that the peak of SSVEP amplitude appears near 15 Hz in the 5–25 Hz range and has a high signal-to-noise ratio [21]. Therefore, in this study, four frequencies (i.e., 7.5, 8.57, 10, and 12 Hz) in the lower range are selected as stimulus frequencies, thereby covering the alpha frequency band.

A 21.5-inch liquid-crystal display (LCD) with a resolution of 1920 $\times$ 1080 pixels and a 60 Hz-refresh rate monitor is employed as the visual stimulator. Each stimulus is toggled between black and white. After the command is selected, the color of the stimulus flicker will change from white to green as a visual feedback to the user. The stimulation application is developed in MATLAB using the Psychophysics Toolbox [32].

2.3 Obstacle avoidance based on multi-sensor fusion

The chest position of the robot is equipped with two ultrasonic sensors (sonar) which can estimate the distance of obstacles in the surrounding environment. The detection range of the sonar is 0.20 m–0.80 m. Two contact sensors (bumper) are located at the tip of each foot. The bumper sensor is used to detect some obstacles about 100 mm high from the ground, and the robot will automatically step backward to avoid the obstacle on the ground if one of the bumper sensors is triggered. As for some obstacles over 100 mm high from the ground, once the sonar value reaches the set threshold (0.4 m), the robot will immediately stop walking.

In the process of navigation, if the sensor detects an obstacle, the robot will use the sonar sensor to measure the distance between the left and right sides of the robot. When the distance on one side is greater than the distance on the other side, the robot will bypass the obstacle from the side with the greater distance and continue to move forward. If the bumper is triggered, the robot will return to the previous position and bypass the other side. Figure 3 shows how the robot avoids the obstacle on the ground. Both categories of sensors guarantee the safety of Nao by avoiding the collisions between Nao and obstacles while improving the system executive efficiency.

Figure 3.

Schematic of how the robot avoids the obstacle.

2.4 Target location based on Naomark

In order to grab an object, the robot needs to calculate the walking distance to navigate to the object. A special landmark called Naomark [33] can detect the location, which was used as a range finder in the experiment. An example of Naomark is shown in Fig. 4. The sector rotates around the center of the circle and the angles formed by different sectors distinguish different Naomarks with different IDs. It contains a lot of information that the robot can identify, among which SizeX and ID are needed in our study. SizeX is the target landmark image pixel size in the Nao robot’s vision in radians (rad), and MarkId is the unique number of the landmark.

Figure 4.

Naomark.

The Nao camera defaults to the 640*480 resolution, with 640 pixels in horizontal direction and 480 pixels in the vertical direction. The equation for sizeX is:

$\displaystyle\text{sizeX}=\frac{\text{pixel}}{640}\times\text{HOV}\times\frac{% \pi}{180}$ (1)

where pixel is the diameter size of the landmark imaging pixel, HOV represents the camera’s horizontal angle of view, HOV $=$ 60.97 ${}^{\circ}$ . The equation results show that the target has a certain relationship between the distance of the target and the camera. Liu et al. [34] use machine learning to determine the relationship used to locate the robot Nao.

Through machine learning, the model of the relationship between sizeX and the distance within 1m is described by Eq. (2). In the distance between 1m–2m, the relationship between sizeX and distance is described by Eq. (3):

$\displaystyle f(x)=3.556-51.111x+389.47x^{2}-1707x^{3}+4314.3x^{4}-5837.9x^{5}% +3272.1x^{6}$ (2) $\displaystyle f(x)=0.0689x^{-1.168}$ (3)

where $x$ is sizeX, $f(x)$ represents distance, it can be seen that the distance can be calculated as long as the image size sizeX of the target is input.

2.5 Robotic kinematics modeling

In this paper, the double-arm grabbing operation was implemented the robot Nao. Since the left arm and right arm of Nao are symmetrical, the analysis of the kinematics is performed using the left arm as an example. As shown in Fig. 5, the left arm of the robot consists of three parts (upper, lower arm and three fingers), which are connected by five joints.

Since the fingers have no relationship with the joint movement of the left arm of the robot, we ignored the degrees of freedom on the finger part and only constructed the model with five degrees of freedom for one arm. According to the Denavit-Hartenberg (D-H) method [35], a kinematics model for the left arm of the Nao robot is built. The D-H parameter table is obtained and shown in Table 1. In this Table 1, $\theta$ is the angle about the $Z_{n}$ -axes describing the rotation of the $X_{n-1}$ to ${X}_{n}$ , d is the distance between the $X_{n-1}$ and the ${X}_{n}$ -axes in the direction of the $Z_{n}$ -axes, a represents the distance between the $Z_{n-1}$ -axes and the next $Z_{n}$ -axes along the $X_{n-1}$ -axes, and $\alpha$ represents the angle about $X_{n-1}$ -axis from $Z_{n-1}$ to $Z_{n}$ -axes.

To obtain the positive kinematics of the Nao manipulator, the D-H method is needed to determine the link parameters. The D-H parameters of the left arm of the robot are shown in Table 1. Equation (4) is the transformation matrix of the adjacent two-link coordinate system.

$\displaystyle K^{i-1}_{i}=\left[\begin{array}[]{cccc}{\cos{\theta}_{i}}&-{\sin% {\theta}_{i}}{\cos{\alpha}_{i}}&{\sin{\theta}_{i}}{\sin{\alpha}_{i}}&a_{i}{% \cos{\theta}_{i}}\\ {\sin{\theta}_{i}}&{\cos{\theta}_{i}}{\cos{\alpha}_{i}}&-{\cos{\theta}_{i}}{% \sin{\alpha}_{i}}&a_{i}{\sin{\theta}_{i}}\\ 0&{\sin{\alpha}_{i}}&{\cos{\alpha}_{i}}&d_{i}\\ 0&0&0&0\end{array}\right]$ (4)

where $K$ is the coordinate transformation matrix, and $i$ stands for the $i$ -th joint. The posture of the first joint relative to the end-effector of the Nao robot can be obtained by Eq. (5):

$\displaystyle K^{0}_{5}=K^{0}_{1}K^{1}_{2}K^{2}_{3}K^{3}_{4}K^{4}_{5}$ (5)

Table 1

D-H parameters of the left arm

Joint $i$	$d_{i}$ /mm	${\alpha}_{i}$ /rad
1	0	${\pi}/{2}$
2	0	$-{\pi}/{2}$
3	105	$-{\pi}/{2}$
4	0	${\pi}/{2}$
5	55.95	$-{\pi}/{2}$

Figure 5.

The structure of the left arm [33].

The inverse kinematics solution of the robot is the inverse solution of forward kinematics. Inverse kinematics solution means that the required pose of the robot end-effector on the reference coordinate system is known, and then the joint motion parameters of the robot need to be found. Defining the desired pose of the robot Nao’s end-effector as:

$\displaystyle K^{0}_{5}=\left[\begin{array}[]{cccc}n_{x}&o_{x}&a_{x}&p_{x}\\ n_{y}&o_{y}&a_{y}&p_{y}\\ n_{z}&o_{z}&a_{z}&p_{z}\\ 0&0&0&1\end{array}\right]$ (6)

When Eqs (5) and (6) are equal, the five joints’ variable angles of robot Nao’s left arm are obtained [36].

$\displaystyle\left\{\begin{array}[]{l}{\theta}_{1}={\tan^{-1}{\displaystyle% \frac{p_{y}+0.05595o_{y}}{p_{x}+0.05595o_{x}}}}\\ {\theta}_{2}={\tan^{-1}{\displaystyle\frac{-c_{1}(p_{x}+0.05595o_{x})-s_{1}(p_% {y}+0.05595o_{y})}{p_{z}+0.05595o_{z}}}}\\ {\theta}_{3}={\tan^{-1}{\displaystyle\frac{c_{1}o_{y}-s_{1}o_{x}}{s_{2}o_{z}+c% _{1}c_{2}o_{x}+c_{2}s_{1}o_{y}}}}\\ {\theta}_{4}={\tan^{-1}{\displaystyle\frac{c_{1}o_{y}-s_{1}o_{x}}{s_{2}s_{3}o_% {z}-c_{1}s_{2}s_{3}o_{x}-s_{1}s_{2}s_{3}o_{y}}}}\\ {\theta}_{5}={\tan^{-1}{\displaystyle\frac{c_{1}s_{2}a_{x}+s_{1}s_{2}a_{y}-c_{% 2}a_{z}}{c_{2}n_{z}-c_{1}s_{2}n_{x}-s_{1}s_{2}n_{y}}}}\end{array}\right.$ (7)

where ${\theta}_{1}$ is the LShoulderPitch, ${\theta}_{2}$ is the LShoulderRoll, ${\theta}_{3}$ is the LElbowYaw, ${\theta}_{4}$ is the LElbowRoll, and ${\theta}_{5}$ is the LWristYaw.

2.6 EEG data acquisition

Emotiv EPOC headset combined with cost-effective and portable features will be used to collect EEG signals, as shown in Fig. 6. Compared with wet sensors, the dry sensor has some advantages, including no need to inject conductive gels or glues needed during operation, easy to attach to the brain scalp through the hair, and can be reused many times. For brain activity recording, according to the 10–20 international system, 14 channels are placed on the standard positions. Moreover, CMS/DRL reference positions are also employed, which are located behind the ear of the subject. According to our previous research [31], O1, O2, P7 and P8 channels belonging to the occipital region will be used. In each EEG channel, the sampling frequency is down-sampled from a 2048 Hz to 128 Hz. The subject’s EEG signal is filtered using fourth order Butterworth band pass filter with fL (equals to 7 Hz) and fH (equals to 49 Hz).

Figure 6.

EEG acquisition device. (a) Emotiv EPOC, (b) electrode position according to 10-20 EEG placement.

2.7 Offline BCI experiment

In the BCI experiment based on SSVEP, the subject comfortably sat on a chair, and the display was placed 60 cm in front of the subject. For the offline experiment, the subjects performed a simulated online experiment to record EEG data for offline analysis and the humanoid robot remained stationary. Subjects stared at one of the four stimulation targets indicated in a random order by computer prompts. Each subject completed 10 runs, and each run was composed of eight trials. After five runs, subjects were asked to rest for two minutes to reduce eye fatigue. Each trial lasted 5 s and consisted of two parts: a 1 s cue phase and a stimulation phase of 4 s. Figure 7 shows the timing scheme of the entire procedure. The subject was required to avoid blinking and eye movement during the stimulation process for less eye artifacts. The first six runs are used to train the data and optimize the parameters of each subject. The six-fold cross-validation was utilized to evaluate the precision of SSVEP recognition for one subject. The last four runs of data are used to obtain the average classification accuracy and ITR of the offline test.

Figure 7.

The timing of the entire procedure.

Figure 8.

(a) The real experimental environment, (b) the plan of the experimental environment.

2.8 Online BCI experiment

An experimental area of 800 cm $\times$ 650 cm was used to conduct the online experiment. The experimental area consisted of four Naomark boxes loaded with different object, and the boxes were placed in the complex background, as shown in Fig. 8a. Figure 8b shows a plan of the experimental environment to describe the area clearly. The subject was asked to perform a move-grab-back task four times by controlling the humanoid by means of the proposed SSVEP BCI without visual cues. Before the robot control session, the subjects were given 15 minutes to practice controlling the robot using the SSVEP-based BCI and manual interface. In order to reduce the burden of manual control, in manual interface, the robot can continuously walk and automatically grab objects. After practice, the subjects went through the real-time control experiment as follows. The muscle artifact (EMG) of “teeth clenching” condition recorded from the EEG signal was used to start the robot and cancel the wrong command. In our previous study [10], the accuracy of detecting the “teeth clenching” state was higher than other states in EEG-based BCI system, and this EMG can improve the performance of the entire system as well. So, the subject needed to send the “teeth clenching” state signal to start the robot, control the robot to reach the object placement area and grab the object. After grabbing the object, the robot will automatically return to the initial position. The completion time (in seconds) was measured on a stopwatch by an experimenter. After finishing the experiment, each subject was asked to reply his/her experiences in operating the proposed SSVEP BCI robot control system.

Classification accuracy, completion time, and ITR were calculated for the online experiment to reflect the overall performance of the system. The ITR is a well-known parameter for BCI system evaluation [37]. For a trial with $N$ possible targets in which each target has the same possibility, the classification accuracy P that the target will be hit is the same for each target. The higher ITR means that the BCI system can transfer more information per unit of time. The bits of information communicated per minute were calculated as follows:

$\displaystyle ITR=\frac{60}{T}\left[{\log_{2}N}+P{\log_{2}P}+(1-P)\log_{2}% \left(\frac{1-P}{N-1}\right)\right]$ (8)

In this study, the number of targets $N$ is 4, and the time window length $T$ is 4 s.

2.9 Target recognition algorithm

Canonical correlation analysis (CCA) is a statistical method used to measure the underlying correlation between two multidimensional variables. Therefore, CCA extends the ordinary correlation to two sets of random variables and has been widely used for SSVEP recognition. Due to the power law distribution of the power spectrum spontaneous, the electroencephalogram (EEG) signal will affect the detectability of SSVEP at different frequencies. Thus, CCA may not give the best accuracy for SSVEP classification, even though many researchers have proven that the performance of CCA is powerful [38]. To alleviate this problem, normalized canonical correlation coefficients for CCA needed to enhance the frequency detection of SSVEP. A signal with a higher characteristic representation is used instead of the sine and cosine signal as a reference signal to improve the recognition accuracy.

Figure 9 shows the flowchart of the CP-CCA [30]. We used the CCA method to determine the best data to represent multiple trials of EEG data recorded on a single channel when subjects gazed at the same frequency of visual stimulation. Suppose that recorded EEG data of multi-trials in the specific stimulus frequency are $X_{h,f_{m}}\in R^{n\times i}$ , n is the number of trials, and h represents four different channels (O1, O2, P7, and P8). Here two vectors $w_{h,x}\in R^{n\times 1}$ and $w_{h,y}\in R^{j\times 1}$ are selected to find the maximum correlation coefficient of ${\hat{X}_{{h,f}_{m}}=w^{\rm T}_{h,x}\times X}_{h,f_{m}}$ and ${\hat{Y}}_{f_{m}}=w^{\rm T}_{h,y}\times Y_{f_{m}}$ . The maximum correlation of one channel can be described as:

$\displaystyle\max_{w_{h,x},w_{h,y}}\rho(x,y)=\frac{E[\hat{X}_{{h,f}_{m}}{\hat{% Y}}^{\rm T}_{f_{m}}]}{\sqrt{E[\hat{X}_{{h,f}_{m}}{\hat{X}}^{\rm T}_{h,f_{m}}]E% [{\hat{Y}}_{f_{m}}{\hat{Y}}^{\rm T}_{f_{m}}]}}=\frac{E[w^{\rm T}_{h,x}X_{h,f_{% m}}Y^{\rm T}_{f_{m}}w_{h,y}]}{\sqrt{E[w^{\rm T}_{h,x}X_{h,f_{m}}X^{\rm T}_{h,f% _{m}}w_{h,x}]E[w^{\rm T}_{h,y}Y_{f_{m}}Y^{\rm T}_{f_{m}}w_{h,y}]}}$ (9)

The reference signal $\hat{X}_{{h,f}_{m}}$ reflects the frequency component of SSVEP of different channels. Moreover, it contains the common character of the single channel with multi-trials for the same stimulation frequency. When optimal reference signals of different stimulus frequencies $\hat{X}_{h,f_{1}},\hat{X}_{h,f_{2}},\ldots,\hat{X}_{f_{m}}$ were obtained, the correlation coefficient ${\rho}_{h,f_{m}}$ between the test trial are recognized according to the maximum value ${\rho}_{f_{m}}$ , which is sum of ${\rho}_{h,f_{m}}$ , and can be defined as

$\displaystyle{\rho}_{f_{m}}={\rho}_{O1,f_{m}}+{\rho}_{O2,f_{m}}+{\rho}_{P7,f_{% m}}+{\rho}_{P8,f_{m}}$ (10)

In this work, the number of target stimulus frequency n=4. For the reference signal, its fundamental and second frequency are considered.

Figure 9.

Flowchart of the CP-CCA method for frequency recognition in an SSVEP-based BCI. For the same target frequency $f_{m}$ , the different trials of recorded EEG data are dispersed and then reorganized according to the different channels, $X_{O1,f_{m}}$ , $X_{O2,f_{m}}$ , $X_{P7,f_{m}}$ , and $X_{P8,f_{m}}$ . The optimal reference signals of different channels ( $\hat{X}_{O1,f_{m}}$ , $\hat{X}_{O2,f_{m}}$ , $\hat{X}_{P7,f_{m}}$ and $\hat{X}_{P8,f_{m}}$ ) under certain stimulus frequency $f_{m}$ are obtained by the CCA between the channel-based EEG data and the sine-cosine signals $Y_{f_{m}}$ . The SSVEP target frequency $f_{s}$ of a new test data of single trial is recognized according to the maximum value of the sum of ${\rho}_{h,f_{m}}$ .

3. Results

Table 2 illustrates the classification accuracy and ITR of all subjects in the target recognition task where the system sent commands at a speed of 4 s per command. A total of 32 trails are required for each subject. The BCI accuracy was evaluated by taking the ratio of the correct commands to the total commands. The average classification accuracy in the object recognition task was 91.88 $\pm$ 5.15%, and the ITR of subjects was 23.15 $\pm$ 3.27 bits/min. For all individuals, the maximal and minimal classification accuracy were 100% (subject S7) and 81.25% (subject S4) respectively. The offline results of the target recognition task prove the feasibility of the proposed SSVEP-based BCI implementation in the BCI-controlled robotic system.

Table 2
Classification accuracy and ITR in the offline experiment

Subject	Accuracy (%)	ITR (bits/min)
S1	93.75	23.45
S2	93.75	23.45
S3	90.63	21.04
S4	81.25	15.10
S5	90.63	21.04
S6	93.75	23.45
S7	100	29.80
S8	90.63	21.04
S9	96.88	26.25
S10	87.50	18.87
Mean $\pm$ SD	91.88 $\pm$ 5.15	23.15 $\pm$ 3.27

Figure 10.

Snapshots of the proposed brain-controlled robot system performing the move-grab-back task.

Table 3

Results of the move-grab-lift robot control task

Subject	Session	Completion time (sec)				Number of commands
		O1	O2	O3	O4	O1	O2	O3	O4
S1	BCI	680 ${}^{*}$	655	675 ${}^{*}$	667	2	2	2	2
	Manual	608	595	600	608	10	9	10	9
S2	BCI	655	680 ${}^{**}$	672	690 ${}^{**}$	2	2	2	2
	Manual	613	610	618	603	9	11	9	11
S3	BCI	653	660	650	662	2	2	2	2
	Manual	600	611	593	605	9	9	9	9
S4	BCI	661	667	663	655	2	2	4	2
	Manual	597	610	607	599	9	9	9	9
S5	BCI	681	663	697 ${}^{**}$	688 ${}^{*}$	2	2	2	2
	Manual	602	605	618	610	9	9	11	10
S6	BCI	667	673	662	670	2	2	2	2
	Manual	607	610	593	600	9	9	9	9
S7	BCI	698 ${}^{*}$	662	659	680 ${}^{*}$	2	2	2	2
	Manual	618	602	597	620	10	9	9	10
S8	BCI	669	695 ${}^{*}$	708 ${}^{**}$	670	2	2	2	2
	Manual	610	605	620	601	9	10	11	9
S9	BCI	662	666	657	670	2	2	2	2
	Manual	601	607	595	611	9	9	9	9
S10	BCI	675	653	662	699 ${}^{*}$	2	2	2	4
	Manual	597	590	606	625	9	9	9	10
BCI	Mean $\pm$ SD	670.1 $\pm$ 13.7	667.4 $\pm$ 12.6	670.5 $\pm$ 18.5	675.1 $\pm$ 13.8	2 $\pm$ 0	2 $\pm$ 0	2.2 $\pm$ 0.63	2.2 $\pm$ 0.63
Manual		605.3 $\pm$ 7.1	604.5 $\pm$ 7.0	604.7 $\pm$ 10.8	608.2 $\pm$ 8.7	9.2 $\pm$ 0.4	9.3 $\pm$ 0.6	9.5 $\pm$ 0.8	9.5 $\pm$ 0.7
Ratio of BCI/Manual		1.11	1.10	1.11	1.11	0.22	0.22	0.23	0.23

${}^{*}$ indicates one obstacle ${}^{**}$ indicates two obstacles.

Table 3 shows the results of the real-time robot service task. The subjects were asked to perform a move-grab-back task four times using both the manual interface and BCI. The subjects grabbed objects back in the sequence they liked, including Object 1 (O1), Object2 (O2), Object3 (O3) and Object4 (O4). After some practicing trials ( $\sim$ 15 min) for familiarizing with the system, all subjects were able to control the robot system and successfully finished the entire grabbing task with two interfaces. The results of the real-time control indicate that manual control can complete the task quickly and take less time (O1: 605.3s, O2: 604.5s, O3: 604.7s, O4: 608.2s). However, subjects generally believe that manual control requires a large number of commands to complete a grabbing task, which can tire the subject. Compared with the BCI control method, although the time required to perform a complete grabbing task is longer than that of the manual control, the subject can complete the task with minimum commands. Apart from one command signal (EMG) that starts the robot, only one minimum EEG command is needed to grab an object. If there is a recognition error, two additional control commands are required. One command is used to cancel the current instruction and the other is used to select the correct target. The biggest difference from manual control is that, using the characteristics of intelligent robots, the subject does not need continuous control, reducing the load of body and brain. There are two main reasons why the BCI control takes a long time: 1) Obstacle avoidance procedures use sonar to detect left and right distances, and 2) Naomark is used for navigation and identification of grabbed objects. In terms of the average total completion time, the BCI control took 1.11 times longer than the manual control.

In general, most subjects believe that although the time required for the BCI control robot to complete the grab task is longer than the manual control, the user experience is better. Subjects are in a relaxed state when using the system, and only need to make decisions instead of controlling the system all the time. Except for subject S5, which failed to grab O4 with the BCI control, all subjects eventually successfully grabbed the object. Figure 10 shows snapshots in which subject S3 first performed the move-grab-back robot service task by the proposed brain-controlled robotic system. Obstacles are randomly placed in each subject’s task and the obstacle positions of the two control interface tasks are the same.

4. Discussion and conclusion

In this study, an efficient BCI system was established with limited ITR. We showed how healthy subjects operate a non-invasive SSVEP BCI to control an intelligent humanoid robot to perform move-grab-back domestic service tasks. The result of the offline experiment without the robot movement session showed that the average accuracy of all subjects was 91.88 $\pm$ 5.15%, and the average ITR was 23.15 $\pm$ 3.27 bits/min. In addition, it turned out that all naive subjects were able to use the proposed BCI system to deliver commands with high accuracy. These results demonstrated that SSVEP-based BCIs is extremely promising and could control the humanoid robot successfully to complete the grab task in an indoor environment without collision.

The main aim of this study was to design a new type of service robot which can help elderly users and disabled groups to grab telepresence objects with the least brain load and improve their ability of self-care. The intelligent humanoid robot is featured with navigating, scanning, grabbing and avoiding obstacles and can help users to reach any position in the house and grab objects in the complex indoor environment. SSVEP was a periodic response evoked by external visual stimulus at a constant frequency, and subjects did not need to perform additional operations in addition to focusing on a specific target. Compared with the traditional brain-controlled robot systems, this system changed the communication of BCI from one-command-to-one-motion to one-command-to-multi-motions. For target recognition, we used the special landmark (Naomark) of the robot Nao to enable the robot to quickly identify objects and locate their position in a complex background, thereby reducing the calculation time of the system to process images and improving the overall efficiency of the system. Users only need to make decisions instead of controlling the robot to walk, scan objects, grab objects and avoid obstacles. In other words, they can complete complex tasks with minimum commands by the improved interaction system. Subjects generally agreed that the user interface enable susers to feel relaxed as fewer and simpler commands are needed. In this study, a portable, lightweight and wireless EEG device was used to measure EEG signals to further increase the practicality of the proposed system in daily life. Without the wires, the subjects could move their bodies properly instead of staying still. User comfort can be improved during the process, and subjects do not feel uncomfortable on their scalp when dry electrodes are used. The high accuracy of the proposed system indicated that the portable, lightweight and wireless EEG device that is utilized here could be efficiently for SSVEP-based BCI applications.

It should also be noted that subject S5 failed once in the process of grabbing objects. So the system needs to be further improved in the following directions. First, in order to improve system response and enhance user experience, the coding and decoding method of the BCI system should be improved. Second, it is necessary to further develop the robot, optimize the program and improve the execution precision for higher success rate. Third, more advantage should be taken of the fusion of more sensors and actuators, such as an accelerometer and gyroscope. Under the premise of ensuring the balance of the robot and not falling, the walking speed of navigation and the execution efficiency of the system needs to be improved.

Footnotes

Acknowledgments

This work was supported by the Young and Middle-Aged Innovation Talents Cultivation Plan of Higher Institutions in Tianjin (Grant no. 20130830) and the National Natural Science Foundation of Tianjin (Grant no. 18JCYBJC87700).

Conflict of interest

The authors declare that there is no conflict of interest regarding the publication of this manuscript.

References

Cipriani

Segil

Birdwell

Weir

. Dexterous control of a prosthetic hand using fine-wire intramuscular electrodes in targeted extrinsic muscles. IEEE Transactions on Neural Systems and Rehabilitation Engineering. 2014; 22(4): 828-836.

Fukuma

Yanagisawa

Saitoh

Hosomi

Kishima

Shimizu

, et al. Real-time control of a neuroprosthetic hand by magnetoencephalographic signals from paralysed patients. Scientific Reports. 2016; 6(1): 21781-21781.

Fernandezrodriguez

Velascoalvarez

Ronangevin

. Review of real brain-controlled wheelchairs. Journal of Neural Engineering. 2016; 13(6): 061001.

Mittal

Konno

Komizunai

. Implementation of HOAP-2 humanoid walking motion in OpenHRP simulation. 2015; 29-34.

Saeedvand

Jafari

Aghdasi

Baltes

. A comprehensive survey on humanoid robot development. Knowledge Engineering Review. 2019; 34.

Shigemi

. ASIMO and humanoid robot research at Honda. 2017; 55-90.

Brose

Weber

Salatin

Grindle

Wang

Vazquez

, et al. The role of assistive robotics in the lives of persons with disability. American Journal of Physical Medicine & Rehabilitation. 2010; 89(6): 509-521.

Fan

Liu

. EEG-based brain-controlled mobile robots: A survey. IEEE Transactions on Human-Machine Systems. 2013; 43(2): 161-176.

Vidal

. Toward direct brain-computer communication. 1973; 2(1): 157-180.

10.

Gao

Dou

Belkacem

Chen

. Noninvasive electroencephalogram based control of a robotic arm for writing task using hybrid BCI system. BioMed Research International. 2017; 2017: 1-8.

11.

Gao

Zhao

Song

Wang

. Controlling of smart home system based on brain-computer interface. Technology and Health Care. 2018; 26(5): 769-783.

12.

Mc Farland

Wolpaw

. Brain-computer interfaces for communication and control. COMMUN ACM 2011. 2011; 54(5): 60-66.

13.

Hochberg

Bacher

Jarosiewicz

Masse

Simeral

Vogel

, et al. Reach and grasp by people with tetraplegia using a neurally controlled robotic arm. Nature. 2012; 485(7398): 372-375.

14.

Wolpaw

Mcfarland

Neat

Forneris

. An EEG-based brain-computer interface for cursor control. Electroencephalography and Clinical Neurophysiology. 1991; 78(3): 252-259.

15.

Aljuaid

Salem

. A survey of electroencephalogram based brain computer interface applications. International Journal of Engineering Research and Technology. 2019; .

16.

Cecotti

. Spelling with non-invasive Brain-Computer Interfaces-current and future trends. Journal of Physiology-paris. 2011; 105(1): 106-114.

17.

Liu

Chen

Xie

. Review: Recent development of signal processing algorithms for SSVEP-based brain computer interfaces. Journal of Medical and Biological Engineering. 2014; 34(4): 299-309.

18.

Nakanishi

Wang

Mitsukura

Jung

. A high-speed brain speller using steady-state visual evoked potentials. International Journal of Neural Systems. 2014; 24(6): 1450019.

19.

Chen

Wang

Gao

Jung

Gao

. Filter bank canonical correlation analysis for implementing a high-speed SSVEP-based brainâ€“computer interface. Journal of Neural Engineering. 2015; 12(4): 046008.

20.

Chen

Wang

Nakanishi

Gao

Jung

Gao

. High-speed spelling with a noninvasive brainâ€“computer interface. Proceedings of the National Academy of Sciences of the United States of America. 2015; 112(44): 201508080.

21.

Wang

Gao

Hong

Gao

. A practical VEP-based brain-computer interface. IEEE Transactions on Neural Systems and Rehabilitation Engineering. 2006; 14(2): 234-240.

22.

Wang

Cheng

Jung

. Measuring steady-state visual evoked potentials from non-hair-bearing areas. 2012; 2012: 1806-1809.

23.

Lin

Zhang

Gao

. Frequency recognition based on canonical correlation analysis for SSVEP-based BCIs. IEEE Transactions on Biomedical Engineering. 2007; 54(6): 1172-1176.

24.

Millan

Renkens

Mourino

Gerstner

. Noninvasive brain-actuated control of a mobile robot by human EEG. IEEE Transactions on Biomedical Engineering. 2004; 51(6): 1026-1033.

25.

Chen

Zhao

Wang

Gao

. Combination of high-frequency SSVEP-based BCI and computer vision for controlling a robotic arm. Journal of Neural Engineering. 2019; 16(2): 026012.

26.

Tsui

CSL

Gan

. A self-paced motor imagery based brain-computer interface for robotic wheelchair control. Clinical Eeg and Neuroscience. 2011; 42(4): 225-229.

27.

Zhang

Yan

Zhang

, et al. Control of a wheelchair in an indoor environment based on a brainâ€“computer interface and automated navigation. IEEE Transactions on Neural Systems and Rehabilitation Engineering. 2016; 24(1): 128-139.

28.

Duan

Lin

Zhang

. Design of a multimodal EEG-based hybrid BCI system with visual servo module. IEEE Transactions on Autonomous Mental Development. 2015; 7(4): 332-341.

29.

Zander

Kothe

Jatzev

Gaertner

. Gaertner

. Enhancing human-computer interaction with input from active and passive brain-computer interfaces. in: Brain-Computer Interfaces: Applying our Minds to Human-Computer Interaction London: Springer London. Tan

Nijholt

, editors, 2010; 181-199.

30.

Sellers

Krusienski

Mcfarland

Vaughan

Wolpaw

. A P300 event-related potential brainâ€“computer interface (BCI): The effects of matrix size and inter stimulus interval on performance. Biological Psychology. 2006; 73(3): 242-252.

31.

Gao

Zhang

Wang

Dong

Song

. Channel projection-based CCA target identification method for an SSVEP-Based BCI system of quadrotor helicopter control. Computational Intelligence and Neuroscience. 2019; 2019: 1-13.

32.

Brainard

. The psychophysics toolbox. Spatial Vision. 1997; 10(4): 433-436.

33.

http://doc.aldebaran.com/2-1/naoqi/vision/allandmarkdetection.html.

34.

Liu

Luo

Zhang

. Target recognition and heavy load operation posture control of humanoid robot for trolley operation. 2018; 280-283.

35.

Yang

. Kinematics modeling and experimental verification of baxter robot. 2014; 8518-8523.

36.

Yuan

. Visual servo grasping of household objects for NAO robot. Journal of Shandong University. 2014.

37.

Wolpaw

Birbaumer

Mcfarland

Pfurtscheller

Vaughan

. Brainâ€“computer interfaces for communication and control. Clinical Neurophysiology. 2002; 113(6): 767-791.

38.

Bin

Gao

Yan

Hong

Gao

. An online multi-channel SSVEP-based brain-computer interface using a canonical correlation analysis method. Journal of Neural Engineering. 2009; 6(4): 046002.

Implementation of an SSVEP-based intelligent home service robot system

Abstract

BACKGROUND:

OBJECTIVE:

METHODS:

RESULTS:

CONCLUSION:

Keywords

1. Introduction

2.1 Subjects

2.2 System description

Table 2 Classification accuracy and ITR in the offline experiment

Footnotes

Acknowledgments

Conflict of interest

References

Table 2
Classification accuracy and ITR in the offline experiment