Ergonomics analysis based on intention inference

Abstract

Traditional intention inference methods rely solely on EEG, eye movement or tactile feedback, and the recognition rate is low. To improve the accuracy of a pilot’s intention recognition, a human-computer interaction intention inference method is proposed in this paper with the fusion of EEG, eye movement and tactile feedback. Firstly, EEG signals are collected near the frontal lobe of the human brain to extract features, which includes eight channels, i.e., AF7, F7, FT7, T7, AF8, F8, FT8, and T8. Secondly, the signal datas are preprocessed by baseline removal, normalization, and least-squares noise reduction. Thirdly, the support vector machine (SVM) is applied to carry out multiple binary classifications of the eye movement direction. Finally, the 8-direction recognition of the eye movement direction is realized through data fusion. Experimental results have shown that the accuracy of classification with the proposed method can reach 75.77%, 76.7%, 83.38%, 83.64%, 60.49%,60.93%, 66.03% and 64.49%, respectively. Compared with traditional methods, the classification accuracy and the realization process of the proposed algorithm are higher and simpler. The feasibility and effectiveness of EEG signals are further verified to identify eye movement directions for intention recognition.

Keywords

EM EEG tactile feedback wireless sensor network flying driving brain electrical signals data fusion

1 Introduction

It has been a research focus in the field of human-machine interaction to make full use of various sensing form such as visual, auditory and tactile to enable people to obtain information, quickly and comprehensively. Improving the aircraft’s human-computer interaction system’s ability to perceive the pilot’s intentions and state inference is one of the key objectives of aircraft human-computer interaction intelligence [1]. The intelligence provides theoretical basis and technical support for the design of adaptive aircraft human-computer interaction system, and reduces human errors during operation [2]. Human-computer interaction intention refers to the user’s goals and expectations when operating a computer system. Traditional human-computer interaction intention inference methods mainly rely on EEG signals or EM signals.

Brain-computer-interface (BCI) is an application of EEG signals. It can establish a communication relationship between human brain and external equipment, and then control external equipment through the brain [3]. BCI has already been applied in medicine [4], nerves Biology [5], psychology [6] and other fields. Motor Imagery (MI) EEG has the characteristics of flexibility, non-invasiveness, low environmental requirements and high resolution. The frequency band power of EEG signals changes with the content of the MI task during the moving imaging process, which is called synchronization/desynchronization (ERS/ERD). The generation of ERS/ERD is related to internal or external events. When one limb is moving or performing motor imagery, the sensory motor area on the opposite side of the brain reduces the μ rhythm and β rhythm energy, and the ipsilateral motor sensory area μ rhythm and β rhythm Rhythmic energy increases. This rule makes it possible for ERS/ERD to control external devices or perform motor imagery intention inference. Based on the inherent time scale decomposition and Support Vector Machine (SVM), the dataset of the third BCI competition and the dataset2a of the fourth BCI competition were classified by Guihu Jiang. The final classification results can reach 94.65% and 90.63%, respectively. In [7], all the experiments were designed to complete continuous grasping tasks through MI controlled robotic arms, and the success rate can reach 85%. However, these intention inference methods only rely on the advantages of EEG signals without fusing EM signals, and the accuracy of inference needs to be further improved.

Many studies have shown that visual channels can provide more than 80% of external information to people. In recent years, many research have been conducted on user-computer interaction intentions based on human visual behavior. In [8], EM data were used by Deng to analyze the users’ behavioral intentions and emotional experience. The users’ interaction intentions are determined by Qijie Zhao through eye and head movements, and the accuracy of the gaze intention focus area is 92%. In addition, eye tracking is also widely applied in user interaction behavior analysis, user visual search analysis [9] and visual stimulus interest analysis [10]. These methods only rely on EM signals without fusing EEG signals, so it cannot make full use of EEG signals to analyze the cognitive state of human brain.

Many scholars have also tried to fuse multiple physiological information in order to improve the accuracy of human search intention, action intention or cognitive state inference. In [11], EEG and EM signals were fused by Park to infer the implicit interaction intention of people in the visual search process, and the inference accuracy of EEG and EM signals fusion is about 5% higher than that of a single physiological signal. In [12], Ping Xie fused EEG, ECG and EMG signals to evaluate driving fatigue, and the results show that the inference accuracy of multi-physiological feature fusion is significantly higher than that of a single physiological signal. The fusion of EEG and EMG signals is used to perform action patterns inference, and the results show that the inference accuracy of EEG fusion signal is 98%, higher than that of only EEG features, which is 75%. Obviously, the fusion of multiple physiological signals can improve the accuracy of human cognition and intention inference, but the above research does not involve human-computer interaction intention inference.

It is of great significance for intention inference to solve the problem of human factors in pilot status safety and selection training. The detection of a pilot’s intention through the combination of sensors, software technology, mathematical modeling and intelligence is both important and urgent, together with the establishment of a monitoring and evaluation model based on the intention state and accurate assessment of a pilot’s. Although a lot of research has been carried out in this field, the actual results have not been widely applied due to the following two technical bottlenecks:

Some issues exist, such as low objectivity and poor accuracy because of impractical detection methods and the limitations of feature extraction. The fatigue state of people cannot be accurately reflected. In addition, data analysis methods cannot deeply explore effective features, and cannot effectively integrate multi-dimensional data for intention inference.

Existing methods are not effective and optimized to evaluate and verify the detection. As a result, it is difficult to apply the selection training with reliable detection data.

To address the above problems, we propose in this paper a new approach by fusing multi-dimensional data to build a cognitive analysis model, through long short-term memory (LSTM) deep neural networks. First, multi-dimensional data are composed of visual tracking, EEG signals and tactile perception. Then, during a reaction time test, the pilot monitoring experiment in simulated flight training environment is carried out. Combined with LSTM the fatigue state characteristics in multi-dimensional data are deeply explored, and the pilot state detection model is established to analyze the pilot fatigue state intention inference from the perspective of pilot operation response. Finally, based on our experience on aviation medicine and existing pilot training evaluation system, the fatigue state detection model validation evaluation research is carried out, and the model is iteratively optimized through feedback.

More specifically, the implementation of our proposed approach on human-computer interaction intention inference incudes a few steps: (i) to collect users’ EEG and EM signals for feature extraction; (ii) to use pattern inference algorithm to classify and infer physiological signal features; (iii) to perform decision-level fusion of classification algorithm to obtain the final result, design users’ intention induction experiment and verify the feasibility. The effects of different EEG feature extraction methods and different machine learning algorithms are compared on the inference accuracy in this paper.

2 Materials and methods

EEG data analysis involves a variety of signal processing techniques, including signal acquisition, preprocessing, feature extraction, etc. A variety of methods are widely used in data classification, such as KNN based on sample feature distance, linear SVM based on VC dimension theory and deep neural networks. In this chapter, the key technologies of the above two fields involved in this research are respectively introduced.

2.1 Signal acquisition

Carlo Matteucci [13] first obtained the muscle nerve electrical signal with a galvanometer in 2017 and established the concept of neurophysiology. The widely used non-invasive collection methods in the field of signals include EEG, FMRI, NIRS and MEG [14]. The multi-channel EEG method, which integrates high time resolution, low cost and non-invasive safety, is the most widely used. Due to the significant growth of the continuous advancement, the 10-20 standard lead, 10-10 standard lead and 10-5 standard lead established by the American Society of Clinical Neurophysiology are the most common in clinical trials [15], as shown in Figs. 1, 2 and 3.

Fig. 1

10-5 standard lead system.

Fig. 2

10-10 standard lead.

Fig. 3

10-20 standard lead system.

Three standard system guides are extensions of each other and keep the same overlap in the naming rules. It simplifies the EEG signal research process and reduces the difficulty of technical communication, and clears the obstacles to the electrode naming rules. Specific electrode naming and the spatial coordinates can be queried on the website of the American Society of Clinical Neurophysiology.

2.1.1 Correlation analysis results between EEG characteristics and different missions

Based on the 8 EEG characteristics extracted from 32 ports, we have analyzed the differences between flight missions and target capture tasks, target capture tasks and return flight tasks, flight tasks and return flight tasks in order to obtain EEG characteristics that significantly show the difference between the above tasks.

As shown in Table 1, during the flight mission and the task of capturing the target, the θ wave power from Port AF4 in the frontal area, the differential entropy from Port P7 in the temporal area, the θ wave power from Port P3 port in the apical area and the θ wave from Port POz in the occipital area have strong significance. During the flight mission and the return mission, the θ wave power from Port AF4 in the frontal area, the differential entropy from Port P7 in the temporal area, the differential entropy from Port Pz in the apical area and the power Port θ wave from Port POz in the occipital area have strong significance. During the mission of capturing the target and the mission of returning to the flight, the EEG features collected are not significant. $R = \frac{nf}{NF}$ (1)

Table 1

EEG characteristic information table with significant differences among different tasks

Correlation analysis task	Port	Features	Mann-Whitney U test p i
Flight mission and target acquisition mission	AF5	Θ power	0.002352
	P7	Differential entropy	0.005625
	P3	Θ power	0.000997
	POz	Θ power	0.000194
Mission and return mission	AF5	Θ power	0.005458
	P7	Differential entropy	0.000891
	Pz	Differential entropy	0.000283
	POz	Θ power	8.5e-05

In order to further analyze the diversities of different brain regions when different tasks are completed, a difference index (as shown in (1)) is designed. Based on this index, as shown in Fig. 10, four brain regions have the most significant differences between the flight mission and the return flight mission, compared with other tasks. A relatively strong correlation exists between the diversities of the occipital region and the temporal region during the flight mission and return flight mission and the diversities during the flight mission and the task of capturing the target.nf is the total number of features from the significant ports in the brain region. NF is the total number of features from all ports in the brain region.

In this paper, the electrode position recommended by the 10-10 standard lead system is used as a benchmark to carry out the experiment. At the same time, the standard of Jayshree [16] is used to divide EEG signals into 5 different bands according to different frequencies, as shown in Table 2.

Table 2

EEG signals of different bands

Band name	Delta	Theta	Alpha	Beta	Gamma
Frequency range HZ	<4	4–7	8–15	16–31	>32
bandwidth	4	3	7	15

EEG signals are generally considered to be non-stationary. Although the multi-channel EEG distribution is usually regarded as a multivariate Gaussian distribution, the mean and variance characteristics usually change between each segment [17]. Therefore, the research can only be carried out as a steady state in a short time interval. The change of the signal segment distribution can be measured according to the parameters and distribution of the Gaussian process, and the non-Gaussian distribution of EEG signals can be checked by measuring or estimating some high-order matrices such as skewness and kurtosis [18].

The skewness is in measurement for lack of symmetry, and the skewness of the actual signal is: $\begin{matrix} Skewness = \frac{E [{(x (n) - μ)}^{3}]}{σ^{3}} \end{matrix}$ (2) where μ and σ are the mean and standard deviation, respectively, and E is the statistical expectation. Similarly, the kurtosis of the actual signal is: $\begin{matrix} kurt = \frac{m_{4} [x (n)]}{m_{2}^{2} [x (n)]} \end{matrix}$ (3) where m_i [x (n)] is the i-th order central moment of signal x (n). Usually, a normally distributed signal with a kurtosis of 3 is used [19]. $\begin{matrix} Exkurt = \frac{m_{4} [x (n)]}{m_{2}^{2} [x (n)]} - 3 \end{matrix}$ (4)

Given that the statistical average value of the signal is the same as the time average value, the time average value can be estimated for its negative entropy: $\begin{matrix} J_{neg} [x (n)] = H [x_{Gauss} (n)] - H [x (n)] \end{matrix}$ (5) where x_Gauss (n) is a Gaussian random signal with the same covariance as signal x (n). H [x (n)] is the differential entropy [20]. $\begin{matrix} H [x (n)] = \int_{- \infty}^{\infty} p [x (n)] log \frac{1}{p [x (n)]} dx (n) \end{matrix}$ (6) where p [x (n)] is the signal distribution, which defines the KL distance between two distributions is: $\begin{matrix} KL = \int_{- \infty}^{\infty} p_{1} (x_{1} (n)) log \frac{p_{1} (x_{1} (n))}{p_{2} (x_{2} (n))} dz \end{matrix}$ (7)

Therefore, the distribution and the distribution distance of the signal itself can be obtained respectively.

2.2 EEG signal processing

EEG signal processing is mainly composed of signal acquisition, conversion reference, filtering, artifact removal, segmentation, independent component analysis and other operations. In addition, it needs to pay attention whether the collected data is chosen for down-sample according to the actual situation. When down-sampling, it may be necessary to perform linear or non-linear interpolation to complement the disappeared features. The following section will introduce the above operations based on previous experimental experience or existing standard suggestions.

2.3 Data classification

Many research have been conducted on EEG signal classification, which are divided into two categories: statistical methods and neural network methods. In [21], Garrett has compared with linear, non-linear and feature selection methods in EEG signal classification, and found that non-linear classifiers are easier to achieve better results than linear classifiers. Recent research on EEG signal classification are listed in Table 3.

Table 3
Research on EEG signal processing in recent years

Author Data Set Method Classification Accuracy

Lotte BCI competition III CSP SVM 80.43%

Lemm BCI competition III CSP SVM 81.67%

Higashi BCI competition III CSP LDA 89.06±2.43%

Muhammad Emotiv Epoc &BCI competition III Sub-band CSP SBFS 85.00±0.80%

Rajaguru self CSP Logistic Regression 95.88%

Kim self SL Non-Linear Regression 98.11%

Jain self CSP Auto-Regression –

Torabi self ICA SVM+KNN 93.80%

Hramov self CSP ANN 95.00%

Lee self CSP ANN+SVM 98.07%

Biswa self ICA Naïve Bayes 97.53%

Author	Data Set	Method	Classification	Accuracy
Lotte	BCI competition III	CSP	SVM	80.43%
Lemm	BCI competition III	CSP	SVM	81.67%
Higashi	BCI competition III	CSP	LDA	89.06±2.43%
Muhammad	Emotiv Epoc &BCI competition III	Sub-band CSP	SBFS	85.00±0.80%
Rajaguru	self	CSP	Logistic Regression	95.88%
Kim	self	SL	Non-Linear Regression	98.11%
Jain	self	CSP	Auto-Regression	–
Torabi	self	ICA	SVM+KNN	93.80%
Hramov	self	CSP	ANN	95.00%
Lee	self	CSP	ANN+SVM	98.07%
Biswa	self	ICA	Naïve Bayes	97.53%

Besides, many researches are utilizing random forest, KNN, K-means and other methods for the classification of EEG signals. As shown in the above table, although many classification methods are based on statistical methods, more accurate good results can be achieved by the classification methods with neural network. Therefore, an overview of the classification of EEG signals will be provided from these two kinds of methods.

SVM is a popular approach in machine learning. Compared with deep neural networks, it is particularly good at handling situations where feature dimensions are more than the number of samples. When the number of samples are small, SVM can perform better than deep neural network [22]. A new method is proposed for solving the multiple linear regression tasks via a linear polynomial as a constructive formula [23].

The linear SVM is designed to find a hyperplane far away from all types of samples. When the samples randomly fluctuate, the hyperplane far away from the samples has a strong tolerance, which makes SVM not easy to over-fit. The essence of linear SVM is a convex quadratic programming problem, and its optimization objective is: $\begin{matrix} \underset{w, b}{argmax} (min_{i} \frac{2}{∥ w ∥} | w^{T} x_{i} + b |) \end{matrix}$ (8)

The dual problem is equivalent to finding a suitable parameter α: $\begin{matrix} min_{α} \frac{1}{2} \sum_{i = 1}^{m} \sum_{j = 1}^{m} α_{i} α_{j} y_{i} y_{j} x_{i}^{T} x_{j} - \sum_{i = 1}^{m} α_{i} \end{matrix}$ (9)

For linearly inseparable samples, the kernel function can be used to solve the problem. A mapping is constructed to convert the linearly inseparable domain to the linearly separable domain $ϕ : ℝ^{d} \to ℝ^{\tilde{d}}$ , so that the data in $ℝ^{\tilde{d}}$ is linearly separable. Due to the uncertainty of the mapped space $ℝ^{\tilde{d}}$ , that is, when the dimension of the mapped space is very high or even infinite, its computational complexity is $O (\tilde{d})$ . Thus, core techniques are introduced to reduce the complexity. $\begin{matrix} K (x_{i}, x_{j}) = ϕ {(x_{i})}^{T} ϕ (x_{j}) \end{matrix}$ (10)

The appropriate kernel function of SVM can significantly improve the effect of model training in the study of EEG signals. Subasi [24] found that PCA, ICA, and LDA can reduce the data dimension in the study of EEG signals in patients with epilepsy, and it can improve the model generalization of SVM.

3 Experimental process and result analysis

3.1 Subjects

In this section, the users’ EEG and EM data are collected in order to verify the effectiveness of the proposed method. Eight male users, aged 18–22 (Mean = 22.3, SD = 1.8), are recruited in Table 4, who have signed informed consent forms before the experiment. Their vision or corrected vision are normal, and no history of neurological or psychiatric diseases are reported. Also, no one has participated in similar tests before.

Table 4
Physiological signs information

Experimenter information sheetAge/y Height/cm Weight/kg Reaction Level

22.4±2.3 170.2±6.6 61.9±9.0 Good

Experimenter information sheetAge/y	Height/cm	Weight/kg	Reaction Level
22.4±2.3	170.2±6.6	61.9±9.0	Good

In this study, a set of experiments is set to 1 hour because the EEG cap will affect the subjects’ electric waves. Long time may cause the subjects’ physical discomfort. As a result, the subjects are arranged to conduct simulated flight experiments at different times of the day.

3.2 Experimental method

The experiment, testing the subjects individually, is divided into two stages: practice stage and experimental stage.

Practice stage: A subject sits in front of the experimental instrument and performs the same practice test as the formal experiment. The subject is required to complete each task as quickly as possible, ensuring that it is as correct as possible to determine the target position of the enemy aircraft and quickly avoid it operating. In the practice stage, the subject is required to have an accuracy rate of more than 90% and no less than 80 sets of tests before the formal test. It can ensure that test scores become stable, and avoid practice effects in subsequent formal tests and affect the accuracy of experimental results, as shown in Fig. 4.

Fig. 4

Experimental flowchart.

Experimental equipment includes Dell computer with Windows10 Ultimate operating system (x64, 3.10-ghz Intel Core i5-3450 processor, 8-GB RAM, 512-gssd, NVIDIA GeForce GT630 graphics card), RED-M type EM tester (SMI Compan, Berlin, Germany) and Neuroscan NuAmps32 EEG (ErgoLAB), as shown in Fig. 5. The EM signal sampling frequency is 60 Hz. The display resolution is 1280×1024 pixels. The screen brightness is 300 cd/m2. The distance between the user and the screen is about 70 cm, and the eyes of the test user are roughly at the same height as the center of the screen. The electrode distribution of the EEG adopts the international standard 10-20. The left mastoid is selected as the reference electrode, and the middle prefrontal lobe is the ground electrode. The vertical and horizontal channel electrooculogram signals are collected. The sampling frequency is 250 Hz. The 50 Hz notch and 0.05 10 Hz on-line band-pass filtering are performed to ensure that the electrode impedance is less than 5 kΩ. The EMG and EOG artifacts are removed after collecting the signals, and the specific position of the electrode is shown in Fig. 6.

Fig. 5

EEG acquisition signal equipment.

Fig. 6

EEG cap port diagram.

The classic motor imagery experiment paradigm is adopted in this paper. Before the operation imagination starts, the word “relax” is displayed on the screen for 2 seconds, and the subject is relaxed and ready to start. Then, the word “prepare” is displayed on the screen for 1 second, which prompts the subject to start operating the imagination. Next, the screen presents the operation interface, as shown in Figure 4. The subject’s operations include “move left”, “move right” and “launch missiles”. The icon turns yellow when the subject needs to operate and imagine. The subject can imagine different operations according to different prompts. The total time of each operation is 1 second.

3.3 Pilot fatigue analysis and modeling

In this section, main work includes coding, fusion processing, computer cognitive analysis of the data obtained by the sensors, the analysis model construction of a pilot fatigue state, and the detection realization of a pilot fatigue state. The integration is divided into three processes:

(1) Data preprocessing

The real time three-wheel pose and spatial position of the head are obtained by head motion measurement, and the eye tracking measurement is carried out by eye tracker to obtain the eye movement pose data and the eye gaze point. The facial image is acquired, and the facial feature points are identified in real time by facial phase to obtain facial expression feature profile data. 32 groups of EEG data are obtained in real time by 32-lead EEG, and the physiological information of four rhythm waves is calculated to obtain pulse, heart rate, skin temperature, etc.

(2) Multi-dimensional data dimensionality reduction

The multi-dimensional raw data include facial image, eye space posture, eye line trajectory, head space posture, 32 groups of EEG electrode power spectrum data, physiological parameters information, which have strong correlation and more dimensions. Through further feature detection and extraction, key features are obtained from a large number of different forms of data as analysis materials to achieve data dimensionality reduction, which can simplify the training and prediction of the model and improve the detection efficiency. Our research focuses on the analysis and research of visual fixation behavior and facial emotion, in which the EEG and physiological data use the current mature analysis algorithm and medical evaluation standard. Four aspects are included:

Visual fixation behavior analysis based on head-eye motion line of sight tracking: the flight instrument model is established according to the cockpit environment. Combined with the gaze data, the pilot’s gaze object can be detected in order to obtain the pilot’s instrument gaze information and instrument gaze response time data. At the same time, according to the eye movement information, six characteristic indicators of visual gaze are calculated: duration of the first gaze point, duration of the second gaze point, the number of gazes, duration of one gaze, pupil diameter and saccade distance.

Analysis of EEG data graph: Based on the power spectrum obtained from EEG information, EEG signals are divided into four bands according to different frequencies, and an infinite mixture Gaussian model is established to analyze the brain activity degree of four bands.

Analysis of physiological data: Main physiological indicators used in this study are pulse, heart rate and skin temperature. The state changes related to the above index changes have been clearly defined in medicine, and these indexes can be directly used to study the relationship between physiological parameters and cognitive state of pilots.

(3) Fusion analysis modeling

In order to identify the fatigue cognitive status of pilots, a deep learning model is established based on parameter-driven LSTM to identify the fatigue status of pilots and construct a computer cognitive system. Four aspects of data as the input include visual gaze behavior, facial emotion, brain, and physiological information. The associated data is extracted through the LSTM situation analysis method, and the three key indicators of alertness, attention and operational response in the fatigue state are respectively extracted. Quantitative analysis is performed to assess the fatigue status of pilots and make predictions.

3.4 Experimental design

8 subjects are recruited in the experiment, whose average age is 24 years old. Their visual acuity reports indicate normal or corrected vision. Each subject performs 4 sets of experiments, and each set of experiments lasts for 50 s (up, down, left, right, top left, top right, bottom left and bottom right, 8 directions). The subject sits in front of the monitor with a horizontal distance of 80 cm from the eye to the monitor screen, as shown in Fig. 7.

Fig. 7

Experimental environment diagram.

All the curtains are kept closed during the experiment. In each experiment, an experimental user and an operator are the only one to be allowed to enter and turn off other electronic devices to eliminate the interference of light changes and other electromagnetic signals.

A red square is in the center of the screen, and 4 blue squares around it move randomly in 8 directions (up, down, left, right, top left, top right, bottom left and bottom right). The subject

Keeps watching the randomly moved blue squares, and operates the red square for avoidance. The movement speed of blue squares is gradually increasing.

The process of each experiment is as follows: Firstly, the red square stays in the center of the screen. Then, blue squares move randomly in one of eight directions, and the whole experiment lasts for 50 s. In the experiment, the subject is required not to blink as far as possible when the ball is moving. The red square is a rest time when blue squares are not touched, and the subject can blink. The experimental program is written in e-prime, and the subject sends a marker signal to the EEG during operation to ensure that the EM behavior is synchronized with the collected EEG signal in time in order to collect more accurate data.

3.5 Result analysis

The preprocessing can obtain the corresponding valid data of each channel (that contains 52 values) through simulating the 1 second time window of the aircraft driving and moving. The 8-channel data (Fp1, Fp2, F7, F8, T7, T8, P7, P8) is normalized with baseline removed, and the 10th order least square polynomial is used to eliminate noise in order to achieve signal smoothness. The polynomial P(x) is fit to obtain the fitted curve, as shown in Figure 8. Black solid lines are EEG signals of Fp1, F7, T7 and P7 channels respectively, and black dashed lines are EEG signals of Fp2, F8, T8, and P8 channels respectively.

Fig. 8

Denoising image of 8-channel EEG signal in 8 directions.

The EEG signal change patterns of 8 channels are included in Figure 8. When the eyes look up, the amplitudes of 8 channels on the forehead have a downward trend, as shown in Fig. 8-a. When the eyes look down, the amplitudes of 8 channels have an upward trend, as shown in Fig. 8-b. When the eyes look to the left and right, the potentials of 4 channels (Fp1, F7, T7, P7) on the left side of the forehead and 4 channels (Fp2, F8, T8, P8) on the right side of the forehead have the same polarity, as shown in Fig. 8-c and Fig. 8-d. When the eyes look to the upper left and lower right, the potentials of 4 channels (Fp1, F7, T7, P7) on the left side of the forehead and 4 channels (Fp2, F8, T8, P8) on the right side of the forehead have the opposite polarity, as shown in Fig. 8-e and Fig. 8-f. The voltage difference between the EEG on both sides of the forehead have similar trends of decreasing or increasing. When the eyes look to the upper right and lower left, the potentials of 4 channels (Fp1, F7, T7, P7) on the left side of the forehead and 4 channels (Fp2, F8, T8, P8) on the right side of the forehead have the opposite polarity, as shown in Fig. 8-g and Fig. 8-h. The voltage difference between the EEG on both sides of the forehead have similar trends of decreasing or increasing. Then, SVM is used to classify the preprocessed data. The total of 8 subjects have participated in the experiment. Each subject can collect 10 sets of data for analysis and classification, and 70% of the samples are selected as the training set.

Each subject has completed 8×10×80 = 6400 experiments in total. 8×800×70% =4480 experiments are selected as the training set, and the rest are the test set. The sampling method of the training set is: 10 groups of experiments are in total. Each group includes 80 data, which includes 10 data in each of 8 directions. Each group selects 40 data, which includes 5 data in each of 8 directions. The training set is used to train the SVM classifier, and this classifier perform classification with the training set in order to obtain the final result, as shown in Table 5.

Table 5

Classification rates of 100∼1100 ms time period (Unit: %)

Participant	Up	Top Left	Down	Bottom Left	Left	Top Right	Right	Bottom Right	Average classification rate
S1	79.09	59.58	70.55	62.32	84.37	70.75	82.72	68.13	72.19
S2	60.15	60.31	75.69	63.88	89.56	61.10	90.23	65.05	70.75
S3	72.23	57.75	69.96	60.11	75.00	62.23	76.61	64.10	67.25
S4	72.58	61.23	80.58	64.50	79.45	66.01	78.55	63.02	70.74
S5	88.85	65.59	83.96	61.96	90.27	70.85	91.00	62.31	76.85
S6	70.58	58.85	72.36	59.96	88.86	65.23	85.99	64.10	70.74
S7	85.12	62.30	81.96	59.35	79.52	65.14	78.63	67.82	72.48
S8	77.54	58.28	78.55	55.32	80.03	68.58	85.36	61.39	70.63
Average value±	75.77±	60.49±	76.70±	60.93±	83.38±	66.03±	83.64±	64.49±	71.45±
Standard deviation	8.41	2.39	5.02	2.74	5.34	3.39	5.09	2.28	2.51

As shown in Table 3, the average classification rate of this method is 71.45%, and the classification rate is relatively high, which shows that the SVM algorithm has strong generalization ability. It shows that EEG data has great reference significance for the accuracy of human-computer interaction intention recognition. The S5 classification rate of subjects with EEG data was the highest, reaching 76.85%. The reason may be that the subjects’ EEG signals were relatively stable. The classification rate of subject S3 is low, because the short contact of the EEG device electrode in the experiment is poor, which affects the accuracy of data collection and calculation. Generally speaking, the average classification effect of the left and right eyes is better than that of the upper and lower eyes, because the selected brain electrode arrangement is horizontally arranged near the frontal lobe, which is more sensitive to the inference of the direction of movement of the left and right eyes. The average classification effect of upper left and lower left is better than upper right and lower right, because the left brain is generally faster than the right brain.

After decision-level fusion of data based on DS evidence theory, the recognition accuracy can reach up to 96.03%, and the average recognition accuracy can reach 92.34%, which is higher than the recognition accuracy of only relying on eye movement and EEG data, and the variance of the data recognition accuracy is only It is 1.82, which indicates that the data fusion method based on DS theory has low sample sensitivity and strong generalization ability, which verifies that DS theory has advantages in the intention recognition of multiple physiological information. From Table 3, it can also be seen that the DS evidence theory is based The decision-level fusion method has the characteristics of high accuracy, low sensitivity to samples, and strong generalization ability. It can be used in the adaptive design of human-computer interaction systems of aircraft and weapon systems in the next step.

To verify the influence of EEG artifacts on the inferred EM direction of EEG signals, the independent component method is used to remove EEG artifacts from original EEG signals, and then the SVM method is used to classify the EEG data.

The experimental results indicate that the inference rate of test samples is lower than that of training samples, but the decline is not large. The accuracy of eye tracking data drops by 0.99%, and the accuracy of the EEG data inference drops by 0.72%. This shows that the SVM has strong generalization ability. Compared with the inference accuracy rate of EM and EEG data, the inference accuracy rate of EM data is significantly higher than that of EEG data (P < 0.005). This shows that the accuracy of EM data inferring human-computer interaction intention is higher than EEG data. After decision-level fusion of data based on D-S evidence theory, the inference accuracy rate can reach up to 96.03%, and the average inference accuracy rate can reach 92.34%. It is higher than the inference accuracy rate which relies on EM and EEG data only. The variance of the data inference accuracy rate is only 1.82, which shows that the data fusion method based on D-S theory has low sensitivity to samples and strong generalization ability. It verifies that D-S theory has advantages in the intention inference of multiple physiological information. The decision-level fusion based on D-S evidence theory has the characteristics of high accuracy, low sensitivity to samples and strong generalization ability, which can be used in the adaptive design of human-computer interaction systems of aircraft and weapon systems in the next step.

We applied 2×2 factor analysis in the tactile test. The EM signals without EEG are A1. The EM signals with EEG are A2. The feedback signals without tactile feedback are B1. The signals with tactile feedback are B2.

Result analysis:

(1) The effective duration of tasks assisted with EEG and EM signals is better than that without EEG and EM signals. (Condition 1: with EEG and EM signals; Condition 2: without EEG and EM signals)

The two-sided t-test is used to compare the mean difference of the EEG and EM signal schemes with that of no EEG and EM signals at four combinations of A and B, as shown in Fig. 9(a). The effective response time is compared by EEG and EM data as a typical comparative analysis. The level of no EEG and EM signals is relatively low.

Fig. 9

The influence of EEG, EM signals and tactile on effective task duration (a,b.c.d).

Fig. 10

Physical picture of vibration and tactile expression device (excluding mobile phones and computers).

As shown in Fig. 9(a), at four combinations of A and B, the performance of A-assisted intention inference is better than that of B-assisted, which does not use EEG or EM signals. The scheme B with EEG and EM signals is significantly better than scheme A without EEG and EM signals, because EM signals can quickly capture the direction of the subject’s sight and transmit to EEG signals, which can help the subject faster make direction judgments.

(2) The application with tactile feedback is better than that without tactile feedback. (Condition 1: with tactile feedback signals; Condition 2: without tactile feedback signals)

The two-sided t-test is used to compare the mean difference of the tactile feedback scheme with no tactile feedback scheme under two combination, as shown in Fig. 9(b). The effective response time takes the tactile feedback scheme as a typical comparative analysis, and it is obviously lower than that without the tactile feedback scheme.

As shown in Fig. 9(b), under two combination schemes, the performance of intention inference with tactile feedback scheme is better than that without tactile feedback scheme. The scheme with tactile feedback is significantly better than that without tactile feedback, because tactile feedback can help the human body respond more quickly, which is more in line with the human sensory experience.

(3) To verify that EEG, EM and tactile feedback are indeed effective for the effective duration, 6 valid subjects are re-selected to conduct a reverse experiment, that is, the subjects are first added with EEG, EM and tactile feedback signals to complete the experiment, and then the subjects are tested without EEG, EM and tactile feedback signals to complete the experiment. (Condition 1: with EEG, EM signals and tactile feedback; Condition 2: without EEG, EM signals and tactile feedback)

The two-sided t-test is used to compare the mean difference of the EEG, EM signals and tactile feedback scheme with the scheme without EEG, EM signals and tactile feedback at two combinations, as shown in Fig. 9(c). The effective response time takes EEG, EM signals and the tactile feedback scheme as a typical comparative analysis. Compared with it, it is obviously lower without EEG, EM signals and the tactile feedback scheme.

As shown in Fig.9(c), under two combination schemes, EEG, EM signals and the tactile feedback scheme assisted intention inference has significant effects on the duration of effective task. The scheme with EEG, EM signals and tactile feedback is significantly better than that without EEG, EM signals and tactile feedback, because multi-channel ergonomic feedback can improve the rapid response ability of human body.

(4) The scheme with EEG, EM signals and tactile feedback signals is better than that without EEG, EM signals and tactile feedback signals. (Condition 1: without EEG, EM signals and tactile feedback signals; Condition 2: with EEG, EM signals and without tactile feedback signals; Condition 3: with EEG, EM signals and tactile feedback signals)

The two-sided t-test is used to compare the mean difference of the EEG, EM signals and tactile feedback scheme with the scheme without EEG, EM signals and tactile feedback at two combinations, as shown in Fig. 9(c). The effective response time takes EEG, EM signals and the tactile feedback scheme as a typical comparative analysis. The scheme with EEG and EM signals is relatively low. Compared with it, it is the lowest without EEG, EM signals and the tactile feedback scheme.

As shown in Fig. 9(d), under two combination schemes, EEG, EM signals and the tactile feedback scheme assisted intention inference has significant effects on the duration of effective task. The scheme with EEG, EM signals and tactile feedback is significantly better than that with EEG, EM signals and with EEG, EM signals and tactile feedback, because multi-channel ergonomic feedback can improve the rapid response ability of human body, which is more in line with the human sensory experience..

4 WSN deployment

In this section, the sensing materials and sensing accuracy are considered. A wireless sensor network deployment case, which is suitable for enhancing ergonomic analysis of human factors, is proposed through a series of calibrated sensing hardware tools. A complete wireless sensor network time synchronization mechanism is introduced to focus on the reliability of wireless communication technology in the scene. Especially in terms of security management, the effect of human existence on the system’s propagation is evaluated. Finally, combined with actual case, an optimized wireless sensor network deployment scheme transmission model is proposed to study the form and topology of wireless sensing and signal reception.

4.1 Research status

(1) The craftsmanship is not enough in the industry.

Most companies are prevented from moving towards tactile sensors because of the strict requirements of accuracy and stability. At present, most domestic sensor companies are engaged in the production of gas and temperature sensors. Among more than 100 companies, there is almost no sensor manufacturers that can produce tactile sensors.

(2) The material is not pure enough outside the industry.

The quality of domestic materials and production levels are not stable. The production of graphene should be fine, but the technology of making sensors with graphene is not yet mature.

To obtain high-quality materials, conductive rubber can be produced by conductive paste, which can meet the standards. Conductive rubber is made from uniformly distributed conductive particles, such as glass silver-plated, aluminum-plated silver and silver, in silicone rubber. Conductive particles are squeezed to connect to each other, and then the electric current generates. The more uniform the distribution is, the more regular the relationship between current generation and pressure is.

There are 100 sensitive elements in a matrix of 10 cm×10 cm that can be distributed by the array sensor. Due to the soft substrate, the calculation of forces from different directions and the elimination of coupling interference between forces can make the elements and the distance between each other more sensitive. The shorter, the more difficult it is to accurately output. In addition, the force dimension of each sensitive element also increases the technique complexity. The force includes six dimensions (3 directions: X, Y, Z axis and the corresponding torque direction). It also requires efforts in basic research on eliminating the coupling interference between dimensions.

4.2 RBS –time synchronization mechanism of sensor network

A beacon packet is broadcasted by a sending node, and both nodes in the broadcast domain can receive this packet. Each receiving node records the time according to its own local time while the packet is received, and exchanges their receiving time of recorded packets. The difference between two receiving time is equivalent to the time difference between two receiving nodes, and one of the receiving nodes can change the local time according to the difference of time.

4.3 System hardware design

The device can receive the surrounding environment or other specific information, and convert the information into a corresponding vibration tactile stimulation spatial combination sequence according to the tactile coding scheme. It can control the vibrators distributed in different parts of the human skin with different vibration intensity and different duration on the skin. Tactile stimuli are generated from different space landing points to realize the vibration tactile expression of information, as shown in Fig 6.

From the perspective of system hardware architecture, the vibration tactile expression device is composed of five modules: computer platform client module, smart phone client module, embedded system controller module, low-power inductive load driver module and multi-path distributed vibration sensor module.

4.4 System S-R compatibility

Stimulus-Response-Compatibility (SRC) research is originated from the research of the US Air Force on the display during World War II. In the 1950 s, Fitts, an American engineering psychologist, first proposed the definition of SR compatibility: When a certain combination of stimulus and response (referred to as SR combination) can produce better and faster results, this matching of SR combination is compatible. When subject performs information processing, the consistency of received input information is called the stimulus (Stimulus) and the consistency of the processing result is the called response (Response). It can simplify the individual information processing and improve the performance of the processing to obtain faster and better processing efficiency. Since the concept of S compatibility is proposed, it has become an important principle recognized in the design of “ergonomic systems” in ergonomics and comfortable man-machine-ring system.

5 Conclusion

To improve the accuracy of a pilot’s intention recognition, a human-computer interaction intention inference method is proposed with EEG, EM and tactile feedback in this paper. The more features of EM signals, the higher the accuracy of the inference. In the experiment, EEG signals of 8 channels (AF7, F7, FT7, T7, AF8, F8, FT8 and T8) near the frontal lobe of the human brain are first collected by a computer. Then, baseline removal, normalization and least square noise are used to preprocess the data for noise reduction. In addition, SVM is used for multi-classification and 8-way recognition of EM direction is realized through the data. The experimental results show that the classification rate of the proposed method can reach 75.77%, 76.7%, 83.38%, 83.64%, 60.49%, 60.93%, 66.03, and 64.49%, respectively in the eight corresponding directions of up, down, left, right, top left, bottom left, top right and bottom right. Compared with traditional classification methods, the proposed method has higher classification accuracy and simpler realization process. Also, it further verifies the feasibility and effectiveness of EEG signals to identify EM directions for intention recognition.

The scheme with EEG, EM signals and tactile feedback is significantly better than that without EEG, EM signals and that without EEG, EM signals and tactile feedback. The reason is that the multi-channel ergonomic feedback can improve the rapid response ability of human body, which is more in line with the sensory perception of human body. It can provide theoretical basis and technical support for the adaptive design of the human-computer interaction system in the next step.

Footnotes

Acknowledgments

This work is supported by The National Natural Science Foundation of China (No. 52072293) and The National Defense Science and Technology Innovation Zone (No. ZT001007104).

References

Mitsukura

, EEG signal processing for real applications, Journal of Signal Processing 20(1) (2016), 1–7.

Lakshmi

M.R.

, Prasad

T.V.

and Prakash

D.V.C.

, Survey on EEG signal processing methods, International Journal of Advanced Research in Computer Science and Software Engineering 4(1) (2014).

Sanei

and Chambers

J.A.

, EEG signal processing, John Wiley & Sons, 2013.

Rozgić

, Vitaladevuni

S.N.

and Prasad

, 2013 IEEE international conference on acoustics, speech and signal processing, IEEE 2013, pp. 1286–1290.

Seeck

, Koessler

, Bast

, Leijten

, Michel

, Baumgartner

, He

and Beniczky

, The standardized EEG electrode array of the IFCN, Clinical Neurophysiology 128(10) (2017), 2070–2077.

Rojas

G.M.

, Alvarez

, Montoya

C.E.

, de la Iglesia-Vayá

, Cisternas

J.E.

and Gálvez

, Study of resting-state functional connectivity networks using EEG electrodes position as seed, Frontiers in Neuroscience 12((2018), 235.

Desai

, Electroencephalography (EEG) Data Collection and Processing through Machine Learning, (2014).

Huster

R.J.

, Mokom

Z.N.

, Enriquez-Geppert

and Herrmann

C.S.

, Brain–computer interfaces for EEG neurofeedback: Peculiarities and solutions, International Journal of Psychophysiology 91(1) (2014), 36–45.

Bhatti

M.H.

, Khan

M.U.G.

, Iqbal

, Aloqaily

, Jararweh

and Gupta

, Soft computing-based EEG classification by optimal feature selection and neural networks, IEEE Transactions On Industrial Informatics 15(10) (2019), 5747–5754.

10.

Guo

, Wang

, Wu

and Zhang

, Dynamic analysis of motor imagery EEG using kurtosis based independent component analysis, Advances in Cognitive Neurodynamics ICCN 2007, Springer 2008, pp. 381–385.

11.

Cover

T.M.

, Elements of information theory, John Wiley & Sons, 1999.

12.

Sundararajan

, Pons

and Sarwat

A.I.

, 2015 12th International Conference on Information Technology-New Generations, IEEE 2015, pp. 139–144.

13.

Lei

and Liao

, Understanding the influences of EEG reference: a large-scale brain network perspective, Frontiers in Neuroscience 11((2017), 205.

14.

Chella

, Pizzella

, Zappasodi

and Marzetti

, Impact of the reference choice on scalp EEG connectivity estimation, Journal of Neural Engineering 13(3) (2016), 36016.

15.

Yao

, A method to standardize a reference of scalp EEG recordings to a point at infinity, Physiological Measurement 22(4) (2001), 693.

16.

Essl

and Rappelsberger

, EEG cohererence and reference signals: experimental results and mathematical explanations, Medical and Biological Engineering and Computing 36(4) (1998), 399–406.

17.

Teplan

, Fundamentals of EEG measurement, Measurement Science Review 2(2) (2002), 1–11.

18.

Van Alste

J.A.

, Van Eck

and Herrmann

O.E.

, ECG baseline wander reduction using linear phase filters, Computers and Biomedical Research 19(5) (1986), 417–427.

19.

Huang

, Wang

and Jia

, Braille and Morse Code Based Tactile Information Coding and its Applications, International Journal of Pattern Recognition and Artificial Intelligence (2020), 2150014.

20.

Murray

, Amin

, Maciver

and Benbadis

S.R.

, EEG findings in posterior reversible encephalopathy syndrome, Clinical Eeg and Neuroscience 50(5) (2019), 366–369.

21.

Meyer

M.C.

, Scheeringa

, Webb

A.G.

, Petridou

, Kraff

and Norris

D.G.

, Adapted cabling of an EEG cap improves simultaneous measurement of EEG and fMRI at 7T, Journal of Neuroscience Methods 331 (2020), 108518.

22.

Patro

S.G.K.

, Mishra

B.K.

, Panda

S.K.

, Kumar

, Long

H.V.

, Taniar

and Priyadarshini

, A Hybrid Action-Related K-Nearest Neighbour (HAR-KNN) Approach for Recommendation Systems, IEEE Access 8 (2020), 90978–90991.

23.

Bai

, Nakamura

, Ikeda

and Shibasaki

, Nonlinear Markov process amplitude EEG model for nonlinear coupling interaction of spontaneous EEG, Ieee Transactions On Biomedical Engineering 47(9) (2000), 1141–1146.

24.

Higashi

and Tanaka

, Simultaneous design of FIR filter banks and spatial patterns for EEG signal classification, Ieee Transactions On Biomedical Engineering 60(4) (2012), 1100–1110.