Virtual reality-induced motion sickness under autonomous vehicle driving conditions: EEG-based recognition and ANOVA analysis of various driving modes

Abstract

Motion sickness induced by autonomous driving technology poses a new challenge to the emerging sustainable transportation systems. This study investigates the association between motion sickness in autonomous driving and electroencephalogram (EEG) signals under three laboratory-based simulated scenarios: manual driving, resting, and autonomous driving. EEG data were recorded from participants in each mode, alongside the collection of motion sickness symptoms through questionnaires. Data analysis and exploration were conducted to explore the relationship between autonomous driving-induced motion sickness and EEG signals. The results indicate a significantly higher probability of motion sickness among passengers in autonomous driving mode than in manual one. Across different driving modes, a correlation was observed between the amplitude and latency of N200 and P300 event-related potentials (ERPs) in the Go/Nogo paradigm, reflecting response inhibition and the occurrence of motion sickness. Temporal analysis of EEG signals revealed significant differences in the Kolmogorov complexity values at Cz, Fz, and Pz channels, suggesting the potential use of EEG-based detection of motion sickness. Frequency domain analysis indicated increased activity in alpha and gamma waves and decreased activity in beta waves following the onset of motion sickness during autonomous driving. Distinct changes were observed in the electrocortical topography of N200 and P300 components in autonomous driving through event-related potential waveforms and topographic maps. These findings provide new insights into the neural mechanisms of motion sickness in autonomous driving and offer guidance for future intervention methods and improvements in the design of autonomous driving systems, thereby promoting their sustainability and safety.

Keywords

autonomous driving virtual reality visually-induced motion sickness (VIMS)EEG characteristics response inhibition ANOVA

1. Introduction

Autonomous driving, enabled by computers, sensors, and/or lidars, allows vehicles and aircraft to perform some driving tasks in autopilot mode without requiring manual intervention, thereby improving driving efficiency and safety. However, in immersive virtual reality environments or autonomous driving navigation, SAE Level 3 automation requires drivers to regain control of the vehicle within a short time frame when it reaches its operational limits. During this transition, drivers may experience a loss of situational awareness, making it challenging to accurately perform takeover tasks.¹ When the operational information received by the driver's vestibular system does not match the visual information in the virtual reality environment, visually induced motion sickness (VIMS) is likely to occur.² VIMS is characterized by symptoms such as nausea, vomiting, eye discomfort, and disorientation, which may adversely affect operational behavior.³ These symptoms significantly impact user trust and the overall user experience with autonomous driving systems. VIMS is related to an individual's perception of movement and balance,⁴ occurring when the central nervous system receives conflicting multisensory information about movements. Changes in drivers’ roles, styles, and behavioral posture in autonomous driving can lead to motion sickness.⁵ Although the basic origin of motion sickness is generally attributed to sensory conflicts or postural instability, the relationships between the neural mechanisms involved in motion sickness have not been fully clarified.⁶

EEG signals can reflect the brain's activity state. In contrast, the occipital areas are responsible for visual input integration, and the parietal and central areas are involved in proprioceptive and vestibular input integration. Therefore, the correlation study between EEG signals and VIMS under autonomous driving conditions is important for understanding cerebral areas that allow precise and robust perception. ERP, as a special type of brain-evoked potential, reflects the neurophysiological changes in brain cognition processes.⁷ Brain rhythmic activities can oscillate between neuronal populations, and EEG signals associated with cognitive behavioral functions generally include delta (0.5–4 Hz), theta (4–8 Hz), alpha (8–13 Hz), beta (12–30 Hz), and gamma (30–80 Hz) frequency bands. Cross-frequency rhythm wave phase-amplitude coupling (PAC) mechanisms can achieve cognitive behavioral control of higher-order functional areas over lower-order functional areas.⁸ When performing cognitive-behavioral activities, the EEG signal characteristics, including ERP and PSD (power spectral density), will change accordingly.⁹ Chai et al.¹⁰ recruited volunteers to experience head-mounted virtual reality to investigate the impact of virtual reality-induced motion sickness on brain neural activity. They recorded EEG signals and extracted EEG characteristics using sample entropy and power spectral methods. The results indicated that sample entropy and power spectral analysis could serve as indices for measuring virtual reality sickness, particularly at electrode sites F8, F12, Cz, and CPz. Zhao et al.² conducted a simulated driving experiment to study the physiological effects of autonomous driving on recruited healthy participants. They provided visual and vestibular stimulation and concurrently collected and compared EEG characteristics during autonomous and manual driving modes (ADM and MDM, respectively). The findings revealed that EEG signals can quantitatively assess the severity of motion sickness, particularly at electrode sites FC2, Cz, and P3. Liu et al.¹¹ induced motion sickness symptoms using a VR vehicle-driving simulator and collected data using wearable EEG devices with four electrodes. They reported substantial individual differences in tolerance, susceptibility, and recovery from motion sickness and suggested the feasibility of assessing motion sickness using EEG devices with a small number of electrodes. Lim et al.⁶ investigated using EEG in testing and assessing recovery from VR-induced motion sickness. They found significant differences in EEG (delta, theta, and alpha) between baseline and VR-induced sickness in two regions (frontal lobe and central area). Li et al.¹² addressed the challenge of poor applicability of motion sickness identification methods based on EEG signals for multiple subjects. The study accurately identified VR-induced motion sickness when utilizing rhythm features from EEG electrodes FP1, FP2, C3, C4, P3, P4, O1, and O2 as the feature vectors.

The studies above on motion sickness based on EEG primarily focus on exploring the EEG characteristics that can be used to characterize motion sickness. While EEG can provide a rough measure of brain activity, it cannot reflect the neurophysiological changes in the brain during cognitive processes. Therefore, this study is based on virtual reality technology to investigate ERPs induced by virtual reality-induced motion sickness under autonomous driving, aiming to explore the effects of motion sickness on response inhibition, different frequency components of brain electrical activity, and the neural mechanisms of cognitive changes. This will provide a scientific basis for further optimizing autonomous driving systems and improving user experience. Additionally, it will contribute to addressing safety concerns during the transition period of a takeover and promoting the application of autonomous vehicles to meet the urgent need for more sustainable transport solutions.

2. Materials and methods

2.1. Participants

Sixteen healthy right-handed participants (age 23.1 ± 6.5) with no prior experience in autonomous driving or VR simulation and no history of gastrointestinal or cardiovascular diseases. Furthermore, participants signed informed consent forms before the experiment and were compensated with $100 after their participation. The Ethical Committee of Air Force Medical Center approved the study and the manner of Consent.

2.2. Experimental tasks

The experimental tasks involved participants completing both autonomous and manual driving mode (ADM and MDM) tasks on a simulated driving platform comprising a real vehicle, a three-dimensional visual system, and a motion simulation system.

The combination of the Go/Nogo paradigm and event-related potential (ERP) measurements to assess inhibition control has been widely used.¹³ Additionally, the Simulator Sickness Questionnaire (SSQ) subjective survey has been extensively employed in studies of visually induced motion sickness (VIMS) in VR environments, which phenomenon was coined as cybersickness in.¹⁴ Therefore, this study integrated the Go/Nogo paradigm, SSQ subjective questionnaire, and ERP analysis of brain electrical signals. The two experimental tasks were outlined, as shown in Figure 1.

Figure 1.

Experimental task and flowchart.

Task 1: Participants were required to complete a 45-min manual driving (MDM) task in a VR environment. Following the task, they were engaged in a 10-min Go/Nogo experiment while their EEG signals were recorded. Finally, they had to complete the SSQ questionnaire within 1 min.

Task 2: Participants had to perform a 45-min autonomous driving mode (ADM) task in a VR environment, during which they only needed to focus on the driving screen. After this ADM task, they participated in a 10-min Go/Nogo experiment while their EEG signals were recorded. Subsequently, they had to complete the SSQ questionnaire within 1 min.

Each participant performed these two tasks randomly over two days. Before the tasks, each participant had to rest quietly for 3 min while his/her EEG signals were recorded. EEG data were collected using a 24-electrode Ag/AgCl cap based on the international 10–10 system electrode placement (FP1, FP2, F3, F4, Fz, F7, F8, Cz, C3, C4, CPz, T7, T8, Pz, P3, P4, P7, P8, O1, O2, etc.).

2.3. Data processing

The offline analysis of the EEG data was conducted using MATLAB R2022b (The MathWorks Inc., Natick, USA) along with the EEGLAB toolbox. The preprocessing of the EEG data involved recording EEG signals at a sampling rate of 1000 Hz using a 0.1–100 Hz bandpass filter. Subsequently, the calibrated EEG data was digitally filtered using a 0.5–50 Hz bandpass filter. The EEGLAB toolbox, based on the independent component analysis (ICA), was also utilized to remove artifacts such as eye blinks and eye movements.

Subjective SSQ data were analyzed using one-way repeated measures analysis of variance (ANOVA). To analyze the ERPs, a similar one-way repeated measures ANOVA was applied along with Greenhouse-Geisser correction to account for violations of sphericity. Simple effects analysis was performed to examine interaction effects, and statistical significance was determined with Bonferroni-corrected p-values, with p < 0.05 considered significant statistical significance.

3. Results

3.1. Go/Nogo paradigm performance results

Each participant performed the prolonged driving task for no less than 45 min. Before and after the task, a Go/Nogo paradigm was conducted to establish a baseline measure before the driving task and to determine whether there were changes in inhibitory control and short-term memory capacity after completing different driving tasks. The objective performance data of the Go/Nogo task before and after the various driving tasks are listed in Table 1. The quantified objective measures of dizziness were compared with the baseline state using single-factor repeated measures ANOVA to test for differences. A Bonferroni-corrected P value of < 0.05 indicated statistically significant differences, indicating that the participants exhibited dizziness in objective performance. If the performance scores of the Go/Nogo paradigm significantly decreased, it indicated a decline in the participants’ inhibitory control ability and impairment of brain-related cognitive-behavioral functions.

Table 1.
Objective performance results of the Go/Nogo paradigm for MDM tasks.

Stage Time (min) Mean ± standard deviation F P Motion sickness (Yes/No)

Baseline 5 0.93 ± 0.02 - -

Before MDM 6 0.91 ± 0.02 2.44 0.10 No

8 0.90 ± 0.02 2.27 0.29 No

10 0.91 ± 0.03 2.27 0.54 No

13 0.89 ± 0.01 8.42 0.63 No

After MDM 6 0.90 ± 0.03 0.89 0.27 No

8 0.89 ± 0.02 10.34 0.15 No

10 0.88 ± 0.04 42.13 <0.01 Yes

13 0.87 ± 0.02 37.06 <0.01 Yes

Stage	Time (min)	Mean ± standard deviation	F	P	Motion sickness (Yes/No)
Baseline	5	0.93 ± 0.02	-	-
Before MDM	6	0.91 ± 0.02	2.44	0.10	No
8	0.90 ± 0.02	2.27	0.29	No
10	0.91 ± 0.03	2.27	0.54	No
13	0.89 ± 0.01	8.42	0.63	No
After MDM	6	0.90 ± 0.03	0.89	0.27	No
8	0.89 ± 0.02	10.34	0.15	No
10	0.88 ± 0.04	42.13	<0.01	Yes
13	0.87 ± 0.02	37.06	<0.01	Yes

According to the statistical results in Table 1, the performance scores of the Go/Nogo paradigm did not show significant differences during the before-MDM phase. During the MDM task, participants’ attention was more focused, and the duration of actively engaging in the driving task was shorter, which not only did not result in mental fatigue but also did not lead to dizziness, nausea, or other symptoms of dizziness. However, differences were observed in the performance data of the Go/Nogo paradigm after completing the MDM task, possibly due to cognitive mental fatigue induced by prolonged driving.

In Table 2, during the ADM phase, after the participants completed the 45-min task, both inhibitory control and short-term memory abilities declined, and the Go/Nogo paradigm performance scores significantly decreased. Although the participants were not engaged in driving control during the ADM process, prolonged focus on the virtual screen could lead to situations where visual, somatosensory, and vestibular signals do not correspond with a participant's anticipated experiences. Therefore, in the later stages of the task, some participants reported experiencing symptoms of dizziness. This suggests that the decline in performance scores of the Go/Nogo paradigm is an objective representation of dizziness.

Table 2.

Objective performance results of the Go/Nogo paradigm for ADM tasks.

Stage	Time (min)	Mean ± standard deviation	F	P	Motion sickness (Yes/No)
Baseline	5	0.92 ± 0.01	-	-
Before ADM	6	0.91 ± 0.03	0.87	0.28	No
	8	0.92 ± 0.01	2.32	0.44	No
	10	0.90 ± 0.02	4.55	0.42	No
	13	0.90 ± 0.01	7.32	0.51	No
After ADM	6	0.84 ± 0.03	10.36	<0.05	Yes
	8	0.82 ± 0.01	15.82	<0.05	Yes
	10	0.79 ± 0.03	33.89	<0.05	Yes
	13	0.78 ± 0.02	34.56	<0.05	Yes

The above results indicate that the Go/Nogo paradigm is instrumental in detecting the negative impacts of brain activity on cognitive control. During the Go/Nogo task, individuals must respond quickly to Go stimuli and withhold responses to Nogo stimuli. As the task duration increases, the number of correct trials, missed Go stimuli, and false alarms to Nogo stimuli all increase, indicating a detrimental impact of brain activity on response inhibition. Performance metrics from the Go/Nogo paradigm can assess response inhibition ability but cannot directly elucidate the cognitive processes related to response inhibition in humans.

3.2. Statistical analysis of SSQ scale results for participants with motion sickness

Table 3 presents the statistical analysis results of the Simulator Sickness Questionnaire (SSQ) for the participants. It can be observed that there are significant differences in the SSQ scores for nausea, oculomotor, disorientation, and total scale between assessments in the ADM and MDMs compared to the assessment in the resting mode (p < 0.05), indicating statistically significant differences in SSQ scores between different driving modes.

Table 3.
Statistical analysis results of SSQ scale.

Mode Nausea Oculomotor nerve Orientation disorder Total score F P

Resting mode 0 0 0 0 - -

MDM 5.23 ± 0.53 8.65 ± 0.27 9.64 ± 0.51 8.91 ± 0.73 5.371 0.526

ADM 98.25 ± 0.43* 75.34 ± 0.37* 124.56 ± 0.26* 109.16 ± 0.86* 18.016 0.003

Mode	Nausea	Oculomotor nerve	Orientation disorder	Total score	F	P
Resting mode	0	0	0	0	-	-
MDM	5.23 ± 0.53	8.65 ± 0.27	9.64 ± 0.51	8.91 ± 0.73	5.371	0.526
ADM	98.25 ± 0.43*	75.34 ± 0.37*	124.56 ± 0.26*	109.16 ± 0.86*	18.016	0.003

The F and p values are the results of a repeated measures analysis of variance for the factors. A p-value of less than 0.05 indicates a significant statistical difference denoted by *. This indicates a significant statistical difference between the ADM and MDM.

After Bonferroni correction, a statistically significant decrease in SSQ scores was observed in both driving modes, especially after ADM (F = 18.016, p = 0.003). This implies that both driving tasks are likely to induce VIMS in participants, with a greater likelihood of induction in the ADM.

3.3. ERP analysis of Go/Nogo paradigm in response inhibition

ERP provides temporal information on cognitive behavioral control processes, highlighting critical nodes in brain resource allocation and information processing. ERP theory suggests that cognitive resource reserves are required for every data processing stage and that the brain's central executive or working memory units control the allocation of cognitive resources. The characteristics of ERP can be further elucidated based on the time it takes for stimulus signals to be transmitted from external to internal sources. Early ERPs that peak within 100 ms are predominantly exogenous, largely dependent on the physical parameters of the stimuli, while later occurring ERPs are endogenous and closely associated with the brain's evaluative processing of the stimuli. For example, N100 can reflect attention to sensory information and measure attentional inhibition,¹⁵ while N200 can detect conflict and discrimination processes during decision-making, reflecting general executive and cognitive control functions.³ P200 is related to higher cognitive processes, such as attention and memory, and is considered to reflect the conscious processing of stimuli.¹⁶ Besides, P300 can reflect cognitive evaluation and decision-making processes for stimulus information.¹⁷

ERP can be used to differentiate various functional stages of brain cognitive-behavioral processes. If the brain's allocation and control pathways of cognitive resources are hindered in a motion sickness state, corresponding abnormalities may appear in the ERP. Motion sickness is a reaction inhibition caused by sensory conflicts between the visual and vestibular systems. In contrast, the vestibular system is indispensable in maintaining body balance and directional control.¹⁸ Analyzing ERP features can help clarify the mechanisms of cognitive behavioral function impairments. Therefore, selecting four ERP components (such as N100, N200, P200, and P300) to study the neural mechanisms of cognitive changes in motion sickness is essential.

Table 4 lists ERP components’ amplitudes in the Go/Nogo task under three modes. As shown in Table 4, compared with the resting mode, participants in the MDM exhibit significant correlations in the ERP components N200 for error-Nogo (F = 75.36, p = 0.041), error-Go (F = 89.47, p = 0.039), correct-Nogo (F = 96.21, p = 0.031), and correct-Go (F = 85.34, p = 0.038). Similarly, in the MDM, the participants demonstrate significant correlations in the ERP component P300 for error-Nogo (F = 82.19, p = 0.032), error-Go (F = 91.55, p = 0.037), correct-Nogo (F = 94.56, p = 0.043), and correct-Go (F = 91.72, p = 0.039) compared to the resting mode.

Table 4.
Amplitudes of ERP components in the Go/Nogo task after different modes.

Error-Nogo Error -Go Correct -Nogo Correct -Go

Mode ERP Amplitude (μV) p Amplitude (μV) p Amplitude (μV) p Amplitude (μV) p

Resting mode N100 −3.23 ± 0.27 - −3.18 ± 0.33 - −3.19 ± 0.64 - −3.18 ± 0.82 -

N200 −3.56 ± 0.14 - −3.87 ± 0.37 - −3.67 ± 0.25 - −3.63 ± 0.49 -

P200 2.36 ± 0.83 - 2.68 ± 0.54 - 2.87 ± 0.44 - 3.01 ± 0.42 -

P300 4.89 ± 0.53 - 5.16 ± 0.42 - 4.75 ± 0.63 - 5.28 ± 0.42 -

MDM N100 −3.16 ± 0.31 0.163 −3.15 ± 0.65 0.824 −3.12 ± 0.53 0.541 −3.09 ± 0.64 0.129

N200 −3.67 ± 0.25 0.041 −3.91 ± 0.24 0.039 −3.84 ± 0.46 0.031 −3.68 ± 0.21 0.038

P200 2.45 ± 0.49 0.832 2.72 ± 0.63 0.681 2.99 ± 0.56 0.853 2.89 ± 0.72 0.786

P300 4.16 ± 0.17 0.032 4.35 ± 0.18 0.037 3.95 ± 0.26 0.043 4.82 ± 0.27 0.039

ADM N100 −3.18 ± 0.25 0.234 −3.09 ± 0.46 0.541 −3.08 ± 0.15 0.439 −3.03 ± 0.57 0.762

N200 −3.79 ± 0.16 0.029 −3.98 ± 0.53 0.025 −3.89 ± 0.27 0.028 −3.89 ± 0.37 0.032

P200 2.38 ± 0.21 0.803 2.59 ± 0.48 0.805 2.92 ± 0.36 0.762 2.99 ± 0.53 0.841

P300 4.12 ± 0.26 0.025 4.13 ± 0.25 0.031 3.76 ± 0.48 0.027 4.69 ± 0.28 0.016

		Error-Nogo	Error -Go	Correct -Nogo	Correct -Go
Resting mode	N100	−3.23 ± 0.27	-	−3.18 ± 0.33	-	−3.19 ± 0.64	-	−3.18 ± 0.82	-
N200	−3.56 ± 0.14	-	−3.87 ± 0.37	-	−3.67 ± 0.25	-	−3.63 ± 0.49	-
P200	2.36 ± 0.83	-	2.68 ± 0.54	-	2.87 ± 0.44	-	3.01 ± 0.42	-
P300	4.89 ± 0.53	-	5.16 ± 0.42	-	4.75 ± 0.63	-	5.28 ± 0.42	-
MDM	N100	−3.16 ± 0.31	0.163	−3.15 ± 0.65	0.824	−3.12 ± 0.53	0.541	−3.09 ± 0.64	0.129
N200	−3.67 ± 0.25	0.041	−3.91 ± 0.24	0.039	−3.84 ± 0.46	0.031	−3.68 ± 0.21	0.038
P200	2.45 ± 0.49	0.832	2.72 ± 0.63	0.681	2.99 ± 0.56	0.853	2.89 ± 0.72	0.786
P300	4.16 ± 0.17	0.032	4.35 ± 0.18	0.037	3.95 ± 0.26	0.043	4.82 ± 0.27	0.039
ADM	N100	−3.18 ± 0.25	0.234	−3.09 ± 0.46	0.541	−3.08 ± 0.15	0.439	−3.03 ± 0.57	0.762
N200	−3.79 ± 0.16	0.029	−3.98 ± 0.53	0.025	−3.89 ± 0.27	0.028	−3.89 ± 0.37	0.032
P200	2.38 ± 0.21	0.803	2.59 ± 0.48	0.805	2.92 ± 0.36	0.762	2.99 ± 0.53	0.841
P300	4.12 ± 0.26	0.025	4.13 ± 0.25	0.031	3.76 ± 0.48	0.027	4.69 ± 0.28	0.016

Furthermore, when comparing with the resting mode, participants in the MDM also show significant correlations in the ERP components N200 for error-Nogo (F = 79.26, p = 0.029), error-Go (F = 72.31, p = 0.025), correct-Nogo (F = 73.22, p = 0.028), and correct-Go (F = 83.56, p = 0.032). Additionally, in the MDM, the participants displayed significant correlations in the ERP component P300 for error-Nogo (F = 71.39, p = 0.024), error-Go (F = 76.25, p = 0.031), correct-Nogo (F = 72.81, p = 0.027), and correct-Go (F = 70.65, p = 0.016) compared to the resting mode.

The changes in the N100 amplitude under the Go/Nogo paradigm in both active and MDMs did not exhibit significant differences (p > 0.05). Similarly, the amplitude changes in the P200 also did not show significant differences (p > 0.05). These results indicate that the subjects’ attention in different driving modes did not lead to wide-ranging changes in the N100 amplitude, typically associated with auditory stimulus-induced arousal, spatial attention, and perception. Likewise, minimal variations were observed in higher cognitive processes such as attention and memory among subjects in different driving modes, as indicated by the insignificant changes in P200 amplitude.

The above results prove that VIMS under ADM and MDM leads to changes in the N200 and P300 amplitudes in the Go/Nogo paradigm ERP, indicating the involvement of neural oscillatory mechanisms related to response inhibition functions. The amplitude changes can characterize the obstructed cognitive behavior and operational pathways in individuals experiencing motion sickness, indicating the effectiveness of the dual-choice task in inducing and measuring response inhibition. Therefore, further time-domain, frequency-domain analysis, and ERP analysis can be conducted using electrode locations of Cz, Fz, and Pz.

3.4. Time-Domain analysis of visual evoked complexity

The EEG data from three channels, Cz, Fz, and Pz, were divided using non-overlapping sliding windows for each subject. The Kolmogorov complexity (KC) was calculated by converting the EEG data into binary strings and computing their length. Figure 2 illustrates the significant changes in KC under the Go/Nogo paradigm across different driving modes.

Figure 2.

Scatterplot of the standard deviation of KC for different electrodes.

In the Go/Nogo paradigm, the significance of KC varies under different driving modes. Specifically, there are significant differences in the average KC values at Cz, Fz, and Pz between the non-VIMS (visual immediate memory scan) and VIMS states. Furthermore, with the onset of VIMS, the average KC values exhibit a significant decrease (p < 0.05).

Under ADM, the data points are positioned above the diagonal line, particularly for the Fz and Pz channels. This indicates that when the driving task induces VIMS, the standard deviation of KC decreases. This may suggest that the driver's brain activity is more stable under autonomous driving conditions and that the complexity of information is lower.

By comparing the complexity values of KC under different driving modes and conducting paired t-tests on the complexity values before and after inducing VIMS, as shown in Table 5.

Table 5.

Paired t-test of KC between non-VIMS and VIMS states.

Channel	Cz	Fz	Pz
Sig	0.002	0.001	0.001

Results indicate that the paired t-test significance values for the KC complexity of the EEG signals in the three channels are 0.002, 0.001, and 0.001, respectively, demonstrating significant differences in KC values across different channels before and after the onset of motion sickness. This suggests that KC complexity in the Cz, Fz, and Pz channels can be used to quantify the complexity of EEG signals. This indicates that brain activity experiences significant inhibition when VIMS occurs, leading to a systematic decrease in the temporal signal. Therefore, in research on virtual reality-induced autonomous driving motion sickness, sensitive and quantifiable KC complexity can be utilized to detect motion sickness occurrence rapidly.

3.5. Frequency domain analysis of LFP power Spectrum

The power of EEG rhythm waves is closely related to cognitive behavioral functions. In the Go/Nogo paradigm, the significance of alpha, beta, and gamma waves varies under different driving modes. Suppressing alpha waves (8–12 Hz) may affect the driver's attention and reaction capabilities, while inhibiting beta waves (13–30 Hz) may impact the driver's decision-making and response abilities. Similarly, suppressing gamma waves (30–100 Hz) may influence the driver's higher cognitive processes.

An analysis was conducted in Figure 3 on the changes in power spectra at the Pz, Cz, and Fz channels and the correlation with the significance of alpha, beta, and gamma waves in ADM and MDMs. Neuronal responses to cognitive tasks exhibit similar oscillation power characteristics in the alpha and theta frequency bands. In the ADM, increased suppression of alpha waves indicates decreased attention or weakened driver response capabilities. Additionally, increased suppression of gamma waves suggests an influence on the driver's higher cognitive abilities, such as planning and judgment. Increased rhythm wave power reflects enhanced neuronal synchrony, indicating reinforced internal neural activity to counteract the trend of motion sickness. Moreover, increased suppression of beta waves implies reduced driver decision-making ability or response speed.

Figure 3.

Power spectrum of LFPs from different electrodes.

In the ADM, the vehicle's autonomous driving system assumes some attention and response tasks, resulting in less pronounced suppression of alpha waves than the MDM. As the vehicle makes decisions and responds automatically, the inhibition of beta waves is also less evident. Since the MDM system primarily handles these higher cognitive processes, suppressing gamma waves is similarly less pronounced.

Following the onset of motion sickness, an increase in alpha and gamma wave activity and a decrease in beta wave activity were observed. This suggests impairment of top-down cognitive-behavioral functions in the brain during motion sickness, leading to reduced efficiency in performing high-load tasks and potentially resulting in behaviors such as closing one's eyes, eye discomfort, or directional disorders, either intentionally or unintentionally. These changes indicate that individuals experiencing motion sickness may be relaxed but inadequately alert, potentially impacting higher cognitive processes.

Therefore, in research on virtual reality-induced motion sickness in ADM, the detection of alpha, beta, and gamma waves can be utilized to assess the presence of motion sickness.

3.6. ERP topographic map analysis of response inhibition

Based on the ERP analysis of the response inhibition Go/Nogo paradigm, participants’ motion sickness is most pronounced in the ADM. Figure 4 illustrates the topographic maps of the brain changes for the Go/Nogo paradigm.

Figure 4.

The grand average waves of ERP under different stimulus types and driving modes.

From Figure 4, it can be seen that time significantly affects the amplitude of N200 and P300 (F = 85.71, p = 0.043). The N200 amplitude in ADM is smaller than in MDM, while the P300 amplitude shows the opposite pattern. This indicates that when the N200 is triggered in tasks requiring response inhibition, the functional interpretation of this component relates to response control, with the N200 amplitude positively correlated with the success rate of inhibition. Meanwhile, the P300 amplitude reflects the allocation of resources for working memory representation and response selection. There is a significant main effect of the Fz electrode for N200 (F = 82.71, p = 0.031) and a significant main effect of the Pz electrode for P300 (F = 78.24, p = 0.028), but no significant effect is observed at Cz. In the main effect for different types of Go/Nogo paradigm stimuli, the main effect of stimulus type for N200 is insignificant (p > 0.05). Still, the interaction of stimulus type and time is significant (F = 72.34, p = 0.027), while for P300, the main effect of stimulus type and its interaction with time are significant (F = 71.25, p = 0.013).

For the latency of N200 and P300, there is a main effect of time for both (F = 87.71, p = 0.038), and the latency in ADM is shorter than in MDM. The main effect of the electrode for N200 and P300 latencies is insignificant (p > 0.05). Still, the main effect of stimulus type is significant (F = 85.43, p = 0.036), with shorter latencies for deviant stimuli than standard ones. Moreover, there is a significant interaction between stimulus type and time for N200 (F = 71.62, p = 0.021), and compared to MDM, only the latency of the standard stimulus is significantly reduced in autonomous driving. However, there is no significant difference in the interaction effect of P300 (p > 0.05).

Combining the topographic maps of the brainwaves (Figure 5), it is evident that the latency and reaction time of N200 are modulated simultaneously under task-switching conditions. Combining this with Figure 2, it can be seen that N200 in the Go state represents both the completion of stimulus classification and the preparation and execution of the P300 in the Go state, indicating a more sensitive change in the topographic map of P300 in Figure 2. Considering attentional orientation and shifting issues, the delayed latency of P300 may be due to the N200-caused delay.

Figure 5.

Topographic maps for different driving modes.

It can be seen that both virtual MDM and ADM can cause motion sickness. Especially compared with MDM, the changes in N200 and P300 electroencephalogram topography in ADM are significant, indicating a heavy load on response inhibition tasks. N200 can serve as a typical index for monitoring response conflict and can also be used as a typical index for VIMS caused by attention impairment.¹⁹ The maximum N200 wave amplitude is distributed in the central-frontal area in Fz, indicating a close relationship between the central-frontal area and conflict detection. This confirms that N200 can be used as an indicator for monitoring response conflict in response inhibition tasks.

The above results indicate that response inhibition is impaired in ADM compared to MDM, suggesting that the latter makes participants more susceptible to motion sickness. This reveals that the early-stage ability to detect abnormal stimuli based on VR visual and operational processing is relatively weak, which may lead to a decline in behavioral performance. It also indicates that VR environments cause neurofunctional changes when adapting to new surroundings.

4. Discussion

In this study, based on VR vehicle driving, both subjective and objective methods were employed to analyze motion sickness induced by various driving tasks. From the perspective of EEG characteristics, ERPs were used to investigate the neural mechanisms underlying cognitive changes associated with motion sickness. The research findings are obtained as follows:

Firstly, VR-based vehicle driving can induce motion sickness in participants, and significant amplitude changes of N200 and P300 were observed in participants during driving tasks, which were more obvious in ADM than in MDM. In addition, N200 and P300 waveforms and EEG topographic maps based on ERPs can characterize cognitive behavioral changes during motion sickness. Previous studies proved that N200 can detect conflict and discrimination processes during decision-making, reflecting general executive and cognitive control functions. In contrast, P300 can reflect cognitive evaluation and decision-making processes for stimulus information. These results indicated that detecting conflict and discrimination and decision-making processes are impaired in both ADM and MDM, and the former makes participants more susceptible to motion sickness.

Secondly, research based on LFP power spectra at different electrodes indicates that the suppression of alpha and beta waves in MDM may be more pronounced due to the driver's increased attention and response tasks. However, suppressing these waveforms may need to be more evident in MDM, where the vehicle automates these tasks. Additionally, suppressing gamma waves may be less evident since the system mainly handles advanced cognitive processes in ADM. Following the onset of motion sickness, there is an increase in alpha and gamma wave activities, while beta wave activity may decrease. This change suggests that motion sickness patients may be relaxed yet inadequately alert, potentially impacting advanced cognitive processes. These analyses provide insights into the brain's electrical activity under different driving conditions, aiding in designing and optimizing driving environments.

5. Conclusion

Given the expansion of Waymo self-driving car project in US and Ilon Mask's challenging plans to furnish each Tesla car with self-driving software enabling autonomous driving systems in 2025, the ongoing research on related cybersickness and VIMS in new users of such systems should be intensified to avoid driving safety risks, casualties, and material losses. Compared with previous studies, this study provides a new perspective for understanding the neural mechanism of motion sickness by analyzing ERPs. At the same time, the combination of virtual reality technology and autopilot motion sickness provides a new experimental paradigm for the study of autopilot motion sickness. In addition, this study combined a subjective questionnaire, ERP analysis, and KC complexity calculation to provide a multi-dimensional perspective for assessing motion sickness. It explored the impact of autopilot motion sickness on response inhibition of brain electrical activity, providing a scientific basis for understanding the impact of motion sickness on cognitive function.

However, some limitations remain in recognizing virtual reality-induced motion sickness in autopilot. First, experimental tasks cannot fully simulate the complexity of the real world, affecting the induction of motion sickness and the recording of electrical signals. Second, the experiment is too short, may not capture the long-term effects of motion sickness, and mainly relies on EEG and subjective questionnaires, lacking a comprehensive evaluation of other physiological indicators. In addition, there are still some limitations in the number, age and gender of samples selected. Future research should further integrate multidisciplinary knowledge to fully understand the complex mechanisms of autonomous driving-induced motion sickness and conduct longer experimental and field studies to improve the real-world applicability of the findings. At the same time, various physiological measurement tools were combined to obtain a more comprehensive assessment of motion sickness and consider the impact of individual differences (age range, gender and so on) on motion sickness.

Footnotes

Informed consent statement

Not applicable

Institutional review board statement

Not applicable.

Author contributions

Writing—original draft preparation, SY S.; Software, Y Z.; Methodology—review and editing, HJ W.; Validation, XL F. All co-authors have read and agreed to the published version of the manuscript.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was funded by the National Natural Science Foundation of China (82102176) and the R&D Program of the Beijing Municipal Education Commission (KM202210037001).

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Data availability

All data included in this study are available upon request.

References

Zhou

Tilbury

, et al. Behavioral and physiological responses to takeovers in different scenarios during conditionally automated driving. Trans Res Part F Traffic Psychol Behav 2024; 101: 320–331.

Zhao

, et al. EEG Characteristics of motion sickness subjects in MDM based on virtual reality tests. J Tsinghua Univ 2019; 12: 993–998.

Zhou

, et al. Inhibition-related N2 and P3: indicators of visually induced motion sickness (VIMS). Int J Ind Ergon 2020; 78: 993–998.

Smart

Drew

Hadidon

, et al. Using nonlinear kinematic parameters as a means of predicting motion sickness in real-time in virtual environments. Hum Factors 2023; 65: 1830–1840.

Henry

Bougard

Bourdin

, et al. Changes in electroencephalography activity of sensory areas linked to car sickness in real driving conditions. Front Hum Neurosci 2021; 15: 809714.

Lim

Woo

, et al. Test-retest reliability of the virtual reality sickness evaluation using electroencephalography (EEG). Neurosci Lett 2020; 743: 135589.

Wang

Liao

P-C

. Personality assessment based on electroencephalography signals during hazard recognition. Sustainability-Basel 2023; 15: 8906.

Riddle

Alexander

Schiller

, et al. Reward-based decision-making engages distinct modes of cross-frequency coupling. Cereb Cortex 2021; 32: 2079–2094.

Chikara

Yeh

, et al. Neural dynamics of target detection via wireless EEG in embodied cognition. Sensors-Basel 2021; 21: 5213.

10.

Chai

Hua

Zhou

. Analysis of motion sickness induced by VR based on sample entropy and power spectrum of EEG. Elec Meas Tech 2022; 45: 43–52.

11.

Liu

Zhang

, et al. A pilot study on electroencephalogram-based evaluation of visually induced motion sickness. J Imaging Sci Techn 2020; 64: 20501–1.

12.

Zhu

, et al. VR Motion sickness recognition by using EEG rhythm energy ratio based on wavelet packet transform. Comput Meth Prog Bio 2019; 188: 105266.

13.

Piispala

Kallio

Bloigu

, et al. Delayed N2 response in Go condition in a visual Go/Nogo ERP study in children who stutter. J Fluency Disord 2016; 48: 16–26.

14.

Jang

K-M

Kwon

Nam

, et al. Estimating objective (EEG) and subjective (SSQ) cybersickness in people with susceptibility to motion sickness. Appl Ergon 2022; 102: 103731.

15.

Annanmaki

Palmu

Murros

, et al. Altered N100-potential associates with working memory impairment in Parkinson's disease. J Neural Transm 2017; 124: 1197–1203.

16.

Clark

Bontemps

Batky

, et al. Psychopathy and neurodynamic brain functioning: a review of EEG research. Neurosci Biobehav Res 2019; 103: 352–373.

17.

Han

Zhang

Yang

. The influence of cognitive style on the framing effect: an event-related potential study. Psychol Sci 2014; 37: 549–554.

18.

Liu

Cui

Zhao

, et al. VIMSNet: an effective network for visually induced motion sickness detection. Signal Image Video Process 2022; 16: 2029–2036.

19.

Wei

Zheng

Richard

. Allocating less attention to central vision during vection is correlated with less motion sickness. Ergonomics 2018; 61(7): 933–946.

		Error-Nogo		Error -Go		Correct -Nogo		Correct -Go
Mode	ERP	Amplitude (μV)	p	Amplitude (μV)	p	Amplitude (μV)	p	Amplitude (μV)	p
Resting mode	N100	−3.23 ± 0.27	-	−3.18 ± 0.33	-	−3.19 ± 0.64	-	−3.18 ± 0.82	-
	N200	−3.56 ± 0.14	-	−3.87 ± 0.37	-	−3.67 ± 0.25	-	−3.63 ± 0.49	-
	P200	2.36 ± 0.83	-	2.68 ± 0.54	-	2.87 ± 0.44	-	3.01 ± 0.42	-
	P300	4.89 ± 0.53	-	5.16 ± 0.42	-	4.75 ± 0.63	-	5.28 ± 0.42	-
MDM	N100	−3.16 ± 0.31	0.163	−3.15 ± 0.65	0.824	−3.12 ± 0.53	0.541	−3.09 ± 0.64	0.129
	N200	−3.67 ± 0.25	0.041	−3.91 ± 0.24	0.039	−3.84 ± 0.46	0.031	−3.68 ± 0.21	0.038
	P200	2.45 ± 0.49	0.832	2.72 ± 0.63	0.681	2.99 ± 0.56	0.853	2.89 ± 0.72	0.786
	P300	4.16 ± 0.17	0.032	4.35 ± 0.18	0.037	3.95 ± 0.26	0.043	4.82 ± 0.27	0.039
ADM	N100	−3.18 ± 0.25	0.234	−3.09 ± 0.46	0.541	−3.08 ± 0.15	0.439	−3.03 ± 0.57	0.762
	N200	−3.79 ± 0.16	0.029	−3.98 ± 0.53	0.025	−3.89 ± 0.27	0.028	−3.89 ± 0.37	0.032
	P200	2.38 ± 0.21	0.803	2.59 ± 0.48	0.805	2.92 ± 0.36	0.762	2.99 ± 0.53	0.841
	P300	4.12 ± 0.26	0.025	4.13 ± 0.25	0.031	3.76 ± 0.48	0.027	4.69 ± 0.28	0.016

Virtual reality-induced motion sickness under autonomous vehicle driving conditions: EEG-based recognition and ANOVA analysis of various driving modes

Abstract

Keywords

1. Introduction

2. Materials and methods

2.1. Participants

2.2. Experimental tasks

3. Results

3.1. Go/Nogo paradigm performance results

Table 3. Statistical analysis results of SSQ scale. Mode Nausea Oculomotor nerve Orientation disorder Total score F P Resting mode 0 0 0 0 - - MDM 5.23 ± 0.53 8.65 ± 0.27 9.64 ± 0.51 8.91 ± 0.73 5.371 0.526 ADM 98.25 ± 0.43* 75.34 ± 0.37* 124.56 ± 0.26* 109.16 ± 0.86* 18.016 0.003

5. Conclusion

Footnotes

Informed consent statement

Institutional review board statement

Author contributions

Funding

Declaration of conflicting interests

Data availability

References

Table 3.
Statistical analysis results of SSQ scale.

Mode Nausea Oculomotor nerve Orientation disorder Total score F P

Resting mode 0 0 0 0 - -

MDM 5.23 ± 0.53 8.65 ± 0.27 9.64 ± 0.51 8.91 ± 0.73 5.371 0.526

ADM 98.25 ± 0.43* 75.34 ± 0.37* 124.56 ± 0.26* 109.16 ± 0.86* 18.016 0.003