Can We Study Autonomous Driving Comfort in Moving-Base Driving Simulators? A Validation Study

Abstract

Objective:

To lay the basis of studying autonomous driving comfort using driving simulators, we assessed the behavioral validity of two moving-base simulator configurations by contrasting them with a test-track setting.

Background:

With increasing level of automation, driving comfort becomes increasingly important. Simulators provide a safe environment to study perceived comfort in autonomous driving. To date, however, no studies were conducted in relation to comfort in autonomous driving to determine the extent to which results from simulator studies can be transferred to on-road driving conditions.

Method:

Participants (N = 72) experienced six differently parameterized lane-change and deceleration maneuvers and subsequently rated the comfort of each scenario. One group of participants experienced the maneuvers on a test-track setting, whereas two other groups experienced them in one of two moving-base simulator configurations.

Results:

We could demonstrate relative and absolute validity for one of the two simulator configurations. Subsequent analyses revealed that the validity of the simulator highly depends on the parameterization of the motion system.

Conclusion:

Moving-base simulation can be a useful research tool to study driving comfort in autonomous vehicles. However, our results point at a preference for subunity scaling factors for both lateral and longitudinal motion cues, which might be explained by an underestimation of speed in virtual environments.

Application:

In line with previous studies, we recommend lateral- and longitudinal-motion scaling factors of approximately 50% to 60% in order to obtain valid results for both active and passive driving tasks.

Keywords

vehicle automation autonomous driving multisensory integration immersive environments virtual environments trust in automation vehicle design usability/acceptance measurement and research driving simulation behavioral validity visual-vestibular integration

Introduction

In the past decade, autonomous cars have become a rapidly evolving technology that is on current agendas of a plethora of universities, car manufacturers, and other technology and software corporations. The trend of autonomous driving does not only revolutionize the automobile industry, but it fundamentally changes the driving task itself. SAE International (2014) defines six stages of driving automation ranging from Level 0 (no automation) to Level 5 (full automation). As the automation level rises, the driving task changes considerably. Whereas the human driver is in full command at Level 0 and has to perform all tasks associated with driving, more and more aspects of the driving task are automated in the intermediate levels, changing the human’s task from active driving to monitoring the automation and ultimately even removing this responsibility as well.

As automation increases, the question of how drivers prefer to be driven becomes more and more important. For instance, at first glance, traveling in the center of a lane might be a reasonable lane-keeping behavior from a technical point of view. From a psychological perspective, however, driving in the center of a lane is not necessarily the trajectory a driver would prefer, especially when driving in bends or when turning maneuvers are being considered, in which curves are usually cut on the inner side in order to keep lateral accelerations relatively low (e.g., Boer, 1996; Siegler, Reymond, Kemeny, & Berthoz, 2001). Recent surveys revealed that passengers are very hesitant in their acceptance of automated transport systems (MacSween-George, 2003; Schoettle & Sivak, 2014). Therefore, it is even more important to investigate comfort experienced in highly or fully automated driving.

The concept of comfort is complex. A common scientific definition has yet to be established. According to de Looze, Kuijt-Evers, and van Dieën (2003), all definitions of comfort to date share three aspects: (a) comfort is subjective and may vary between persons, (b) experiencing comfort is always a reaction, and (c) comfort can be influenced by internal factors, for example, sensitivity, as well as external factors, for example physical influences. One major point in the discussion on comfort as a construct is whether it should be seen as a dimension with comfort on one end and discomfort on the other (e.g., Vergara & Page, 2000) or whether discomfort is something that is evoked by biomechanical circumstances and can coexist with comfort, which is associated with well-being (Zhang, Helander, & Drury, 1996). This controversy contributes to the existing difficulties in ascertaining a widely acknowledged method of assessment.

Ellinghaus and Schlag (2001) as well as Krist (1994) specifically point at the role of drivers’ expectation in experiencing comfort. According to them, expectancies of maneuvers and actually experienced maneuvers are constantly compared, eliciting a feeling of comfort if expectancies are met. Deviations between expected and actual driving behavior not only might increase symptoms of motion sickness (Reason, 1978; Sivak & Schoettle, 2015) but might considerably threaten user acceptance and user comfort. Sivak and Schoettle (2015) have further found that loss of control and the inability to foresee vehicle actions are identified as major points when rating riding comfort as a passenger. This link between expectancies and actual experience has influenced the described study, as becomes apparent in the Method section. With the intention of making a first step toward identifying preferred automated driving, this study compares different variations of an automated lane-change maneuver as well as an automated deceleration maneuver behind a slower lead vehicle regarding experienced driving comfort. These two maneuvers were chosen not only because they are among the most common maneuvers in highway settings (see Bellem, Schönenberg, Krems, & Schrauf, 2016), where higher levels of automation will be introduced first, but also because they represent a longitudinal maneuver or a lateral maneuver and both involve another traffic member.

To facilitate the study of driving comfort in automated vehicles, driving simulators not only provide a safe and cost-efficient alternative to technically demanding and cost-intensive on-road studies, but they also enable developers to test prototype systems. However, valuable insights obtained from a driving simulator are of particular use only when they can be transferred to the real world. In literature, the distinction between physical validity and behavioral validity was established in order to assess the validity of driving simulators (e.g., Blana, 1996; Mullen, Charlton, Devlin, & Bédard, 2011). Physical validity refers to the extent to which a driving simulator is capable of reproducing physical reality. For instance, a simulator demonstrates physical validity if physical components, such as layout, dynamic characteristics, or visual displays, correspond to on-road driving. Behavioral validity, on the other hand, refers to the behavioral correspondence between driving behavior in the simulator and that on real roads. Thus, it describes to what extent the behavior, performance, and experience of drivers in a simulator match those on a real road.

Blaauw (1982) subdivided behavioral validity into absolute validity and relative validity. Absolute validity is established when dependent variables, such as driving parameters, psychophysiological measures, or subjective evaluations, take on the same numerical values in a driving simulator as in a real setting. Relative validity is given when differences of the dependent variable between conditions are of the same order, direction, and magnitude (Blaauw, 1982; Godley, Triggs, & Fildes, 2002; Mullen et al., 2011; Wang et al., 2010). For most research questions, establishing relative validity is sufficient (Reed & Green, 1999; Törnros, 1998). Absolute validity, however, is required when determining absolute numerical values, such as general takeover request times, or when intervention thresholds of an advanced driver assistance system need to be identified (Gemou, 2013).

The reservation must be made, however, that behavioral validity of a driving simulator is always limited to a specifically defined research question or driving task (Allen et al., 1991; Mullen et al., 2011; Godley et al., 2002; Nilsson, 1993) and that due to the many different architectures, algorithms, and parameterizations of driving simulators employed in driving simulator research (Fischer, Eriksson, & Oeltze, 2012; Greenberg & Blommer, 2011; Nilsson, 1993), generalizations across studies and simulators concerning the effect of motion cuing can be made only with caution.

To lay the basis for studying driving comfort in autonomous vehicles using driving simulators, we determined the behavioral validity of two simulator configurations of a high-fidelity hexapod moving-base simulator on a linear rail system. The simulator configurations differed in the vehicle’s being placed in the hexapod dome either in parallel to the rail system (MBS_Long) or in transverse (MBS_Lat; see Method section for details).

Even though physical and behavioral validity are often assumed or found to be positively related (e.g., Ba, Zhang, & Salvendy, 2014; Blana, 1996; Klüver, Herrigel, Heinrich, Schöner, & Hecht 2016; Lee et al., 2013; but see Grabe, Pretto, Giordano, & Bülthoff, 2010; Reed & Green, 1999), there is growing evidence that not a scaling factor of 100%, but subunity scaling factors of around 50% to 70% are preferred in slalom task driving (Berthoz et al., 2013; Pretto, Nusseck, Teufel, & Bülthoff, 2009) or in lane-change maneuvers (Greenberg, Artz, & Cathey, 2003). However, the aforementioned studies did not compare simulators results with results from an on-road study but were based on ratings of perceived realism or driving performance data. In this study, we compared two scaling parameter sets, differing regarding their scaling factors of longitudinal and lateral acceleration, to a test-track setting. In the first parameter set (MBS_Lat), lateral acceleration is simulated in the exact same manner as it was recorded in the test-track study (a scaling factor of 100%), whereas longitudinal acceleration was scaled down to 17%. In the second parameter set (MBS_Long), lateral acceleration was scaled down to 50% and longitudinal acceleration to 60%. Thus, the MBS_Lat has higher physical validity in lane-change maneuvers, whereas the MBS_Long has higher physical validity in deceleration maneuvers (for detailed description of the simulator configurations, see Method section). Ours is the first study that validated two scaling parameter sets against on-road data. Furthermore, our study is the first that validated passive driving rather than active driving.

The purpose of this study is to lay the foundation for studying driving comfort in autonomous vehicles using an advanced moving-base driving simulator. In this study, we (a) evaluate whether the moving-base driving simulator is generally capable to adequately study experienced driving comfort in autonomous vehicles by comparing results from the simulator with results from a test-track study, (b) compare two motion scaling parameter sets, and (c) compare six variations of each a lateral and a longitudinal automated maneuver regarding experienced driving comfort, allowing to draw inferences on the underlying parameters. Results concerning the validity of the driving simulator will influence the setting of future studies, whereas insights from comfort ratings will be used to further improve automated driving toward a comfortable driving experience for passive drivers.

Method

Participants

A total of 52 men and 20 women (age across groups, M = 41.03 years, SD = 11.38 years, range = 21–61 years; data per setting is reported in Table 1) participated in this experiment. Both employees and non-employees of Daimler AG took part in this study and were distributed evenly across groups. All participants held a valid driver’s license, had normal or corrected-to-normal vision, were naive to the purpose of the experiment, and had little or no previous simulator experience. Data were collected anonymously. Informed consent was obtained after the task had been explained. Participants were free to terminate participation in the experiment at any time without any consequences. Participants received €30 for their participation. All experimental procedures were conducted in accordance with the ethical guidelines of the Declaration of Helsinki.

Table 1:

Demographic Data per Setting of Groups and ANOVA or Pearson’s Chi-Square Results Comparing the Groups

Variable	Test Track (n = 28)	Simulator		df	F or χ²	p
Variable	Test Track (n = 28)	Longitudinal (n = 22)	Lateral (n = 22)	df	F or χ²	p
Mean age in years^a	41.32 (10.51)	39.05 (12.82)	42.50 (11.16)	2, 69	F = 0.56	.576
Sex				2	χ² = 0.01	.992
Male	20	16	16
Female	8	6	6
Experience Level 2 automation at least weekly	4	3	3	2	F = 0.01	.997
No experience with driving automation	11	10	6	2	F = 1.61	.446

Standard deviations shown in parentheses.

Design

We chose two classes of common maneuvers for this study, including a lane change to the left (producing large lateral acceleration forces) and a deceleration maneuver as a reaction to a slower vehicle driving ahead (producing longitudinal acceleration forces). These maneuvers were chosen because they represent two of the most common maneuvers shown in highway scenarios (see Bellem et al., 2016). Because highly and fully automated driving will first be introduced on highways, this criterion is of special relevance to the choice of maneuvers. In addition, these two maneuvers provide insight into a primarily lateral and a primarily longitudinal maneuver.

A 3 × 6 factorial design was chosen for each of the two maneuvers. The first factor, environment, is a between-subjects factor describing the setting, with the levels test track, MBS_Lat, and MBS_Long. The second factor, scenario, is a within-subjects factor encompassing six differently parameterized variations of a deceleration maneuver and six variations of a lane-change maneuver (see Figure 1).

Figure 1.

Illustration of the study design. The between-subjects factor, environment, encompasses two within-subjects maneuvers with six within-subjects scenarios each per environment.

Of the 72 participants in total, 28 participants experienced the on-road setting, 22 participants experienced the MBS_Lat setting, and another 22 participants experienced the MBS_Long configuration of the moving-base simulator. For each of the two maneuvers, the six scenarios comprised different combinations of three parameters that were found in earlier studies to be classified by participants as comfortable driving (Bellem et al., 2016). Mean values and their standard deviations from the study by Bellem et al. (2016) are the basis for the parameters of the study reported here. The parameters are described in more detail in Apparatus and Stimuli and are further illustrated in Figure 3. Due to physical and technical constraints, a fully factorial design of the parameters could not be established. Combinations were thus chosen such that they were feasible and comparable, for example, not requiring a significantly longer track, but also in such a way that they could still be expected to be rated differently based on just noticeable differences (see Müller, Hajek, Radic-Weissenfeld, & Bengler, 2013) and still be representative of the original data they were based on. Hence, the variations were designed to be experienced as comfortable but not equally comfortable.

The experiment was divided into two blocks, in which either lane-change maneuvers or deceleration maneuvers were driven. To prevent order effects, the order of the two blocks was counterbalanced, and the order of scenarios within the blocks was randomized across participants (see Figure 1).

Apparatus

Test-track study environment

The test-track study took place on a 280-m-long test track with two lanes closed off to public traffic. The ego vehicle, a Mercedes Benz CLS (C218), drove autonomously and was equipped with a software-and-hardware system enabling the vehicle to drive according to the predefined scenarios without any intervention of the participant. In order to synchronize the behavior of the ego and target vehicles in the deceleration scenario, the target vehicle was controlled through a high-precision GPS tracking system and several software and hardware components.

Driving simulator environment (MBS_Lat and MBS_Long)

The high-fidelity moving-base simulator is based on a 12.5-m-long linear rail system. A hexapod, which moves along this linear rail via air bearings, comprises six linear actuators and carries a spherical dome with a height of 4.5 m and an inner diameter of 7.5 m. In this dome, a full-scale vehicle mock-up can be positioned either parallel or transversely to the linear rail. Hence, depending on the orientation of the mock-up, the motion space of the linear rail can be used to simulate either lateral (MBS_Lat) or longitudinal motion (MBS_Long). Orthogonally to the linear rail, the moving-base simulator has a motion space of ±1 m. The motion-cuing system of the MBS_Lat was parameterized with a lateral scaling factor of 100% and a longitudinal scaling factor of around 17%. In the MBS_Long setting, 60% of the longitudinal accelerations and 50% of the lateral accelerations were simulated by the motion system. In order to reproduce the scenarios from the test-track study, the measured accelerations in the test-track study were directly transmitted to the motion system. Consequently, no vehicle dynamics model was interposed to simulate the scenarios. However, due to the limited workspace, a classical washout algorithm was used, which attenuates low-frequency accelerations.

An eight-channel projection system inside the dome created a 360° horizontal field of view with a resolution of 2,048 × 1,536 pixels for each projector. Additionally, two LCD displays with a resolution of 800 × 600 pixels were mounted on the side mirrors in order to simulate mirror images. An illustration of the MBS_Lat and MBS_Long is given in Figure 2. A more detailed description of the moving-base simulator can be found in Zeeb (2010).

Figure 2.

Illustration of the advanced moving-base simulator at Daimler AG. As can be seen in (a), the full-scale mock-up is positioned transversely to the linear axis so that the linear axis can be fully used to simulate lateral motion cues (MBS_Lat). In (b), the mock-up is positioned parallel to the linear axis (MBS_Long) in order to use the linear axis for the simulation of longitudinal motion cues.

The simulation environment was a replica of the test-track environment. The scenarios were presented as completely automated and needed no input from the driver. Both steering wheel and pedals were monitored, however, in order to record driver input if it occurred.

Stimuli

Deceleration scenario

The deceleration maneuvers started with the ego vehicle accelerating to 50 km/h and the target vehicle synchronously accelerating to 10 km/h. While approaching the target vehicle, the ego vehicle initiated a deceleration maneuver with the given parameterization.

The deceleration scenarios were generated by combining values of jerk upon application of brakes (jerk), gradient of time to minimum distance (TTMD_grad) and TTMD at the onset of the maneuver (TTMD_init). The onset of the maneuver was defined as the moment when the acceleration pedal was released. Jerk is defined as the derivate of acceleration (or second derivate of velocity) upon application of brakes. TTMD is defined similarly to time to collision. However, instead of taking collision as a reference, TTMD refers to the point of minimum headway distance in seconds, which is constant throughout all variations. TTMD_grad is defined as the change rate of TTMD throughout the deceleration. A higher TTMD_grad may be experienced as a faster adaptation to the head vehicle’s speed. TTMD_init is defined as the TTMD at maneuver onset. A smaller TTMD_init may be experienced as a deceleration that begins closer to the obstacle than with a larger TTMD_init. These three parameters had been shown to be relevant for perceived driving comfort in an on-road study (see Bellem et al., 2016). The absolute values of the different metrics either are the mean of metrics classified as comfortable in Bellem et al. (2016) or correspond to the standard deviation around these mean values. Parameters are illustrated in Figure 3.

Figure 3.

Illustration of the parameters for both the lane-change maneuver and the deceleration maneuver for comfortable driving. In (a), the deceleration maneuver, D, is illustrated. In (b), the lane-change maneuver, L, is shown. Additionally, the assignment of parameter combinations to the scenarios is given.

(a) Deceleration Maneuver

		Scenario
Parameter	Description	A	B	C	D	E	F
Jerk	Jerk at T_BrakingOnset in m/s³	−1.67	−1.10	−0.53	−0.53	−1.67	−0.53
TTMD_grad	Gradient or slope of the time to minimum distance after T_BrakingOnset	−0.71	−0.48	−0.26	−0.26	−0.26	−0.71
TTMD_init	Time to minimum distance at T_BrakingOnset in seconds	5.47	3.66	1.86	5.47	1.86	1.86

(b) Lane Change Maneuver

		Scenario
Parameter	Description	G	H	I	J	K	L
A_max	Maximum of acceleration in m/s²	2.68	1.76	0.43	2.50	1.70	0.48
Gamma	Ratio between acceleration minimum and maximum ( \|A_min\| / A_max )	1.21	1.09	1.01	0.93	1.13	1.05
Alpha	Relative time point of A_max in seconds (T_max / Duration of Maneuver)	0.44	0.32	0.32	0.20	0.44	0.20

Lane-change scenario

At the beginning of each lane-change maneuver, the ego vehicle accelerated to 50 km/h in the right lane. After approximately 2.5 s of maintaining the desired velocity, an automated lane change to the left according to the given parameters was performed in order to avoid a stationary vehicle, which was positioned in the right lane. During the lane-change maneuver, the steering wheel moved in accordance with the vehicle’s lateral movement.

The scenarios were generated by combining the acceleration maximum (A_max), the ratio between acceleration minimum (A_min) and A_max (Gamma), and the relative point in the maneuver at which the A_max occurs (Alpha). Whereas acceleration is a widely used metric (see Winner, Hakuli, and Wolf, 2012), Alpha and Gamma are rather novel. They are an approach to capturing the course of the lane change. Alpha describes how symmetrical the buildup of acceleration is with respect to maneuver duration. A small Alpha represents a lane change that is performed with an early maximum of lateral acceleration and may thus be experienced as a lane change with earlier distinctly perceivable onset. Gamma, on the other hand, expresses how symmetrical the maneuver is with regard to the magnitude of lateral acceleration. A Gamma of 1 represents a lane change during which lateral A_max both to the left and to the right have identical strength. The smaller the Gamma, the stronger the acceleration to the left (A_max in Figure 3) compared with the acceleration to the right (A_min in Figure 3). This value may be experienced as jerkier to the left than to the right because a larger A_max has to be built up in less time when compared with a Gamma of 1 with the same Alpha. As in the deceleration maneuver, the absolute values of the parameters are based on mean and standard deviation values classified as comfortable in a previous study (see Bellem et al., 2016). Lane-change onset was held constant to allow results to be based on the varied characteristics of the lane change and not on mere rating of risk.

Procedure

At the beginning of the experiment, participants filled in a questionnaire to obtain both standard demographic information and subjective ratings of their experience with driving simulation, automated driving, and various driver assistance systems. Subsequently, participants were made familiar with the experimental procedure. They were instructed to provide a comfort rating for each scenario, indicating the extent to which the experienced scenario corresponded to their preference for comfortable automated driving. The rating was given on a scale from 1 to 7, with 1 meaning not at all and 7 meaning exactly this way. Participants also had the opportunity to comment on the maneuvers with regard to comfortable automated driving.

In general, the experimenter made the participants aware that they should base their judgments on the lane-change or deceleration maneuver alone and not, for example, on the basis of the acceleration at the beginning of each scenario. Thus, the rating was intended to capture the comfort of the subject rather than evaluate the presence or quality of the simulation in case of the simulator studies. The intertrial interval lasted approximately 60 s for the lane-change maneuvers and 150 s for the deceleration maneuvers.

Each of the two blocks began with a training scenario to familiarize participants with the procedure (Scenarios B and H as middle variation). After each scenario, the participant took over the ego vehicle in order to drive it back to its initial starting position in the test-track study, whereas in the simulator studies, the simulation was reloaded with a new scenario.

Results

For each of the two maneuvers, we performed two separate repeated-measures ANOVAs using the afex-package in R (Singmann & Bolker, 2014; R Core Team, 2013) to contrast each simulator with the test-track study. Environment (levels: test track vs. MBS_Lat or test track vs. MBS_Long) was treated as a between-subjects factor and scenario as a within-subjects factor. Comfort ratings were squared to correct for their negatively skewed distribution. A visual inspection of the skewness-corrected ratings revealed no obvious deviations from normality, and subsequent Box’s M tests indicated no violation of homoscedasticity for each model. Since the assumption of sphericity was violated, we used the Greenhouse-Geisser correction for the degrees of freedom. Results are given in Table 2. In order to further illustrate the results, we plotted the means in Figure 4.

Table 2:

Results From the Repeated-Measures ANOVAs Concerning Comfort Rating

	MBS_Long				MBS_Lat
Maneuver	df₁, df₂	F	p	η²	df₁, df₂	F	p	η²
Deceleration
Environment	1, 48	0.66	.42	.01	1, 48	4.07	.05*	.08
Scenario	3.73, 179.21	54.84	<.0001***	.53	4.21, 202.08	47.62	<.0001***	.50
Environment × Scenario	3.73, 179.21	1.39	.24	.03	4.21, 202.08	2.36	.05*	.05
Lane Change
Environment	1,48	1.93	.17	.04	1, 48	8.52	.005**	.15
Scenario	3.33, 159,91	43.22	<.0001***	.47	3.54, 169.76	64.22	<.0001***	.57
Environment × Scenario	3.33, 159.91	1.73	.16	.03	3.54, 169.76	9.70	<.0001***	.17

Note. MBS_Long = simulator setting with vehicle positioned parallel to the linear rail system; MBS_Lat = simulator setting with vehicle positioned traversely to the linear rail system.

p ≤ .05. **p ≤ .01. ***p ≤ .001.

Figure 4.

Illustration of the comfort ratings. Squared comfort ratings were averaged across environments and scenarios and subsequently transformed back to their original scaling. Error bars indicate 95% confidence intervals.

Matching of the verbal comments showed that in scenarios with higher comfort ratings, it was mentioned more often that the scenario had been experienced as smooth and/or anticipatory. Further analysis of the comments will be omitted here but can be obtained from the authors.

In statistical terms, absolute validity is supported by the absence of significant main and interaction effects. Relative validity is given when no interaction between environment and scenario can be found. Hence, the nonsignificant interaction and main effects of the MBS_Long reported in Table 2 support that relative validity and absolute validity can be concluded for the MBS_Long. The MBS_Lat, however, was not found to have relative validity. Whereas in the lane-change maneuver the interaction effect is particularly large and relative validity has to be clearly rejected, in the deceleration maneuver it is a matter of argument whether an effect size of 0.05 and a statistical significance of 0.05 are of practical relevance, especially when considering Bonferroni-corrected significance levels of 0.0125.

To further examine which maneuver parameters might be at the basis of these differences, we performed a multilevel regression analysis using the lme4-package in R (Bates, Maechler, Bolker, & Walker, 2014) with repeated measures on the first level and participants on the second level. As the dependent variable, the comfort rating was used. For each of the six parameters describing the deceleration and lane-change maneuvers, we built a separate model and added the maneuver parameter and its square to the model as first-level predictors to allow for a curvilinear relationship. At the second level, we added environment as a dummy-coded variable to the model, with the test-track environment being set as the reference level. Additionally, a cross-level interaction between environment and the maneuver parameters was entered. Again, comfort ratings were squared to meet the assumption of normally distributed residuals. Subsequently, comfort ratings and maneuver parameters were subjected to a z-transformation before model fitting to prevent biased parameter estimation due to multicollinearity produced by the high correlation between the linear and curvilinear term of the maneuver parameter. For each model, a visual inspection of the residual plots revealed no obvious deviations from normality or homoscedasticity. To determine the variance explained by the parameters, we calculated R² for each model according to the algorithm suggested by Xu (2003). To illustrate the results, we simulated expected values based on the multilevel regression models as proposed by King, Tomz, and Wittenberg (2000). Results and illustrations of the multilevel analyses are given in Figure 5.

Figure 5.

Results of the multilevel analyses. In the left column, standardized regression coefficients are illustrated in a coefficient plot. The thick error bars indicate the 95% confidence interval, and the thin error bars indicate the 99.9% confidence interval. A fixed effect is significant on the alpha level of 0.05 (0.001) when zero is outside of its thicker (thinner) confidence interval. In the right column, simulated expected values are plotted for each environment. Note that comfort ratings were transformed back to their original scaling. Error bands indicate the 95% confidence interval. R² was calculated using the algorithm proposed by Xu (2003).

No significant effects of both linear and curvilinear components of the six parameters in the MBS_Long can be found. Therefore, the effects of the parameters on driving comfort are identical between the test-track study and the MBS_Long. In the MBS_Lat, however, the effect of the linear and curvilinear component of the predictor A_max was significantly different from those of the test-track study. In addition, the results show, A_max and TTMD_grad are able to explain most variance in the lane-change or deceleration maneuver, respectively.

Discussion

For both lane-change and deceleration maneuvers, results support relative and absolute validity for the MBS_Long. In the MBS_Lat configuration, however, relative validity for the lane-change maneuver cannot be supported. Also, data only hint at a weak relative validity for the deceleration maneuver. Consequently, only the moving-base simulator configured with lateral and longitudinal scaling factors of 50% and 60% (MBS_Long) seems to be a valid research tool to examine driving comfort in automated lane-change and deceleration maneuvers. In contrast, the lateral configuration (MBS_Lat) seems inadequate for studying driving comfort, especially for lane-change maneuvers. Furthermore, we found not only that the relative comfort ratings among the lane-change conditions did not correspond to those of the test-track environment but that driving comfort was generally rated lower in the MBS_Lat, compared with the test-track environment.

Because the MBS_Long configuration optimally exploits the available motion space to simulate longitudinal motion cues, it is not surprising that this configuration is superior, compared with the MBS_Lat configuration in the deceleration maneuver. However, it is surprising that in this study the MBS_Long seems to be superior to the MBS_Lat configuration, which is optimized for simulating these lateral cues, in the lane-change maneuver, even though a lane change requires an adequate simulation of lateral motion. In the following paragraphs, several explanations for these counterintuitive findings are discussed.

As can be seen from the multilevel analyses, the effect in the MBS_Lat of the linear and curvilinear component of the maximal acceleration A_max on driving comfort was significantly different from those in the test-track study, whereas in all other parameters the MBS_Lat produced identical results. At the same time, A_max was found to be the best predictor of driving comfort, as it explained most of the variance. This finding indicates that the source of the poor validity of the MBS_Lat may be traced back to an erroneous simulation of maximal lateral acceleration cues. From the simulated expected values illustrated in Figure 5, we see that already a little increase in lateral acceleration in the MBS_Lat led to a severe decrease in comfort ratings, compared with a rather mild decrease in the test-track study. Thus, A_max seemed to be also responsible for driving comfort being rated significantly lower in the MBS_Lat, compared with the test-track environment. When considering the scaling factors for lateral accelerations in the two simulator configurations, it is evident that simulating only 50% (MBS_Long) of the lateral acceleration produced more valid comfort ratings than a physically valid 100% scaling factor (MBS_Lat).

This counterintuitive result was also found in previous studies. Pretto et al. (2009) varied four scaling factors (0.5, 0.75, 1.0, 1.25) for lateral acceleration in a slalom driving task and concluded a scaling factor of 60% as being the perceived optimal motion gain. Greenberg et al. (2003) varied three scaling factors (0, 0.25, 0.5, 0.7) in a lane-change task and concluded that lateral motion scale factors of around 50% are sufficient to produce naturalistic driving performance. Probably one of the most comprehensive motion-cuing studies was carried out by Berthoz et al. (2013). Across three advanced moving-base driving simulators, they examined the effect of lateral motion scaling on the perceived realism and driving performance in a slalom driving task. They concluded a factor between 0.4 and 0.75 as being the optimal motion gain for lateral acceleration. As potential explanations, they noted that the preference for subunity motion scaling might be attributable to constraints of the vehicle dynamics models. However, our findings contradict this notion, because no vehicle model was interposed in this study. Rather, measured accelerations from the test-track study were directly transmitted to the motion system.

A plausible and promising explanation was proposed by Berthoz et al. (2013). They suggested that motion cuing has to be scaled down in order to match an underestimated speed in virtual environments. Thus, the basis of this effect might be found rather in the physical validity of the display system than in the motion system (Correia Grácio, Bos, van Paassen, & Mulder, 2014). Indeed, many studies showed that speed and distance were underestimated in virtual environments (e.g., Banton, Stefanucci, Durgin, Fass, & Proffitt, 2005; Fischer et al., 2012; Harris, Jenkin, & Zikovitz, 2000; Knapp & Loomis, 2004).

Even though an underestimation of speed is evident in decades in driving simulator research, the theoretical basis still remains unknown. Considering that the aforementioned studies relied on visual systems with monoscopic cues, there is growing empirical evidence that providing stereoscopic cues enhances the visual-vestibular integration (Butler, Campos, Bülthoff, & Smith, 2011) and might reduce the visual-vestibular mismatch, which is often assumed to evoke simulator sickness (Reason, 1978). Furthermore, it was found that stereoscopic cues considerably improved heading judgments (Van den Berg & Brenner, 1994), increased perceptions of ego speed and self-displacement (Palmisano, 2002), and decreased vection onsets and increased vection duration (Palmisano, 1996; but see Ijsselsteijn, Ridder, Freeman, Avons, & Bouwhuis, 2001). As a consequence, introducing stereoscopic cues to fixed-base simulators might enhance their physical validity at first sight, but they would most likely decrease the behavioral validity due to a higher visual-vestibular mismatch. This possibility might explain why a stereoscopic display in an otherwise identical fixed-base driving simulator was barely found to improve driving performance, compared with a monoscopic display (Forster, Paradies, & Bee, 2015).

Nevertheless, this hypothesis remains speculative and it will be the task of researchers to determine whether the preference for subunity scaling factors might dissolve when providing stereoscopic visual cues.

Leaving the theoretical basis aside, our data support that the preference for subunity scaling factors seems to apply not only to active driving tasks but also to passive driving tasks. Furthermore, we could demonstrate for the first time that a moving-base simulator can be utilized to study perceived comfort in autonomous vehicles, given the motion-cuing algorithm is adequately parameterized. However, we want to point out that it is difficult to generalize findings across several research questions and different architectures of driving simulators. Considering that a growing body of traffic psychology research relies on driving simulators, it is crucial that driving simulators are subject to constant validation. Furthermore, as Godley et al. (2002) noted, the accumulated evidence that different driving simulators were found to be useful research tools for a variety of driving tasks adds weight to the validity of driving simulator research.

In addition to the findings on simulator validity, it was possible to identify in which way the characteristics of the maneuvers influence comfort ratings. Results point in the direction of acceleration being the strongest influence on comfort. As expected, less acceleration is perceived as favorable. Participants’ verbal comments regarding the smoothness of scenarios and anticipatory qualities in combination with the found importance of TTMD_grad also suggests the course of the maneuver plays an important role in experiencing comfort.

Potential Limitations

We have performed many statistical tests, which increased the risk of Type I errors. However, we believe that corrections for the alpha level (e.g., Bonferroni correction) would have been too conservative, especially when considering that the simulators were not tested against being valid but tested against not being valid. In addition, we point out that using a within-subjects design would have provided the method of choice but was not feasible due to the large logistical challenge and reasonableness toward participants. It can further be discussed whether a multidimensional measure of comfort may have been a stronger measure. However, participants in pretests have shown difficulty rating comfort on a multidimensional scale. This finding is also reflected in the comments participants gave after each variation, which mainly reduced comfort to smoothness and anticipation.

A further possible limitation addresses the method used to assess driving comfort. Also due to the fact that no common understanding exists on the precise definition of comfort, finding a valid tool for assessment has proven to be difficult. In retrospect, we suspect that the unidimensional 7-point scale used may not have been the optimal choice. There have been different approaches to objectively measure driving comfort, such as via physiological measures (e.g., Engeln & Vratil, 2008), or indirectly by measuring discomfort (e.g., Hartwich, Beggiato, Dettmann, & Krems, 2015). However, a residual variance remains even when using objective measures (Engeln & Vratil, 2008). We are also aware that other aspects, such as feeling safe, may have influenced the ratings as they are closely linked or, in some theories, a part of comfort (see Summala, 2007). However, we also believe that comfort is not obtainable if a person is not feeling safe. A valid and widely supported questionnaire or assessment method has yet to be established.

Especially in simulators, motion sickness can be an important influence on data. Because participants in this study reported mainly no or only seldom little discomfort regarding motion sickness, we did not control our results for symptoms of simulator and motion sickness. Also because trials were randomized, we believe that a systematic bias of motion or simulator sickness is unlikely. At last, our analysis and discussion might be characterized as having a post hoc character. However, we want to point at the explorative nature of this study, which is due to the novelty of studying perceived comfort of autonomous vehicles in driving simulators.

Conclusion

We found that participants preferred downscaled motion cues to physically correct motion cues, which can be plausibly attributed to an underestimation of speed in virtual environments. It would be interesting for further research to test whether providing stereoscopic cues would enhance speed perception and thus resolve the preference for subunity scaling factors. In summary, we were able to demonstrate relative and absolute behavioral validity for the MBS_Long for both lane-change and deceleration maneuvers within the range of parameterization that is relevant for driving comfort in autonomous cars. For the MBS_Lat in the given configuration (100% scaling factor for lateral motion cues, 17% scaling factor for longitudinal motion cues), however, we could conclude only weak relative validity for the deceleration maneuver and no relative validity for the lane-change maneuver. A scaling factor of 50% to 60% appears to be the key factor for behavioral validity of the driving maneuvers under consideration in this study, which can be provided by the MBS_Long configuration consistently for both lateral and longitudinal motion cues.

Regarding experiencing comfort, maximum acceleration and how parameters change over the course of a maneuver seem to be key influences.

Key Points

Perceived driving comfort of differently parameterized lane-change and deceleration maneuvers of an automated vehicle was studied on a test track and in two moving-base simulator configurations.

Acceleration shows to have the strongest influence on experienced comfort in the used maneuvers.

Basically, moving-base simulators are capable of reproducing results from a test-track study.

In line with previous findings, we recommend linear motion scaling factors in the range of approximately 50% to 60% in order to obtain valid results.

Footnotes

Hanna Bellem studied psychology at the Technische Universität (TU) Chemnitz (Germany) and NTNU Trondheim (Norway). She received her master’s degree in 2013 from TU Chemnitz. From 2013 to 2016, she worked as a doctorate student of the TU Chemnitz in cooperation with Daimler AG. Currently, she is employed as a research specialist at BMW AG in Munich, Germany.

Malte Klüver studied psychology at the Universities of Vienna (Austria), Warsaw (Poland), and Mainz (Germany), where he received his master’s degree in 2013 and his PhD in 2016. He wrote his dissertation on the behavioral validity of driving simulator studies at Daimler AG. Currently, he is employed as a research scientist at the IPV GmbH in Kremmen, Germany, and works on the optimization of the nationwide practical driving test in Germany.

Michael Schrauf received his MSc in physics in 1986 and his PhD in physiology in 1989 from Philipps-University of Marburg (Germany). In 2001, he became a private lecturer in psychology at the Heinrich-Heine-University of Düsseldorf (Germany). Since 2002, he is working as a senior scientist in the field of advanced driver assistance systems at Daimler AG, Research and Development, in Sindelfingen, Germany.

Hans-Peter Schöner received his PhD in electrical engineering from the RWTH Aachen University in 1989. Since 2012, he is the head of the Driving Simulators Center at Daimler AG in Sindelfingen, Germany.

Heiko Hecht received his PhD in experimental psychology from the University of Virginia in 1992. He worked as a research scientist and lecturer at the Max-Planck Institut für Psychologische Forschung, NASA Ames Research Center, Universität Bielefeld, and the Massachusetts Institute of Technology. Since 2002, Heiko Hecht holds the chair of experimental psychology at the Johannes Gutenberg-Universität of Mainz, Germany.

Josef F. Krems is the head of the Department of Cognitive and Engineering Psychology at Technische Universität Chemnitz. He received his PhD in psychology from the University of Regensburg in 1984.

References

Allen

R. W.

Mitchell

D. G.

Stein

A. C.

Houge

J. R.

(1991). Validation of real-time man-in-the-loop simulation (VTI Rapport 372A part 4). Linköping, Sweden: Swedish Road and Traffic Research Institute.

Zhang

Salvendy

(2014). Validity of driving simulator for agent–human interaction. In HCI International 2014: Posters’ extended abstracts (pp. 563–569). Berlin, Germany: Springer.

Banton

Stefanucci

Durgin

Fass

Proffitt

D. R.

(2005). The perception of walking speed in a virtual environment. Presence: Teleoperators and Virtual Environments, 14, 394–406.

Bates

Maechler

Bolker

Walker

(2014). lme4: Linear mixed-effects models using Eigen and S4. R Package Version 1.1-7. Retrieved from http://CRAN.R-project.org/package=lme4

Bellem

Schönenberg

Krems

J. F.

Schrauf

(2016). Objective metrics of comfort: Developing a driving style for highly automated vehicles. Transportation Research Part F, 41, 45–54.

Berthoz

Bles

Bülthoff

Correia Gracio

Feenstra

Filliard

Huhne

Kemeny

Mayrhofer

Mulder

Nusseck

H. G.

Pretto

Reymond

Schlüsselberger

Schwandtner

Teufel

Vailleau

van Paasen

M. M. R.

Vidal

Wentink

(2013). Motion scaling for high-performance driving simulators. IEEE Transactions on Human–Machine Systems, 43, 265–276.

Blaauw

G. J.

(1982). Driving experience and task demands in simulator and instrumented car: A validation study. Human Factors, 24, 473–486.

Blana

(1996). Driving simulator validation studies: A literature review (Report No. 480). Leeds, UK: Institute of Transport Studies, University of Leeds.

Boer

E. R.

(1996). Tangent point oriented curve negotiation. Proceedings of the 1996 IEEE Intelligent Vehicles Symposium (pp. 7–12). New York, NY: IEEE.

10.

Butler

J. S.

Campos

J. L.

Bülthoff

H. H.

Smith

S. T.

(2011). The role of stereo vision in visual-vestibular integration. Seeing and Perceiving, 24, 453–470.

11.

Correia Grácio

Bos

J. E.

van Paassen

M. M.

Mulder

. (2014). Perceptual scaling of visual and inertial cues: Effects of field of view, image size, depth cues, and degree of freedom. Experimental Brain Research, 232, 637–646.

12.

De Looze

M. P.

Kuijt-Evers

L. F. M.

van Dieën

(2003). Sitting comfort and discomfort and the relationships with objective measures. Ergonomics, 46, 985–997.

13.

Ellinghaus

Schlag

(2001). Beifahrer. Eine Untersuchung über die psychologischen und soziologischen Aspekte des Zusammenspiels von Fahrer und Beifahrer [Passengers. A study of the psychological and sociological aspects of driver and passenger interaction]. Cologne, Germany: Uniroyal Verkehrsuntersuchung.

14.

Engeln

Vratil

(2008). Fahrkomfort und Fahrgenuss durch den Einsatz von Fahrerassistenzsystemen [Driving comfort and driving pleasure through usage of driver assistance systems]. In Schade

Engeln

(Eds.), Fortschritte der Verkehrspsychologie. Beiträge zum 45. Kongress der Deutschen Gesellschaft für Psychologie (pp. 275–288). Wiesbaden, Germany: VS Verlag für Sozialwissenschaften.

15.

Fischer

Eriksson

Oeltze

(2012, September). Evaluation of methods for measuring speed perception in a driving simulator. Paper presented at the Driving Simulation Conference Europe 2012, Paris, France.

16.

Forster

Paradies

Bee

(2015, September). The third dimension: Stereoscopic displaying in a fully immersive driving simulator. Paper presented at the Driving Simulation Conference 2015, Tübingen, Germany.

17.

Gemou

(2013). Transferability of driver speed and lateral deviation measurable performance from semi-dynamic driving simulator to real traffic conditions. European Transport Research Review, 5, 217–233.

18.

Godley

S. T.

Triggs

T. J.

Fildes

B. N.

(2002). Driving simulator validation for speed research. Accident Analysis & Prevention, 34, 589–600.

19.

Grabe

Pretto

Giordano

P. R.

Bülthoff

H. H.

(2010, September). Influence of display type on drivers’ performance in a motion-based driving simulator. Paper presented at the Driving Simulation Conference 2010, Paris, France.

20.

Greenberg

Artz

Cathey

(2003, October). The effect of lateral motion cues during simulated driving. Paper presented at the Driving Simulation Conference North America 2003, Dearborn, MI.

21.

Greenberg

Blommer

(2011). Physical fidelity of driving simulators. In Fischer

D. L.

Rizzo

Caird

J. K.

Lee

J. D.

(Eds.), Handbook of driving simulation for enigneering, medicine and psychology (pp. 7-1–7-24). London: CRC Press.

22.

Harris

L. R.

Jenkin

Zikovitz

D. C.

(2000). Visual and non-visual cues in the perception of linear self motion. Experimental Brain Research, 135, 12–21.

23.

Hartwich

Beggiato

Dettmann

Krems

J. F.

(2015). Drive me comfortable: Customized automated driving styles for younger and older drivers. In VDI Wissensforum GmbH (Ed.), Der Fahrer im 21. Jahrhundert (8th ed., pp. 271–283). Düsseldorf, Germany: VDI Verlag.

24.

Ijsselsteijn

Ridder

H. d.

Freeman

Avons

S. E.

Bouwhuis

(2001). Effects of stereoscopic presentation, image motion, and screen size on subjective and objective corroborative measures of presence. Presence: Teleoperators and Virtual Environments, 10, 298–311.

25.

King

Tomz

Wittenberg

(2000). Making the most of statistical analyses: Improving interpretation and presentation. American Journal of Political Science, 44, 347–361.

26.

Klüver

Herrigel

Heinrich

Schöner

H. P.

Hecht

(2016). The behavioral validity of dual-task driving performance in fixed and moving base driving simulators. Transportation Research Part F: Traffic Psychology and Behaviour, 37, 78–96.

27.

Knapp

J. M.

Loomis

J. M.

(2004). Limited field of view of head-mounted displays is not the cause of distance underestimation in virtual environments. Presence: Teleoperators and Virtual Environments, 13, 572–577.

28.

Krist

(1994). Modellierung des Sitzkomforts: Eine experimentelle Studie [Modeling seat comfort: An experimental study]. Weiden, Germany: Schuch.

29.

Lee

J. D.

Ward

Boer

Brown

T. L.

Balk

S. A.

Ahmad

(2013). Exploratory advanced research: Making driving simulators more useful for behavioral research. Simulator characteristics comparison and model-based transformation (No. N2013-016). Washington, DC: U.S. Department of Transportation.

30.

MacSween-George

(2003). A public opinion survey: Unmanned aerial vehicles for cargo, commercial, and passenger transportation. In 2nd AIAA “Unmanned Unlimited” Conf. and Workshop & Exhibit. https://dx-doi-org.web.bisu.edu.cn/10.2514/6.2003-6519

31.

Mullen

Charlton

Devlin

Bédard

(2011). Simulator validity: Behaviors observed on the simulator and on the road. In Fisher

D. L.

Rizzo

Caird

Lee

J. D.

(Eds.), Handbook of driving simulation for engineering, medicine and psychology (pp. 13.1–13.18). Boca Raton, FL: CRC Press.

32.

Müller

Hajek

Radic-Weissenfeld

Bengler

(2013). Can you feel the difference? The just noticeable difference of longitudinal acceleration. In Proceedings of the Human Factors and Ergonomics Society 57th Annual Meeting (pp. 1219–1223). Santa Monica, CA: Human Factors and Ergonomics Society.

33.

Nilsson

(1993). Behavioural research in an advanced driving simulator-experiences of the VTI system. In Proceedings of the Human Factors and Ergonomics Society 37th Annual Meeting (pp. 612–616). Seattle, WA: Human Factors and Ergonomics Society.

34.

Palmisano

(1996). Perceiving self-motion in depth: The role of stereoscopic motion and changing-size cues. Perception & Psychophysics, 58, 1168–1176.

35.

Palmisano

(2002). Consistent stereoscopic information increases the perceived speed of vection in depth. Perception, 31, 463–480.

36.

Pretto

Nusseck

Teufel

Bülthoff

H. H.

(2009, February). Effect of lateral motion on drivers’ performance in the MPI motion simulator. Paper presented at the Driving Simulation Conference 2009, Monaco.

37.

R Core Team. (2013). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. Retrieved from http://www.R-project.org/

38.

Reason

(1978). Motion sickness adaptation: A neural mismatch model. Journal of the Royal Society of Medicine, 71, 819–829.

39.

Reed

M. P.

Green

P. A.

(1999). Comparison of driving performance on-road and in a lowcost simulator using a concurrent telephone dialling task. Ergonomics, 42, 1015–1037.

40.

SAE International. (2014). Taxonomy and definitions for terms related to on-road motor vehicle automated driving systems (Standard J3016). Retrieved from http://www.sae.org/misc/pdfs/automated_driving.pdf

41.

Schoettle

Sivak

(2014). A survey of public opinion about autonomous and self-driving vehicles in the US, the UK, and Australia (Report No. UMTRI-2014-21). Ann Arbor: University of Michigan, Transportation Research Institute.

42.

Siegler

Reymond

Kemeny

Berthoz

(2001, September). Sensorimotor integration in a driving simulator: Contributions of motion cueing in elementary driving tasks. Paper presented at the Driving Simulation Conference 2001, Nice, France.

43.

Singmann

Bolker

(2014). afex: Analysis of factorial experiments. R Package Version 0.11-131. Retrieved from http://CRAN.R-project.org/package=afex

44.

Sivak

Schoettle

(2015). Motion sickness in self-driving vehicles (Report No. UMTRI-2015-12). Ann Arbor: University of Michigan, Transportation Research Institute.

45.

Summala

(2007). Towards understanding motivational factors in driver behavior: Comfort through satisficing. In Cacciabue

P. C.

(Ed.), Modelling driver behavior in automotive environments (pp. 189–207). London, UK: Springer.

46.

Törnros

(1998). Driving behaviour in a real and a simulated road tunnel: A validation study. Accident Analysis & Prevention, 30, 497–503.

47.

Van den Berg

Brenner

. (1994). Why two eyes are better than one for judgements of heading. Nature, 371, 700–702.

48.

Vergara

Page

. (2000). System to measure the use of the backrest in sitting-posture office tasks. Applied Ergonomics, 31, 247–254.

49.

Wang

Mehler

Reimer

Lammers

D’Ambrosio

L. A.

Coughlin

J. F.

(2010). The validity of driving simulation for assessing differences between in-vehicle informational interfaces: A comparison with field testing. Ergonomics, 53, 404–420.

50.

Winner

Hakuli

Wolf

(Eds.). (2012). Handbuch Fahrerassistenzsysteme: Grundlagen, Komponenten und Systeme für aktive Sicherheit und Komfort [Handbook of driver assistance systems: Basics, components, and systems for active safety and comfort] (2nd ed.). Wiesbaden, Germany: Vieweg + Teubner.

51.

(2003). Measuring explained variation in linear mixed effects models. Statistics in Medicine, 22, 3527–3541.

52.

Zeeb

(2010, September). Daimler’s new full-scale, high-dynamic driving simulator: A technical overview. Paper presented at the Driving Simulation Conference 2010, Paris, France.

53.

Zhang

Helander

M. G.

Drury

C. G.

(1996). Identifying factors of comfort and discomfort in sitting. Human Factors, 38, 377–389.