Abstract
Vehicles with driving automation are becoming increasingly present despite the reported apprehension of potential consumers. The potential benefits, such as fewer crashes, lighter traffic, and increased transportation access, give merit in researching how to engender appropriate human- automation interaction that will ensure a smoother adoption of the technology. One method involves investigating how users receive information about the vehicle. Using a simulated highly automated vehicle, researchers examined how content temporality and modality affected the situational trust and cognitive workload of 36 participants using subjective measures and 15 participants using non-linear dynamics. Researchers found only one significant main effect of temporality on workload; however, post-hoc comparisons between groups were insignificant. Nevertheless, applying non-linear dynamics to driving research is a novel and underutilized approach. Researchers, designers, and users may benefit from using real-time measures rather than aggregate scores to understand how driver behavior changes based on the environment.
Keywords
Introduction
Within the past few decades, there has been an increasing push to create vehicles with high driving automation, also known as self-driving cars. This new technology has the potential to reduce collisions, traffic, and limited accessibility. The Society of Automotive Engineers (2021) has classified varying levels of autonomy, with L0 having absolutely no autonomous capabilities L5 being entirely autonomous, no operator needed. Currently, L3 and L4 technologies are being phased into the market: at these levels, the driver shifts from an active role to a passive, supervisory role.
Therefore, much consideration has gone into how best to support this transition. Despite this, many drivers of all ages note they are not comfortable with or trusting of automated vehicles, and this in turn results in lower use (AAA, 2019; Muir & Moray, 1996).
Human-Machine Interface Modality
One of the main sources of communication between car and driver is supplied by a Human-Machine Interface (HMI). HMIs are often used to explain to the driver what the intentions and actions of the vehicle are and are thus an important component of building trust in the system. However, there is no set standard on how to best convey this information about behavior and intention to the driver.
As such, researchers have been considering the different strengths and weaknesses of various modalities for communicating information. The results of this research have shown that each modality is best suited to a specific type of information. For example, auditory information is best for quickly grabbing attention, especially for sustained attention tasks (see Szalma, 2004). Systems may combine multiple modalities to set up a cohesive line of communication from the system to the user. In the context of automated vehicles, multimodality has been shown to prime drivers to potential upcoming scenarios, which in turn increases their performance when conducting a take-over in an emergency (Borojeni et al., 2018). Modality type has also been shown to convey different levels of urgency and importance, thus affecting how drivers behave when driving manually (Politis et al., 2013).
Despite this, current research results have not comprehensively explored the relationship between multimodality, trust, and cognitive workload using both linear and non-linear techniques in automated vehicles. Given the dynamic nature of behavior, especially when assessing psychological constructs such as situational trust and cognitive workload, using both linear and non- linear techniques is critical to having a complete picture.
Situational Trust
The trust between humans and automation is defined as “the attitude that an agent will help achieve an individual’s goals in a situation characterized by uncertainty and vulnerability” (Lee & See, 2004, p. 54). Trust increases operator’s usage (Muir & Moray, 1996); however, incommensurate trust with the capabilities of the automation can lead to inappropriate use (Lee & See, 2004; Muir & Moray, 1996; Parasuraman & Riley, 1997). Inappropriate use is of particular concern in automated vehicles as it has resulted in fatalities (National Transportation Safety Board, 2020). Understanding the formation and maintenance of user trust in systems requires assessing complex, dynamic processes.
Hoff and Bashir (2015) developed a three-layer model of trust with each layer capturing a different element involved in the formation of human-automation trust. They defined dispositional trust as an individual's propensity to trust automation, regardless of context. Learned trust involved trust based on previous experiences with the system. Situational trust corresponded to the individual's trust based on internal and external contextual factors. Situational trust highlights the impact of context on trust development and emphasizes the influence of trust on behavioral outcomes. Holthausen et. al (2020) developed a method for measuring dynamic, situational trust focusing on multiple components of situational trust, as determined by Hoff and Bashir (2015), called the Situational Trust Scale for Automated Driving (STS-AD). Due to the dynamic nature of automated vehicles, we focused on investigating situational trust.
Physiological Measures and Cognitive Workload
Given reports of users declaring they would not feel comfortable in an automated vehicle, we sought to investigate psychological comfort. Psychological comfort is a construct lacking a validated measure or agreed-upon definition in the field of human-automation interaction. As such, cognitive workload, or stress, was employed as a proxy of assessing psychological comfort. A commonly used method of measuring cognitive workload involves self-report data through the NASA Task Load Index (TLX). The NASA-TLX is a multi- dimensional self-report measure designed to assess six aspects of workload (Hart & Staveland, 1988). However, objective measures of mental fatigue and workload may provide more accurate insight.
Finding specific physiological measures that may infer psychological stress is difficult as previous literature has unsuccessfully attempted to draw a direct connection between cognitive workload and various physiological measures such as respiration rate, skin response, blood pressure, and others (Charles & Nixon, 2019). However, some research has indicated that heart rate is a reliable measure for mental stress (Delliaux et al., 2019). As such, we chose to compare the relationship between heart rate variability (as measured by inter-beat “RR” interval) and cognitive workload (as measured by the NASA-TLX).
Non-Linear Dynamics and Heart Rate Variability
As heart rate measures became more prominent in human-subjects research, it became increasingly apparent that they were not well represented by the linear analysis techniques in use at the time. Thus, researchers began to apply non-linear dynamic techniques to electrocardiogram (ECG) data. A multitude of techniques have been established for evaluating heart rate data: time domain/frequency domain, fractal analysis, entropy, Poincare plots, and Lyapunov exponents are just a few examples (Henriques et al., 2020). At current, there is not much, if any, research that has applied these analysis methods to HRV in the context of automated vehicles.
Study Objective
The purpose of the following study was to investigate how HMI modality and temporality of information shared through HMIs affects drivers’ trust by comparing heart rate variability and cognitive workload using both linear and non-linear analysis techniques. To the authors’ best knowledge, this study provides the first instance of applying non-linear dynamics to psychological constructs in automated vehicles.
Method
Participants
Thirty-six participants were recruited through the SONA system (1.5 hours of credit) at a state university. The participants ranged in age from 18 to 28 years (M = 19.7, SD = 2.1). Study inclusion criteria included a driver’s license, full body mobility, and normal or corrected-to-normal vision. The study procedures followed standards and approval set forth by the university’s Institutional Review Board.
Materials
Apparatuses
For this study, participants experienced driving scenarios developed using ISAT, a software tool created by the National Advanced Driving Simulator (NADS) that can develop simulated driving scenarios, in a NADS miniSim™ quarter cab simulator. A touchscreen HMI tablet was mounted to the simulator on the right of the steering wheel in a location similar to an infotainment screen. ECG data was collected using a Garmin Forerunner 945 watch and a Garmin HRM-Pro chest strap. Participants responded to questions on a touchscreen tablet.
Drives
All participants experienced two study drives in a suburban city environment. Each drive lasted approximately 17 minutes and included four potentially stress-inducing events. The events from the first drive included a pedestrian stop, red light, construction zone, and a roundabout. The second drive events were an accident scene, a roundabout, a red flashing light, and a traffic jam. The drives occurred in the same order for each participant.
HMI
When a notification appeared on the HMI screen, the left portion provided information about the vehicle’s actions and the right about the environment (see Figure 1). The number of HMI alerts corresponded to the number of events or vehicle behavior within a drive but remained constant throughout all conditions. The exact language of the alerts differed depending on the content temporality type. Future alerts used language indicating an action would take place in the future while current alerts used language indicating the action was presently occurring.
Depictions of how the HMI alerts appeared at a traffic light event for the Future (a) condition and the Current (b) condition.
Further, the HMI provided information through 2 levels of modality, multimodal and unimodal. These differed in that the multimodal condition included a short chime when a notification appeared while the unimodal condition only presented visual information.
Experimental Design
The study involved a mixed design with a between- subjects variable (content temporality) and within- subjects variable (HMI modality). The term content temporality is used here to denote the timing that the information in the HMI is pertaining to but not to the exact degree as with reaction timing. That is, the Future condition received HMI notifications about upcoming events or vehicle actions while the Current condition received HMI notifications about events and vehicle actions that were occurring in real-time. The third condition, Current+Future, included a mix of half Current notifications and half Future conditions.
Participants were randomly assigned to one of the content temporality conditions. All participants experienced both HMI modality conditions, unimodal or multimodal. The HMI modality was counterbalanced across the drives for participants.
Procedure
Participants completed a short drive to test for simulator sickness. Once complete, participants wore the heart rate monitor and began the drives. During each drive, participants immediately turned on the driving automation function and subsequently began a non-driving related task (NDRT) on a tablet. The NDRT was a reading comprehension task. This task simulated the divided attention that would likely occur if a user were to engage in another activity while in a highly automated vehicle. After an event occurred, participants were informed by an automated voice to respond to a short section of questions regarding their current trust status. After responding, participants resumed the NDRT. At the conclusion of the study drive, participants responded to questions about the drive overall.
Results
Situational Trust
To assess the effect of situational trust, as measured by the STS-AD, on HMI modality and content temporality, we ran a mixed method ANCOVA with propensity to trust as a covariate. The main effect of modality, the interaction of content temporality and modality, and the main effect of temporality did not significantly impact situational trust scores. Propensity to trust, however, did significantly impact users situational trust in the automated vehicle, F(1,32) = 6.334, p = 0.017, η2 = 0.147. Interestingly, participants in the Future condition reported consistently higher trust scores, with the Future Unimodal condition corresponding to the highest trust of all the groups (M = 5.912, SD = 0.782), while those in the Current+Future group reported the lowest trust scores, with the Current+Future Multimodal group having the lowest average overall (M = 5.250, SD = 1.284).
Cognitive Workload
ECG Data
Of the original 36 respondents, 21 were excluded in the following analysis due to incomplete or poor ECG data. The final number of participants for the ECG data had an average age of 19.6 years (N = 15, SD = 2.02). To determine the participant heart rate variability during the drives, we transformed each set of RR data from both drives for each participant into a time series. This allowed us to perform phase space reconstruction on each individual drive (see Figure 2), which required calculating the Average Mutual Information (AMI) and percentage of False Nearest Neighbors. Afterwards, we calculated the time delay (tau) and embedding dimension of each reconstruction, which were then used to calculate the Lyapunov exponent for each drive. All non-linear analysis was conducted with MATLAB and MATLAB Runtime.
An example time series (top) and phase space reconstruction (bottom). The time series shows one participant’s HRV data over the course of the drive. The phase space reconstruction depicts HRV variability in three dimensions based on the time delay determined in the analysis.
The Lyapunov exponent (λ) is a measure of how sensitive the system is to its initial conditions. Ultimately, this measure calculates the distance between two near-neighbor points in the system as they move throughout the phase space and is used to indicate if a system is chaotic. If the points diverge over time (λ1 > 0), then the system is considered chaotic. If they remain the same distance (λ1 = 0), then the system is metastable. And finally, if the points converge (λ1 < 0), then the system is considered conservative/converging (Guastello & Liebovitch, 2009).
Preliminary analysis of the Lyapunov exponents showed that the Future condition values ranged from -0.40 to +.20 (µ = -0.10), Current+Future ranged from - .42 to -.001(µ = -0.25), and Current ranged from -.52 to - .05 (µ = -0.23). On average, all conditions trended negatively, indicating converging systems.
We then conducted a mixed methods ANOVA to analyze if content temporality and modality lead to significantly different Lyapunov exponents. All effects did not reach significance, however, there was an interesting trend in the results: all three temporalities converged to a similar point as they moved from unimodal to multimodal, as seen in Figure 3.
Line graph of content temporality and modality on average largest Lyapunov exponents.
Subjective Measures
The NASA-TLX measure provided a measure of subjective cognitive workload and stress. We used a mixed model ANOVA to determine if the participants’ average workload scores differed. We found a significant main effect of temporality on cognitive workload, F(2,33) = 3.735, p < 0.05, η2 = 0.167. The interaction of temporality and modality did not significantly impact the average workload scores. Additionally, participants’ workload scores did not significantly differ based on the modality of the HMI. Post Hoc Bonferroni comparisons did not find significant differences between the specific temporality conditions. However, the Current+Future condition, reported the highest marginal mean workload (M = 38.490, SE = 3.141), while the Future conditions reported the lowest marginal mean workload (M = 27.644, SE= 3.141).
Discussion
The authors sought to investigate how the information communicated by the HMI affects users’ trust through a novel application of non-linear dynamics techniques on HRV. Specifically, we examined how presenting notifications at different times and through different modalities affected situational trust and cognitive workload. Unexpectedly, temporality and modality did not impact situational trust scores or heart rate variability. However, temporality did lead to a significant main effect on workload. The Current+Future temporality may have confused participants as the information provided by the HMI was inconsistent, engendering a higher workload. In assessing the trends within the data, the Current+Future group was consistently the least preferred, with the highest workload and lowest situational trust. The Future group was the most preferred, with the lowest workload and highest situational trust. It is possible that participants in the Future condition did not experience as much stress as other participants because the HMI provided them with an awareness of upcoming events, allowing them to feel more comfortable and trusting in the vehicle.
Researchers were surprised to find no significant effects involving HMI modality. The addition of the chime could have caused participants to be more aware of the notification. Given the subject matter of the alerts, reading the notification could have caused participants some slight stress, as it implied a change in the vehicle or the environment. However, we did not find these results. This may have implications for system transparency needs in higher levels of driving automation.
Further, while there were no significant differences in Lyapunov exponents between modality type or temporality, there was an interesting trend that may indicate that they all reached similar levels of workload. For each condition, the average Lyapunov exponents converged to a similar point from the first to the second drive. This may indicate that the participants’ HRV came to an equilibrium no matter what condition they experienced, where higher HRV in the first drive dropped to a lower HRV in the second drive and vice versa. As such, it would seem that participants’ subjective experience of cognitive workload differed in the first drive, but in the second drive, all participants experienced roughly the same amount of cognitive workload. Due to the limitations of this study, however, further research must be conducted before confirming this trend.
Finally, it should be noted that many study participants were dropped from the heart rate analysis due to poor/incomplete data. In addition, the sample used in this study was predominantly young and educated, which may not be indicative of the population. Despite these instances, the researchers believe that this analysis is a promising new technique for evaluating complex, dynamic human behavior.
Conclusion and Future Directions
The exact impact of modality and content temporality, while inconclusive in our results, may still impact an individual’s trust and workload in an automated vehicle. However, our results are simply indicative of young, tech-savvy drivers. In the future, researchers could explore differences between age groups, more modality distinction groups, and using time series data to understand how complex attitudes change in real time.
