Abstract
The modeling of a new generation of agent-based simulation systems supporting pedestrian and crowd management taking into account affective states represents a new research frontier. Pedestrian behaviour involves human perception processes, based on subjective and psychological aspects. Following the concept of pedestrian environmental awareness, each walker adapts his/her crossing behaviour according to environmental conditions and his/her perception of safety. Different pedestrian behaviours can be related to subjective mobility and readiness to respond, and these factors are strongly dependent on the subjective interaction with the environment. Having additional inputs about pedestrian behaviour related to their perception processes could be useful in order to develop a more representative pedestrian dynamic model. In particular, the subjective perception of the safeness of crossing should be taken into consideration. In order to focus on the pedestrians’ perception of safe road crossing and walking, an experiment in an uncontrolled urban scenario has been carried out. Besides more conventional self-assessment questionnaires, physiological responses have been considered to evaluate the affective state of pedestrians during the interaction with the urban environment. Results from the analysis of the collected data show that physiological responses are reliable indicators of safety perception while road crossing and interacting with real urban environment, suggesting the design of agent-based models for pedestrian dynamics simulations taking in account the representation of affective states.
Introduction
According to the World Health Organization (WHO) [48], more than 1.35 million people lose their life on the streets. More than half of these victims are pedestrians, cyclists or motorcyclists that fall into the category of vulnerable road users, namely the category that is most at risk when speaking of road accidents. This can be a consequence of the fact that this group is composed mainly by children and elders, and that they are the only one not protected by some kind of external structure (i.e. a car’s frame, motorcyclist gear and helmet) when on the streets [12].
In order to protect especially these frail categories, it is important to properly study road safety in order to analyze the pedestrian–vehicle interaction. Since a popular procedure is to perform “in-vitro” experiments using agents in different simulation environments, profiling the attitude that people engage in while being a pedestrian is important to design a more realistic behaviour for the agents. This profiling could help the agents in emulating more closely and accurately people’s attitude in their simulated environment.
While defining an agent-based model to describe pedestrian behaviour, it is necessary to select proper pedestrians’ characteristics to contribute to the detailed behaviour of agents. Variables that are usually considered are: speed, preferential directions, reactions to the presence of obstacles, nearby persons and/or attraction points. These variables, when considered in real life, involve so many different interactions and dependencies that it is hard to obtain precise mathematical solutions. Moreover, in real life environments, human perception processes based on subjective and psychological characteristics are also involved. In particular, human beings decide to approach and react to the presence of the nearby persons or vehicles differently, also according to his/her psychological state, personal trait, nature and experience [8].
The role of emotions in producing realistic agent behaviors is becoming increasingly crucial. Among different computational models of emotions, the Ortony Clore Collins (OCC) model [27] is one the most popular. It relies on cognitive appraisal theories, where emotions are considered as reactions resulting out of an evaluation of the agent-environment relationship.
An interesting and concrete attempt to include human emotions in an agent-based model was performed by Riaz et al. [32]. They proposed an enhanced emotion enabled cognitive agent (EEEC_Agent)-based controller to help autonomous vehicles (AV) avoiding rear-end collisions. They introduced fear intensity as a decision pivot to help the AV make more flexible decisions regarding acceleration, deceleration and brake execution. They proposed an agent architecture with a suitable fear emotion elicitation and generation mechanism using the OCC model, and then verified their proposal through NetLogo simulation.
Supported by these positive results, the work here presented intends to open new perspectives to include human perception processes and emotions to model the pedestrian–vehicle interaction. It is evident that different kinds of pedestrians have different response time, speed and approach to walking in the streets, and having additional inputs about their behaviour could be more informative than just their sheer visual monitoring. Different pedestrian behaviours can be related to subjective mobility, and readiness to respond, and these factors are strongly dependent on the subjective interaction with the environment.
As Riaz et al. reduced the spectrum of possible emotions considering only fear, in this work we limit our analysis to stress as a significant indicator of safety perception. We follow the studies of Lazarus and Folkman [24], where they argued that stress derives from the stimulus-response reaction. Within this perspective, stress can be seen as a defensive reaction that is automatically activated to protect oneself from dangerous events [40]. Arousal is a physiological and psychological state that can be related to sensory alertness, that is thus activated in the interaction between a pedestrian and the environment as a defensive reaction to preserve safety, which is the connotation of stress here adopted. We thus here propose to consider arousal in different walking conditions to obtain useful indications on safety perception.
Our investigation in this direction is supported by the promising work of Leite et al. [25], that measured physiological arousal to quantify perception of safe and unsafe virtual environments considering a population of young adults and elderly people. Arousal can be effectively quantified by several physiological responses, which can be described as bodily reactions automatically activated by the autonomic nervous system and not controlled by the person. In particular, Galvanic Skin Response and Heart Rate can be considered reliable signals to measure arousal.
In order to incorporate affective parameters in the development of agent-based models, a formal design of experimental protocols and sets is crucial both for assessing the validity of the model and facing data and approaches coming from the related scientific community [5]. Movement and physiological data, coming from the related wearable sensors, need to be carefully tested through observations, interviews and rigorous experiments, both “in-vivo” (in a selected portion of the real world) and “in-vitro” (inside a formally designed experimental set, namely under laboratory condition) [4,33].
To pave the way of an affective agent modeling of pedestrian–vehicle interactions is thus mandatory to define a proper experimental protocol and data acquisition in real life environment, and the main contribution of this work is in this direction. From this experiment, several hypotheses should be verified in order to make the affective agent model feasible. In particular, the main hypotheses are:
Different walking conditions, such as crossing in the presence of vehicles or walking on the side walk, relate to different safety perceptions, inducing different uncontrolled physiological reactions in pedestrians.
Arousal is activated when interacting with the environment and can be adopted as a measure of stress considered as a defensive reaction to protect oneself from dangerous events.
It is possible to categorize pedestrians with respect to their physiological uncontrolled reactions.
Behaviour of pedestrians can be different with respect to these categories.
Once we have defined categories of pedestrians, we can tune the variables and decision rules of agent based models with proper parameters depending on the affective attitude of being more or less stressed by the interaction with the environment and in particular with vehicles. We expect that speed, direction, tendency of aggregation will be mainly conditioned by this affective attitude and will require, in the design of a proper agent based model, to be differently modulated for groups of pedestrians with similar affective reactions to safe or unsafe interaction with the environment.
The structure of the paper is the following. Section 2 is related to background and previous work in this field, in order to better frame our work in the right context. In Section 3 we describe the in-vivo experiment that was carried out in an uncontrolled outdoor environment. Signal processing on physiological data, in particular to remove noise and normalize the responses of the subjects, is described in Section 4. This step, together with proper feature extraction, is required to analyze the data in an intra and inter subjects comparison with the aim of finding characteristic patterns corresponding to different affective states. In Section 5 the subjective responses to the self-assessment questionnaires as well as statistical inferences from physiological data are presented, while in Section 6 the hypotheses here presented are verified and discussed. Finally, in the Conclusions, final remarks and possible future developments for this approach are drawn.
Background
The development of models for intelligent agents that are able to incorporate, use and express affects into reasoning and interaction processes or supporting decision making activities is becoming a novel research area in the field of agent-based simulation [31] [35].
Pedestrian simulation
Numerous studies have been developed during recent years in order to investigate different aspects of the pedestrian behaviour. The focus has been on non-signalized pedestrian crossings [11], evasive behaviours in pedestrian interactions, flows and counter-flows in crowds [26] and pedestrian–vehicle interaction in proximity of an unsupervised crossing [18]. As it is highlighted in [18] and in [11] especially, the heterogeneity of the system entities is relevant in order to properly identify the pedestrians’ microscopic (i.e. individual) dynamics. And that is because aggregated dynamics can be of interest for who is regulating the system in its entirety.
In the case of agent-based and crowd and pedestrian dynamics simulations, the modeling of a new generation of systems supporting crowd management that takes into account affective states represents a new research frontier, involving also many human disciplines [46]. Numerous works have already started working towards the integration of emotion in pedestrian agent simulation systems, ranging from simple flows to evacuation plans simulations.
For example, looking at the work done in [29], the agent simulation system was integrated with additional information about human behaviour, allowing each agent to perform according to a specific role, ability, stress level, emotion and so on. This important integration of characteristics and communication was done in order to achieve individualistic behaviour and to obtain a realistic way to spread information across the environment. The same importance to this integration is attributed by [18] which, in its analysis, underlines how pedestrian are heterogeneous and need to be treated as such also in simulations. Moreover, in [28], the agents are given more characteristics in order to act more naturally and realistically, obtaining different psychological (e.g., impatience, panic…) and physiological (e.g., locomotion, energy level) traits in order to display individual heterogeneous behaviour during the simulation. In [21], recognizing how humans do not follow a strict list of goals but, instead, change and adapt their behaviour accordingly to emergent conditions and unexpected events, a prototype emotion model to be implemented in a pedestrian simulation system is presented. The required time to perform a task and the available time were introduced, presenting the agent with the concept of “time pressure” to influence their behaviour, alongside the compulsory, time critical and elective tasks which also provided the agents with the concept of “urgency”.
Shifting to the evacuation plans simulation, there are already some works that have tried implementing some sort of emotion or affect in their agents in order to obtain more realistic and informative simulations. In [42], the WALK project is presented to address the need to build a reliable multi-agent model for human behaviour prediction in critical situations, including factors like emotional stress and complex interactions. The fundamental research brought onward regards the impact of emotions on crowd dynamics, and in this case a multidisciplinary approach and an effective simulation system are two of the fundamental steps needed to further the research. The work presented in [1] finds its objective in studying the influence of emotions on human behaviour in crowds, and more precisely on the influence that their dynamics and propagation from an agent to another have. To the authors, it is clear that adding emotions to the agents provides huge benefits both to the agents behaviour and to the simulation quality, since emotions play a very important role in human beings’ life by influencing their decision-making and reasoning processes just like also their interactions with others. Another example is [13], where the pedestrian dynamics in counter flow integrated with the effect of panic propagation was explored. In this case the only emotion present in the simulation was panic, or fear, but it already provided interesting insights for managing a crowd during a stampede, in order to study the possible damages and to properly organize a safe mass evacuation. In [44], the ESCAPES system is presented, a multi-agent simulation tool that includes four features for simulation modeling in order to create more realistic outputs: different agent types, emotional interactions, informational interactions and behavioural interactions. The different agent types allow the system to create, for example, individuals, families, first time visitors or experienced agents, and the agent interactions allow to show the spread of knowledge, emotional contagion and social comparison throughout the crowd. Finally, the work in [51] presents a system based on a multi-agent BDI architecture enhanced with the Five Factor model of personality, also called OCEAN model, and the OCC model of emotions. This simulation tool takes into account how every agent possesses Openness, Conscientiousness, Extraversion, Agreeableness and Neuroticism in varying degrees, and the concept of Perception that generates Beliefs and of Decision making that create Desires to be fulfilled through Intention are introduced. The involved OCC model also brought the modeling of five positive/negative emotion couples: Joy/Distress, Hope/Fear and Pride/Shame concerning the agent itself, while Admiration/Reproach and SorryFor/HappyFor concerning other agents. The interesting results of this approach is that the agent emotional state may influence its perception, affecting for example its ability to notice an exit sign or an obstacle, while relationships between agents like reproach or admiration can cause a communication message to be ignored or regarded respectively, providing a more realistic interaction between agents and a more reliable simulation. Colombi et al. [9] has introduced in a discrete Helbing-like approach the concept of pedestrian environmental awareness, where each simulated walker is set to change his/her target destination according to new information learned from the surrounding environment and that different perceptions of the surroundings can lead walkers to react in a different way. This approach and the model developed in [8] support our proposal of including perceptual aspects in modeling pedestrian dynamics.
Physiological affect recognition
Physiological responses are bodily reaction automatically activated by the autonomic nervous system and not controlled by the person, rising when the body is reacting to a particular external affective elicitation, and can thus be adopted to measure the level of stress affecting pedestrians in walking and road crossing, for example during dynamic collision avoidance. In particular, arousal can be related to sensory alertness, activated in the interaction between pedestrian and the environment as a defensive reaction to preserve safety.
Physiological signals are already widely used in the emotion and affect recognition fields, having proved their effectiveness and usefulness in this area [7,14] besides, for example, daily activities recognition [50]. Looking at the emotion recognition field, they have been used to understand the physiological changes in people in reaction to different kind of stimuli, being them audio [23], video [2,37] or images [45]. The acquired physiological signals vary greatly depending on the work, but the objective is more or less always the same: to identify differences in the physiological response of the subjects related to stimuli aimed at evoking and inducing different emotions.
These autonomous signals are also greatly used in stress recognition, which is the area we are most interested in. Differently from the usual experimental procedures carried on for the emotion recognition task, when talking about stress recognition the variety of the experiments can be much greater: there are many different situations in which stress can be elicited and measure that are not limited as in showing or acting a certain emotion. For example, in [38], the experimental procedure involved a slight modification of the Montreal Imaging Stress Task [10], a test created to evaluate effects of psycho-social stress on physiology and brain activation. In [20] and [39], different cognitive tasks were designed in order to compare the physiological response of a person while stressed for the activities and while at rest. Other papers used data coming from real-life environments: in [19], researchers monitored the activity of nine call center employees with sensors in order to trace their stress levels during their turns, using self-assessment questionnaires as help, while in [17] they analysed the data coming from the Physionet dataset, obtained from three different driving tasks performed by healthy volunteers. There have also been some more particular approaches to the problem: for example, in [36], the stress of the participants was monitored through a wrist band during a period of five days, registering through a mobile app their health, mood, alertness and sleep schedule of the day; in [47], subjects had their physiological signals recorded for stress detection while using websites with poorly-designed or well-designed interface; finally, in [43], the objective of the study was to focus on stress reduction, monitoring stress levels through physiological sensors before and after the exposition to relaxing videos and audios following a precise relaxation protocol.
New approaches of Artificial Intelligence that rely on affective computing are becoming crucial to design new generations of computer-based systems supporting the creation of services for the future cities [49]. Developing new models incorporating data and dynamics coming from affective parameters could be investigated through the involvement of the scientific community devoted to Affective Computing [30].
The experiment
In order to focus on the pedestrians’ perception of safe road crossing and walking, an experiment in an uncontrolled urban scenario has been carried out. The next subsections show how the experimentation was carried out, who was involved and what data were obtained during the various trials.
The environment
The chosen environment contemplated an unsupervised crossroad on a two-way road, not far from the main buildings of the Unviersity of Milano-Bicocca. Figure 1 shows this experimental environment, and the zebra crossing at the centre of our experiment is highlighted with a red rectangle.
The chosen intersection with the selected crossroads highlighted in red.
This crossing is considered moderately dangerous for pedestrians for the following reasons:
The crosswalk is located on a very busy road, given its location near offices, the university and a shopping mall.
The crosswalk is unsupervised, with no traffic lights for neither the cars or the pedestrians.
The numerous parking lots positioned alongside the intersection limit the view of the pedestrian before and while crossing the road.
Different vehicles travel alongside this road, ranging from bicycles to cars, trucks and buses.
Since we wanted to test the subjects’ safety perception while traversing a stressing crosswalk, the one located at this crossroads presented some difficulties that could effectively elicit a stressful affective state. The only external indication that participants were given was related to the presence of vehicles on the road: in fact, subjects were asked to start the crossing task when there were vehicles approaching the zebra crossing, in order to make the experience even more realistic.
For this preliminary study, the subjects involved in this experiment were chosen from a same social and age group. A total of 14 participants were engaged, 7 males and 7 females, aged between 20 and 26 years (mean = 24.42, standard deviation = 1.65), all university students enrolled in one of the scientific faculties at the University of Milano-Bicocca. Because of their usual attendance on campus for lessons, all of the students were familiar with the chosen intersection, especially since most of them usually crossed the street in that same location in order to reach the Department of Informatics, Systems and Communication.
The experimental procedure was explained in all of its parts to the participants in order to let them know what their tasks consisted of, and to let them grow accustomed with the procedure. All of the subjects in this experiments were volunteers who provided informed consent, and the study had been previously approved by the Ethical Committee of the University of Milano-Bicocca.
Physiological data
In order to measure the participants’ condition during the experiment, we chose sensors aimed at recording the physiological responses of the participants, focusing in particular on three different signals: the Galvanic Skin Response (GSR), also known as Skin Conductance (SC), which is connected to sweating and perspiration on the skin and is a reliable and widely used stress indicator; the Plethysmography (PPG), that measures the blood volume registered just under the skin, which can be used to obtain the heart rate of the subject; the Electromyography (EMG), measured as surface electromyography, which measures the muscle activity of the person.
These physiological signals are some of the most used in the literature concerning affective recognition, and researchers have also started using them specifically for stress recognition given their reliability in this area [19,36,38,39]. On the other hand, regarding the EMG signal, it can be effective in defining pace and speed of the pedestrians, and can be correlated to the affective categories of subjects obtained from the analysis of physiological signals (PPG and GSR).
In order to properly record these three signals, two different sensors from the Irish company Shimmer1
have been adopted: the Simmer3 GSR+ unit, tasked with recording both GSR and PPG data, and the Shimmer3 EMG unit. These low-cost wearable sensors were already utilized in different experiments concerning physiological signals analysis and affective state recognition with encouraging results [6].Figures 2 and 3 show how they were worn by the subjects during the experimentation: the GSR electrodes were placed on two fingers of the non-dominant hand of the subject, the PPG light-emitting diode was placed on an earlobe, and the EMG electrodes were placed on the leg, measuring the muscle activity of the medial gastrocnemius muscle and of the anterior tibial muscle.
One of the subjects wearing the GSR and PPG sensors.
One of the subjects wearing the EMG sensor.
Human affective states are influenced not only by environmental stimuli, but also by several subjective characteristics. In particular, personality traits strongly condition the affective responses of the person [22] and [41].
To profile an aspect of human being personality that could be related to the defensive reaction to preserve safety while crossing a street, we introduced the Rosenberg Self-Esteem questionnaire [34] into our experiment. This survey was selected in order to assess the person’s self-esteem and to acquire data that could prove useful to understand if people with different profiles react differently when faced with the crossing task.
Furthermore, to better correlate physiological responses to safety perception and different environmental conditions, we have also designed an ad-hoc self-assessment custom questionnaire about the crossing task.
The two selected questionnaires are described below:
Rosenberg Self-Esteem Questionnaire: this survey measures the appreciation and confidence that a person has towards him/herself. The subject needs to say how much he/she agrees with the presented sentences on a Likert scale from 1 (Absolutely not) to 4 (Absolutely yes). The items of this questionnaire are the following: I feel that I’m a person of worth, at least on an equal plane with other. I feel that I have a number of good qualities. All in all, I am inclined to feel that I am a failure I am able to do things as well as most other people. I feel I do not have much to be proud of. I take a positive attitude toward myself. On the whole, I am satisfied with myself I wish I could have more respect for myself. I certainly feel useless at times. At times I think I am no good at all. Custom questionnaire about the crossing task: this questionnaire was used to collect subjective perception about the crossing task, such as the stress level of the participant, his/her confidence in drivers, disturbing elements etc. The participant needs to classify every item of this survey as NULL, LOW or HIGH. The items of this questionnaire are the following: Stress level during the crossing. Confidence level towards the cars during the crossing. Interference level brought by other means of transportation during the crossing. Influence level brought by other pedestrians. Confidence level in the crossing without traffic control or traffic lights. Confidence level in the crossing with disturbing elements (parked cars, partially blocked view…)
Experimental protocol
The experimental protocol was composed of four principal parts: the Rosenberg’s questionnaire filling, the walking task, the crossing task and the assessment questionnaire filling. While the first questionnaire was compiled only at the beginning of the experiment, the other three parts that composed the core of the experimental session were repeated four times by every subject, each task divided from the next by a 30 second baseline recording in which the participant had to remain standing still.
The entire experimental protocol is extensively explained below:
Questionnaire filling: Rosenberg Self-Esteem Scale
Experiment Core: repeated 4 times
Walking on sidewalk (non-stressing task), as depicted in Fig. 4.
30 seconds baseline recording, where the subject had to stay straight up and still to record his/her physiological response in absence of tasks.
Crossing the road and coming back at the start point (stressing task), as depicted in Fig. 5.
30 seconds baseline, same as before, also intended to bring the subject back to a neutral state before the next crossing.
Crossing questionnaire filling.
End of trial
One of our subjects during one of her walking tasks.
One of our subjects during one of her crossing tasks.
Some notes to the procedure:
In order to better understand the participant’s behaviour, this task was also filmed with a full HD camera. Every participant gave consent to the recording of their crossings.
The participants were instructed on the procedure they had to follow for the data acquisition once the sensors had been placed.
The experiment had a total duration of approximately 20 minutes, an understandable length since the subject, for their crossings, had to wait for cars to show up and approach the intersection.
The lengths covered by the subjects during the walking task equals the length covered while crossing up and down the street during the crossing task.
During the experiment, the participants were asked to move their arms as little as possible, since movement noise can be of great disturbance in recording GSR and PPG data, especially with finger electrodes.
The experiments have also been video recorded in order to obtain visual data of the participants’ trials.
During this in-vivo data acquisition, a problem emerged: because of the very low temperatures registered during the trial of three of the participants, the GSR+ sensor had some difficulties recording the GSR and PPG signals, thus rendering those three recordings unusable for the final analysis. This likely happened because the GSR+ sensor has an optimal temperature range of acquisition between 20°–28°C in order to function properly, while the registered temperatures during those days were around 8°–10°C.
After completing the experimental phase of the study, we started processing the obtained data in order to work on the analysis on clean and usable data.
Preprocessing
The raw signals obtained during the experimentation need to be preprocessed and cleaned and proper features need to be extracted from the signals before performing the analysis, since the original recordings may contain noise and artifacts (given by the environment, the sensors and the subject) that can throw off the results.
The GSR and PPG signals were sampled with a frequency of 128 Hz, while the EMG signal with a frequency of 512 Hz. The preprocessing was slightly different for the two types of data: for the GSR and the PPG filtering step we used a zero-phase filter, in order to properly remove the noise and the possible high-frequency artifacts that we could expect; for the EMG, instead, we decided to use a zero-lag Butterworth bandpass filter with a cut-off frequency of 20 Hz. Examples of GSR and EMG raw and processed signals can be seen in Figs 6 and 7.
Example of raw (upper box) and filtered (lower box) GSR signal.
Example of raw (upper box) and filtered (lower box) EMG signal.
The filtered signals were then normalized with a z-score function, in order to have all of the signals of all the participants to compare in a same reference range, and were then split into different segments following the markers directions registered during the experimentation. This way, for every subject, we obtained a total of 21 segments:
4 crossing segments
4 walking segments
8 (
5 questionnaire segments
After this step we then proceeded to display all of the signals overlapping with the markers. We obtained for every of the remaining 11 participants a graphic similar to the one displayed in Fig. 8, depicting clearly the subject’s physiological response for the entire duration of the experiment.
The graphic depicting the signals and the markers of a single participant. The orange track (bottom line) depicts the EMG signal and the blue one (top line) the GSR signal. Q indicates the Questionnaire periods (missing in the smallest windows), W the Walking periods, B the Baseline periods and C the Crossing periods.
With such a visualization, every performed task can be easily distinguished along the physiological responses. It is already fairly clear how the muscle activity can be acquired by the EMG signal, given how periods of walking and crossing are very different from periods of standing and baseline, and the GSR peaks in skin conductivity and activation correspond to the crossing periods as we would expect.
As we introduced them before, the event markers were created ad-hoc beforehand to be used during the experiment. Every marker correspond to a different phase/task of the experimentation, and they are differentiated by their heights. The height-marker correspondence is the following:
We decided to calculate a total of 13 different features from the data. Table 1 shows all of the obtained features and for what signals we computed them in order to perform the results analysis, since some of them were signal-specific.2
IBI is the Inter-Beat Interval feature, while RMSSD is the Root Mean Square of the Successive Differences feature.
Table summarizing all of the selected features we computed for the three analysed physiological signals
The only thing that needs to be addressed is that, in order to correctly compute the features for the GSR, we firstly had to separate the two different components of this signal: the Skin Conductance Level (SCL) and the Skin Conductance Response (SCR). The SCL comprises all of the low frequencies of the GSR signal, thus giving the general trend of the signal, while the SCR includes all of the high frequencies and shows clearly all of the peaks that can be categorized as “natural peaks” or “elicitation peaks” (that are more relevant in our analysis since they highlight the person’s physiological response to external events and elicitations). In order to do this, we derived the SCL by using a low-pass filter at 0.05 Hz, obtaining the tonic part of the GSR, and the SCR was then derived using a high-pass filter with the same frequency, thus generating the phasic part. This distinction can be seen more easily in Fig. 9.
An example of GSR signal (upper box) with its phasic part (middle box) and its tonic part (lower box).
All of the GSR features were calculated from the phasic part of the various GSR signals whit the exception of the Regression Coefficient, which was obtained from the tonic part since it contained the necessary information about the signal slope.
Once the data were processed and the features were computed, we passed on to perform the data analysis on both the computed feature and the questionnaires results obtained during the experimentation.
Rosenberg’s questionnaire analysis
Looking at the results obtained from the Rosenberg’s Self-Esteem Scale reported in Fig. 10, what we understood about our subject sampling was that all of the participants had a very good self-perception, tending to approach in a serene way new tasks given them. This was to be expected since, as we said before, the subjects we took into consideration were young students in good health. Other than putting them all at the same level in approaching the designed experiment, the results also confirm the homogeneity of the population considered in the experimentation, reducing the variables to be considered when investigating the physiological results.
The results obtained from the Rosenberg’s questionnaire analysis.
One of the first analysis performed on the obtained features was a Kruskal–Wallis test, a non-parametric method to assess the equality or the diversity of two groups medians, in order to understand if the physiological feature distributions coming from different tasks recorded during our experimentation (baseline, walking and crossing) were statistically different. This was done in order to corroborate or dismiss the hypothesis that the physiological response of a subject can differentiate between different affective states of the person.
The Kruskal–Wallis test was chosen after verifying that our data did not belong to a normal distribution, thus rendering the utilization of other tests, such as the ANOVA, not suitable for the analysis.
The Kruskal–Wallis test provides a null hypothesis, for which two distributions provided as input are similar enough to be considered as coming from the same initial distribution. If the returned result of the test, the p-value, is lower than a certain significance level (that for us was fixed as
The first test we performed was about comparing the features distributions of the Walking tasks with the ones from the Crossing tasks, and Table 2 shows the obtained results.
Kruskal–Wallis p-values: walking-crossing comparison. Table showing the results of the Kruskal–Wallis test comparing the feature values of the walking tasks with the ones from the crossing tasks
Kruskal–Wallis p-values: walking-crossing comparison. Table showing the results of the Kruskal–Wallis test comparing the feature values of the walking tasks with the ones from the crossing tasks
The bolded values highlighted in the table are the ones that were lower than the significance level we selected. In this case, we can see how almost half of the performed tests comparing feature distributions from different activities were found to be genuinely diverse. This kind of results corroborate our hypothesis (number 1 in the list in the Introduction) that different walking conditions such as crossing in the presence of vehicles or walking on the sidewalk, related to different safety perception, induced different uncontrolled physiological reactions in pedestrians.
Kruskal–Wallis p-values: crossing-baseline comparison. Table showing the results of the Kruskal–Wallis test comparing the feature values of the crossing tasks with the ones from their related baseline
Kruskal–Wallis p-values: walking-baseline comparison. Table showing the results of the Kruskal–Wallis test comparing the feature values of the walking tasks with the ones from their related baseline
Answer percentages for every evaluation category obtained for the custom questionnaires about the crossing experience of our subjects.
The same, if not better, response was also achieved from the comparison of Crossing and Baseline and of Walking and Baseline, whose Kruskal–Wallis test results are reported in Tables 3 and 4.
Another thing that emerges from the analysis of the above mentioned tables is that the PPG and the EMG signals do not seem to be really correlated to an affective state (stress or non-stress) but to be more connected to movement in general: comparing Table 2 with Tables 3 and 4 it is clear that the distributions coming from walking and crossing tasks for these two signals, the PPG in particular, seem more similar (thus not passing the KW test) than in the other two cases.
After the signal analysis, we decided also to perform a sample checking analysis in order to better understand what impression of the crossing task the participants had. Therefore, the first thing we did was gather all of the custom questionnaire answers for all of the subjects and all of their crossing tasks, thus obtaining a total of 56 answers to every question we created. From this set we then computed the percentages of NULL, LOW and HIGH answers given by the participants, obtaining the graphic that can be seen in Fig. 11.
As we can see, the majority of the crossings delivered low to null stress to the subjects, and only a few high stress levels were reported through the custom questionnaires after the task. This data is not unexpected since, as we previously highlighted, the subject sample for this experiment was narrowed down to healthy and young students who are also accustomed to crossing this particular intersection while walking through the university campus. Having to face a challenge that was already familiar to them may have let them perceive the task as not as dangerous as it could have been for someone who never saw that intersection before.
Pearson correlation matrix between the answers of the self-assessment questionnaire.
Figure 12, on the other hand, shows the correlation matrix obtained by checking the relations between the answers, a test performed using Pearson correlation index. From left to right, and from low to high, we have these categories: Stress Level, Confidence (Vehicles), Interference (Other Vehicles), Interference (Other Pedestrians), Confidence (Crossing without Controls), Confidence (Crossing with Disturbances). We can see how the highest Pearson correlation coefficient (0.4574) is between Confidence (Crossing without Controls) and Confidence (Crossing with Disturbances): this can mean that many participants were less confident in crossing the street for both these factors. The lowest Pearson correlation coefficient (−0.4089), on the other hand, is between Stress Level and Confidence (Crossing without Controls).
Even if from the self-assessment questionnaires emerges that the subjects involved in the experiment were not particularly stressed by the crossing tasks, the physiological data clearly shows different patterns with respect to the different activities as well as differences in the feature distributions that are statistically significant. These considerations are important hints towards the adoptions of physiological signals as indicators of uncontrolled affective reactions of subjects in the pedestrians-vehicles interaction.
The results of the analysis on the acquired data are encouraging and permit to evaluate the formulated hypotheses, also giving the direction towards the definition of an affective agent model.
The first hypothesis:
Different walking conditions, such as crossing in the presence of vehicles or walking on the sidewalk, related to different safety perception, induced different uncontrolled physiological reactions in pedestrians
has been directly verified from the result analysis, as already pointed out within Section 5.
The second hypothesis:
Arousal is activated when interacting with the environment and can be adopted as a measure of stress considered as a defensive reaction to protect oneself from dangerous events
was partially confirmed by our results. In fact, there is evidence of activation differences in physiological responses comparing stress-free tasks (such as baseline and free walk on the side walk) with more engaging tasks (crossing a busy, double side road) that were evaluated in the 66% of the instances of low or high stress. These self-assessment results are not surprising as not all the time there was a significant traffic flow while the subject was crossing.
To better confirm this hypothesis, further data should be acquired in order to define a classification model for recognizing different levels of stress. Since only 14 subjects were involved in our experiment, with 4 repetitions of each task, we have a total of 56 instances of crossing: a classifier on three classes (using the self-assessment responses as ground truth) could not be significant if trained with such a small dataset.
The third and fourth hypotheses:
It will be possible to categorize pedestrian with respect to their physiological uncontrolled reactions
Behaviour of pedestrians can be different with respect to these categories
and
can not be directly verified with the data acquired with this experiment.
However, we also performed a different experiment in a controlled laboratory environment, this time involving both young adults and elderly people, where these hypotheses could be better explored. There different walking conditions have been investigated, including in particular dynamic collision avoidance. The analysis of this experiment, partially described in [15], reveals significant differences in pace and leg movements between young adults and elderly, analysed using the EMG data. In particular, considering young adults, it has been noticed an increase of the EMG signal power in correspondence to the collision avoidance events, caused by the effort of the subject to accelerate and safely pass the obstacle. On the other hand, analyzing the power of the EMG signals for the elderly, it has been observed a decrease in signal power during collision avoidance events. This decrease in power is related to the observed evidence that the elderly participants decelerated or even stopped while facing a collision avoidance task. This analysis proves that elderly people are used to keep a more careful behavior than the young adults and, in particular, it suggests that a subject categorization is feasible.
To further corroborate these working hypotheses and increase the acquired data, we have also planned to proceed in experimental acquisitions in a real life urban environment (as soon as the COVID-19 emergency will permit these activities). We have already designed two more experimental protocols, described in [16], where both young adults and elderly people will be involved.
Once we have acquired more data, especially including different types of subjects (i.e. students, adults and elderly people), it will be possible to better define pedestrians’ categories with respect to their perception of safety while crossing and walking in an urban environment.
Being more or less stressed during an interaction with an unsupervised environment, in particular populated by vehicles, can translate into different pedestrian behaviours. In order to understand these behaviours, analysis of the video recordings of the sessions will be supported by the information about pace and movements coming from EMG and inertial data.
These observations and correlation to affective categories can thus guide decision rules of agent based models and properly weight variables such as speed, directions, way of interaction with vehicles and other pedestrians.
Conclusions
The paper illustrated an in-vivo experiment to evaluate the pedestrian–vehicle interactions from an affective point of view, and collected physiological data have been proven to be reliable indicators of different affective states during walking and road crossing. The results of this research will allow the design of agent-based models for pedestrian dynamics simulation taking in account the representation of affective states, more precisely the stress elicited during road crossing. Moreover, parallel experiments conducted in in-vitro environments (as illustrated in [4]) will allow a deeper comparison with the collected data, in order to develop affective models for agent-based approaches to the study of pedestrian dynamics.
It will also be important to explore in follow-up experiments how affective states and quantitative measures can be correlated. This kind of connection between a pedestrian’s mood and his/her reaction times, speed and direction may bring great value to agent simulators, being especially useful to help calibrating them for different types of pedestrians. Moreover, as the whole experiment has been video recorded, the analysis of these videos will be helpful to further analysed pedestrian behaviour and related physiological responses in order to integrate our findings in pedestrian dynamic modelling.
It would be interesting, and would also provide insights on different pedestrians’ reactions, to replicate the same experiment involving different populations, such as adults and elderly people. As we have seen, students familiar with the intersection have approached the experiment in a certain way and have reacted also according to their previous experiences of the chosen environment. Another experimentation is already in the plans, for now involving elderly people in another location, to start gaining data in this direction and explore further the information we can obtain from real life to contribute the design of better agents for “in-vitro” simulation.
Finally, as in future scenarios, self-driving vehicles will circulate in urban environments, pedestrian safety perception while crossing in the presence of an autonomous car should be investigated. To this end we have designed an experiment in a controlled urban-like crossing environment and we have prepared a proper self-driving vehicle [3].
Footnotes
Acknowledgements
This research is partially supported by the FONDAZIONE CARIPLO “LONGEVICITY-Social Inclusion for the Elderly through Walkability” (Ref. 2017-0938) and by the Japan Society for the Promotion of Science (Ref. L19513). We want to give our thanks to Maria Elena Manisera and Gianluca Toffanin, for their supporting work during the experimentation and data analysis.