Emotion collector,a wearable multi-sensor band to recognize fear

Abstract

In this study, we classified emotional responses of a user by measuring the physiological response signals with the Emotion Collector, which is a wearable band equipped with a multi-sensor device. We collected physiological signal data to measure the delicate hand movements, heart rate, and skin tension of the user analysis with a focus on picking out fear among the user’s emotions. In experiments classifying fear factors into five categories, we were able to identify the characteristics of the user’s physiological responses according to the type of fear. We were also able to find the similarities and differences of physiological and psychological results for fear. The Emotion Collector is very easy to use, robust, and suitable for mobile and long-time logging of data. It can easily be integrated into other systems or applications. The system is designed for use in emotion research as well as in everyday affective applications such as user-centered service and content.

Keywords

Wearable technology emotion fear multi-sensor physiological data

1 Introduction

With the Fourth Industrial Revolution, wearable technology has been developed by the convergence of various fields such as industry, education, training, and entertainment. In particular, wearable bands that can easily obtain user-centered information such as the user’s life log and health information by being worn on the wrist are rapidly increasing in application in the medical and fitness fields. A fitness band such as Fitbit or Jawbone acts as a user’s exercise coach by measuring their movement and heart rate. In addition, wearable technology can continuously measure various medical data during daily life outside the hospital and obtain previously unattainable data.

Thus, wearable technology is optimal for developing user-centered contents and services by providing the user with suitable services through personal information obtained from devices worn on the user’s body. However, the above wearable technologies basically provide contents and services based on intuitively collected information on simple activities such as the user’s movement and heart rate. That is, there has been little research so far on a wearable band that can provide services suited to the user’s emotional state by comprehensively evaluating the user’s physiological information. Identifying the user’s emotional state is very important to develop the most optimized services and contents with wearable technology. In particular, wearable technology has recently been actively researched and utilized as a tool for human–computer interaction (HCI) with the growth in the virtual reality (VR) and augmented reality (AR) markets. Thus, in this study we developed a wearable system that can infer the user’s emotional state by collecting and analyzing physiological information, which we call the Emotion Collector. Among the human emotions, we focused on fear, which has clear physiological response signals, and subdivided it into five categories. The data can be analyzed to infer the emotional state of the user, which can be utilized in various services and applications in the future.

2 Related works

Emotion recognition based on images such as facial expressions and human movements or on audio speech recognition has been actively studied in the field of HCI. Techniques for recognizing human emotions through physiological signals such as respiration, skin conductance, and brain waves have recently been drawing attention [1, 2]. In facial expression and speech recognition, the emotions of person are recognized by learning and patterning responses. However, the recognition rate drops significantly if no formal pattern is shown [3] and may vary depending on the user’s social background. On the other hand, physiological signals acquired not an artificial state but the natural emotional state because they are controlled by the autonomic nervous system [4]. In addition, these signals provide a direct indication of whether emotions affect physical health [5].

There have been many attempts to study how the emotions of audiences change in response to visual stimuli. For example, Palomba et al. [6] measured changes in the levels of fear, disgust, anger, sadness, surprise, and happiness by monitoring the heart rate changes in audiences who were exposed to unpleasant images. Furthermore, Alvarado [7] measured such as anger, disgust, fear, happiness, pain, and sadness using unpleasant images and a self-reported data collection method.

In existing HCI research, various emotions are classified by analyzing physiological signals with machine learning algorithms such as the linear discriminant function (LDF), support vector machine (SVM), and neural network (NN). According to previous studies, the accuracy of emotion classification is reported to be 40%–80% [5, 6]. The reasons for the low accuracy are that the optimal machine learning algorithm suitable for physiological signal data is not used, too many features are used instead of the optimal feature that clearly reflects the emotional response, or too many emotions are classified at once.

In this study, our aim was to identify the optimal algorithm that best classifies the emotion of fear by applying features extracted from physiological signals to a machine learning algorithm. This is because fear is suitable for dividing features in physiological response signals compared to other emotions such as joy and sadness and thus can be used in research on subdividing the type of emotion.

3 Physiological signals collector

In recent years, it has become possible to measure emotions using bio-signals, owing to the development of various physiological sensor technologies. In Sinha and Parsons” study [10], heart rate, skin conductance level, finger temperature, blood pressure, electrooculogram, and facial EMG were recorded while the subjects were visualizing the imagery scripts given to them to elicit neutrality, fear, joy, action, sadness, and anger. In Vrana”s study [11], personal imagery was used to elicit disgust, anger, pleasure, and joy from participants while their heart rate, skin conductance, and facial electromyogram(EMG) signals were measured. Likewise, there have been many researches that classify the different kinds of emotional status. However, no research focused solely on fear (and investigating and classifying responses based on determining its cause) has been conducted. In this study, we developed a wearable band that can measure the user ‘s hand movement, heartbeat, and skin tension to classify levels of human fear using physiological signals.

Research and development of various sensor technologies has made it possible to measure and analyze various types of physiological signals. In this study, we developed a wearable band “Fig. 1” that is capable of measuring the hand movements, heart rate, and skin tension of the user to classify human fear through physiological signals.

Fig.1

Prototype of Wearable Band ‘Emotion Collector’.

Existing systems for measuring physiological signals, such as electroencephalograms, electrocardiograms, facial expressions, and gestures, are because they install a separate sensor or camera device on the user’s head. Therefore, it is difficult to apply them to commercially available contents or services that require immediate interaction. However, the developed Emotion Collector is a multi-sensor band that can measure three different physiological signals at the same time when worn on one hand, and various physiological signal data of the user can be collected with a simple setting. In addition, the data from the wearable band are collected in real time by a mobile device application through Bluetooth communication. Another advantage is that a huge amount of data is collected.

The Emotion Collector is equipped with an inertial measurement unit (IMU) sensor that can measure the delicate movements of the user’s hand, a heart rate measurement sensor, and a galvanic skin response (GSR) sensor that can measure the degree of skin irritation. GSR sensors measure GSR by measuring the conductivity of the skin. When the participant is feeling strong emotions, it stimulates the sympathetic nervous system and releases much sweat from the sweat gland, which simply makes it simple to attach the electrodes to the two fingers.

The data can be used to infer the user’s emotional state. All physiological information is collected from a wearable wristband. The GSR sensor and heart rate measurement sensor are attached to an elastic band that can easily be worn on the user’s finger for free movement of the user.

While the collecting the physiological signals from the band, the Emotion Collector’s mobile applications show the changes of the data in numbers and graphs. After collecting all the signals, we analyze them to categorize the types of fear. We could find out the distinct features for the 5 types of fear.

4 Cause of fear

Fear is a feeling of dread toward a particular object or situation. These feelings normally occur when seeing a frightening scene or encountering a dangerous object, which causes various changes in the human brain and body. These changes can include the following: hair standing on end, cold sweats, bladder contractions (causing the urge to urinate), and an increase in heart rate. Furthermore, these changes are caused by the autonomic nervous system reacting to signals from the amygdala in the brain. The autonomic nervous system is found only in vertebrate species and reflexively regulates internal organs without self-awareness or conscious effort.

Fear is the most intense feeling that can be accurately observed through objective indicators of the human emotional response [12]. In essence, fear is a response to the sudden awareness of external, non-conflictual threats. Thus, fear can be considered an acute anxiety that occurs when an unconscious anxiety is transferred to a specific external subject [13, 14]. In the field of emotional research, fear has been widely analyzed and discussed in many studies [15].

Likewise, there have been many studies conducted to clarify the biological signal responses to fear; however, their results have been inconsistent. For example, one study [16] reported that fear reduces human heart rate, while others reported an increase [17, 18]. In addition, examinations of the onset of fear showed an increase in the skin conductance level (SCL) and skin conductance response (SCR) [17 , 20]. However, these studies only examined different physiological response patterns to fear and did not reveal the correlation between psychological scales and emotions.

5 Emotional stimulation and physiological signals

Twelve normal subjects (six males and six females) aged between 20 and 40 years (male: 32.2±0.8 years, female: 29±1.7 years) participated in the experiment. Physiological signals were measured while the subjects watched five classified clips from horror movies for 20 min. Experiments were conducted with sufficient notice of the emotion of fear in advance. Because this study involved human subjects, we received written consent and permission from the subjects.

As you see on “Fig. 3”, subjects watched movie clips through a laptop screen, and noise exposure to the outside environment was blocked by noise cancelling headphones.

5.1 Experiment on evoking fear

Clips from top box office horror films over the past decade were used as stimuli to evoke the emotion of fear. For the experiment, we classified materials that evoked fear in horror movies into five categories and selected the best movie scenes. Used movie clips pre-examined to validate for simulating a feeling of fear in advance. This stimulation averaged an average of 94% of the suitability (matching emotion between inductive emotion and induced emotion) and an average of 9.2 points (Intensity of inspiration).

When the participants entered the laboratory, the experimenter tested a band for measuring physiological signals on the wrist and examined physiological signals. After the experiment began, measurements were made for 60 seconds after measuring the steady state of stability.

Fear of killers, vampires, werewolves, murder animals: The Silence of the Lambs

Fear of overwhelming environmental changes such as natural disasters and catastrophic accidents: Cloverfield

The inner and collective psychology of characters, and the psychological fear evoked by mutual conflict: World War Z

Fear of external existences such as demons/evil spirits, aliens, zombies, ghosts, witches, psychics, and monsters: The Priests

Fear of physical damage such as disease, body mutation, body decay, and parasitic life: 28 Weeks Later

Physiological signals such as the hand movements, skin tension, and heart rate of the user were measured with the Emotion Collector. The measured data were stored in real time through a Bluetooth communication using a smartphone application (as shown in Fig. 2). A huge amount of data could be collected and stored for a long time.

Fig.2

Wearable Band and Emotion Collector Application.

5.2 Experimental results

We analyzed a section of the acquired physiological and emotional condition of 30 seconds in the acquired physiological signal. The stabilized state was set to 30 seconds before the stimulus was presented. And the emotional state of the test was set at 15 seconds before and 15 seconds after the test participants felt the most intense fear of fear.

There were two subjects who did not show any particular physiological changes to the clips, because they already had seen the clips before or they are trained to a feeling of fear. Other than these two, according to the analysis of the physiological response signals that were collected from the 10 subjects. The biological reaction data collected after the experiment were analyzed with each image clip (as shown in Fig. 4), and the changes and characteristics of the biological signals were analyzed according to the image source. After analyzing the results of three physiological signals of from each subject, we found that skin tension response measured through the GSR sensor was the clearest and most immediate response to fear.

Fig.3

An Experimenting Subject.

Fig.4

Comparative Analysis of Video Clip and Biometric Data.

Figure 4 shows that the subjects show the maximum stress for the fourth movie clip which showed an evil spirit. And the second stressed clip was the first what showed a murderer. We found that the subjects did not respond well to the fear of natural disasters and collective psychology. We can infer that human fear more about same human being than natural disasters or disease. We believe that this is also because the clips were played for only a short period of time, and the second and third clips required a long period of immersion in the films’ stories and situations. Although the scene in the fifth clip representing fear of terrible body damage was much more violent and stimulating than the scene of the first clip, it remained in the third place. This confirmed that the audience did not feel a greater sense of fear even for a more visually violent and blood-stained scene.

Figure 5 shows the result of subjects’ changes of heart rate during the experiments. The fourth clip also has the highest heart rate. Unlike the result of GSR, the third clip that showed the second highest heart rate. According to the Fig. 5, the heart rate is gradually increasing up to the forth clip and gradually decreasing to the end. For each clips, the heart rate is often increasing and decreasing. We can infer that the GSR is responsive to immediate reaction, whereas heart rate is gradually changing.

Fig.5

Result of GSR.

For the last result, we can see the movements of subjects’ hand through Fig. 6. Even though we couldn’t see many movements, we could find some moves for certain scenes. Especially, the fourth clip and the first clip had many movements that also had high stress scores and heart rates. One interesting point for the hand’s gestural movement is that we can see many movements for the fifth clip. Even though the subjects didn’t feel psychological fear for the visually violent and blood-stained scene, they naturally reacted by their hand movements like covering their mouths or clenching their fists (as shown in Fig. 7).

Fig.6

Result of Heart Rate.

Fig.7

Result of IMU.

Lastly, for the total average of these three results, the users showed the highest fear response in the scene where the evil spirit appeared from the fourth clip (The Priests). The next-highest fear response was for the scene where the first murderer carried out brutal and horrible acts in the first clip (The Silence of the Lambs). However, there were two participants who showed highest physiological changes for the first clip than the fourth clip. We think this difference came from their culture or social experiences that can effect on sources what they feel fear from. However, we were able to infer the subject’s heart rate and hand movements were increased to the two types of movie clips. We also able to see the decrease of conductivity of the skin (GSR) that means the subjects were stressed to both clips.

5.3 In-depth interview results

After a 10-minute break, we conducted an in-depth interview using the questionnaire to investigate whether the results of the psychological and physiological signal responses of the test subjects were identical. The interview included questions about how to express and classify the levels of fear experienced during each of the five video clips.

Question 1: How would you describe the fear you felt after watching each video clip?

Question 1: 1. Dreary 2. Obnoxious 3. Creepy 4. Scary 5. Horrible.

Question 2: Rank the fear you felt for each video clip in order of intensity.

For the second question, which queried psychological responses to fear, five subjects answered that the fourth clip, which was originally ranked first, was the most frightening (as with the physiological signals response). However, three subjects answered that the fifth clip, from the film ’28 Weeks Later’, was the most frightening. This clip was ranked second and produced a psychological response that was different to their biological response signals. An examination of biological response signals found that the biological responses of two subjects produced different psychological results, despite having the highest emotional response to the fourth clip. In a subsequent in-depth interview, subject A responded, “The scream of the woman crying in pain at the fifth clip was the most frightening,” while subject B replied, “In the fifth clip, the male character bites off the woman’s face, and the woman screaming in horror was the most frightening.” Analogizing from these accounts, we could see that the two subjects felt the highest psychological fears during the provocative visual and auditory elements of the fifth image clip. Meanwhile, the subjects who said that they were most frightened of the first clip said that “the scene of eating the brain of a living person was too creepy”, and that “the brain eating scene was too shocking and frightening.” Thus, these subjects felt the greatest fear during a scene in which one person commits the murder of another in an outrageous manner; the most intense fear was triggered by the brutal of the murderer.

Similar interviews showed that there are similarities and differences between biological response signals and psychological manifestations of fear for all test subjects. Although multiple subjects experienced psychological fears matching their biological response signals, some of the test subjects showed a psychological response that was different from their biological response signals. This occurred because the psychological responses varied depending on the provocative audiovisual elements and fearful materials contained in the horror images. In other words, according to the social cognitive experience of an individual’s fear and direct audiovisual stimulation, the psychological responses to fear were different from the physiological signal responses.

Based on the above analysis, we were able to observe the users’ different physiological response signals and psychological responses to the five factors that evoke fear, and we found that we can subdivide the emotion of fear.

However, we still have some limitations such as the limited number of subjects and length of movie clips. We need much more subjects for the further experiments to standardize the human emotion. We also found out that we need to extend the length of movie clips to understand the situation or environment for having certain emotional response to the movie clips. We will apply the data to the machine learning algorithm such as linear discrimination, decision tree and support vector machine with a difference in sensitivity, subtracting stability from the emotional state for the further analysis. By comparing these different machine learning algorithms, we want to find out what is the best way to categorize the specific emotional status like different kind of fear.

6 Conclusions

In this study, we subdivided the emotional factors of fear into five categories, by measuring physiological signals for each evoked emotion, and designed a wearable band called the Emotion Collector to recognize the five types of fear. We collected and analyzed the subjects’ three different physiological signals such as their hands’ movements, heart rate and the degree of skin irritation while they were watching the 5 classified movie clips for fear. Finally, we confirmed that it is possible to classify the feeling of fear through the biological information of a person through the experiments. We also conducted a questionnaire survey on the emotional response of each user and compared the correlation between their biological and psychological reactions. Thus, we confirmed that there was a difference between the psychological manifestations of fear and the physiological signal results of each subject, and that these results can vary depending on the theme, a particular cause of fear, and audiovisual stimuli.

The results of this study may be applicable not only to emotion recognition but also to immediate user-centered services and contents using emotional data in the future. This study focused on negative emotions such as fear, but we plan to carry out further research on positive emotions such as joy, admiration, and pleasure. Emotion recognition technology using physiological signals can be used in human-friendly ways in VR, AR and mobile environments are expected to be extensively used in the future in various fields.

Footnotes

Acknowledgments

This research was supported by the National Research Foundation of Korea (2017R1D1A1B0 3027954)

References

Khan

, Sajid , Riaz

, Akram

and Latif

, Facial expression recognition using computationally intelligent techniques. Journal of Intelligent & Fuzzy Systems28(6) (2015), 2881–2887.

Bailensona

J.N.

, Pontikakisb

E.D.

, Maussca

I.B.

, Grossd

, Jabone

M.E.

, Hutchersond

C.A.C.

, Nassa

and Johnf

, Real-time classification of evoked emotions using facial feature tracking and physiological responses, Int J Hum-Comput Stud66 (2008), 303–317.

Cowie

, Cowie

E.D.

, Tsapatsoulis

, Votsis

, Kollias

, Fellenz

and Taylor

J.G.

, Emotion recognition in human-computer interaction, IEEE Signal Process Mag18 (2001), 32–80.

Oatly

, Jenkins

J.M.

, Understanding Emotions. Oxford: Blackwell Publishers, UK, 1996.

Ekman

and Friesen

W.V.

, Universals and cultural differences in the judgments of facial expressions of emotion, J Pers Soc Psychol (1987), 712–714.

Palomba

, Sarlo

, Angrilli

, Mini

and Stegagno

, Cardiac responses associated with affective processing of unplesant film stimuli, International Jounal of Psychophysiology36 (2000), 45–57.

Alvarado

, Arousal and valence in the direct scaling of emotional response to film clips, Motivation and Emotion21(4) (1997), 323–348.

Calvo

R.A.

, Brown

, Scheding

, Effect of experimental factors on the recognition of affective mental states through physiological measures, Proc of the 22nd Australasian Joint Conference on Artificial Intelligence, Melbourne, Australia, 2009, pp. 62–70.

Picard

R.W.

, Vyzas

and Healey

, Toward machine emotional intelligence: Analysis of affective physiological state, IEEE Trans Pattern Anal Mach Intell23 (2001), 1175–1191.

10.

Sinha

and Parsons

, Multivariate response patterning of fear and anger, Cognition and Emotion10(2) (1996), 173–198.

11.

Vrana

, The psychophysiology of disgust: Differentiating negative emotional contexts with facial EMG, Psychophysiology30(3) (1993), 279–286.

12.

Kim

M.-S.

, Neurobiological contacts of cognition and emotion: Fear memory and amygdala, Korean Psychological Association16(1) (1997), 1–17.

13.

H.-J.

, The effect of facial muscle reaction on positive emotions and negative emotions, Journal of the Master’s Thesis, Chungnam National University, 2005.

14.

Darwin

, The Expression of the Emotions in Man and Animals. London: J. Murray, 1872.

15.

Song

J.-H.

, Development of film model technique program for the reduction of injection fear of hospitalized children and identification of its effect, Ewha Womans University Graduate School Doctoral Thesis, 1991.

16.

Palomba

, Sarlo

, Angrill

, Mini

and Stegagno

, Cardiac responses associated with affective processing of unpleasant film stimuli, International Journal of Psychophysiology36 (2000), 45–57.

17.

Woo

T.-J.

, Autonomic nervous system responses to aversion and fear induced by video stimuli, Master’s Thesis, Chungnam National University, 2005.

18.

Prkachin

K.M.

, Williams-Avery

R.M.

, Zweel

and Mills

D.E.

, Cardiovascular changers during induces emotion: An application of Lang’s theory of emotional imagery, Journal of Psychosomatic Research47(3) (1999), 255–267.

19.

Chang

E.-H.

, Consistency of autonomic nervous system responses induced by emotion and emotion-immediate emotional response pattern, Master’s Thesis, Chungnam National University, 2002.

20.

Fowles

D.C.

, Kochanska

and Murray

, Electrodermal activity and temperament in preschool children, Psychophysiology37, 777–787.