Emotion recognition from physiological signals and video games to detect personality traits

Abstract

This paper presents a system that allows for the identification of two values: arousal and valence, which represent the degree of stimulation in a subject, using Russell’s model of affect as a reference. To identify emotions, a step-by-step structure is used, which, based on statistical data from physiological signal metrics, generates the representative arousal value (direct correlation); from the PANAS questionnaire, the system generates the valence value (inverse correlation), as a first approximation to the techniques of emotion recognition without the use of artificial intelligence. The system gathers information concerning arousal activity from a subject using the following metrics: beats per minute (BPM), heart rate variability (HRV), the number of galvanic skin response (GSR) peaks in the skin conductance response (SCR) and forearm contraction time, using three physiological signals (Electrocardiogram - ECG, Galvanic Skin Response - GSR, Electromyography - EMG).

Keywords

Emotion recognition personality traits Russell’s model of affect eysenck personality theory physiological signals

1 Introduction

The relationship between personality and emotions has been extensively reviewed in social psychology [1 –3]. Eysenck’s theory proposes that extroverted people require more external stimulation than introverted people, and that neurotic people are more easily stimulated [4 –6]. However, there are few works that have conducted research into this direct relationship on a computational level. An example of the use of emotional recognition and personality traits is found in [7], where a database was created with information from individuals who took the Big Five personality test and underwent a laboratory experiment where they watched a series of videos while their physiological responses, such as ECG, EEG (Electroencephalography), GSR, facial expressions, etc. were captured. In some studies, where the relationship between personality and emotion in an individual has been investigated, it was found that Eysenck’s concept of extraversion is correlated with the parameters of Russell’s idea of arousal, and that in the same way, neuroticism is inversely correlated with the parameters of valence [8 –10]. In other works, emotions were investigated through facial expression recognition [11 –13] and in different areas, such as, emotion recognition in social networks [14], emotions in video games [15], emotion recognition using simple recurrent units network and ensemble learning [16], automatic emotion recognition from the analysis of body movement [17], among others.

For the proposed work, two sources of information will be used to identify the emotions in each of the four quadrants of Russell’s model of affect, which is correlated with the Eysenck model. For the arousal axis, three physiological signals are used (ECG, GSR, EMG), the characteristics of which are directly related to the excitation and relaxation of the individual’s BPM, HRV, SCR and forearm contraction time. For the valence axis, the PANAS questionnaire (Positive and Negative Affect Schedule) is used, which is a questionnaire that includes 20 elements to measure positive and negative affect, developed by Watson, Clark and Tellegen [18]. With these two sets of metrics, two values are obtained, which will be correlated with Russell’s model of affect.

2 Materials and methods

The project is focused on the development of a prototype system which can record, analyze and extract metrics related to the activation of the sympathetic and parasympathetic autonomic nervous system, from the ECG, EMG and GSR signals of an individual while they are playing a video game. These metrics can then be related to the Eysenck model in order to identify personality traits. Fig. 1, shows a schematic of the system architecture implemented, which identifies the elements by which the individual will be evaluated, and the visual reports that will be delivered to the professional in charge of analyzing and evaluating the personality of the subject from the resulting data.

For the implementation of the software application, five sections were defined that integrate its basic functions in order to display the information:

Fig. 1

Block diagram of the emotion and personality traits recognition system proposed.

2.1 Registration panel

This shows graphic and quantitative information of the three signals (ECG, GSR, and EMG) using graphs, recording time indicators, sample frequency and the number of samples acquired simultaneously.

2.2 Signal conditioning panel

This shows the information of the three signals, in graphic form, recorded before and after the application of filtering techniques for noise reduction.

2.3 Analysis panel

This shows, both graphically and quantitatively, the metrics extracted from the three signals (Fig. 2).

Fig. 2

User interface for the visualization of the physiological signals metric extraction process.

2.4 EPI / PANAS panel

This allows the input of the information corresponding to the results of the EPI personality test as well as the answers to the questions from the PANAS form (Fig. 3).

Fig. 3

User Interface to enter EPI values and answer the PANAS questionnaire.

2.5 Eysenck / russel panel

This allows for the visualization of the information corresponding to the results of the EPI personality test and the results obtained from the PANAS form and the physiological signals related to Russell’s model of affect.

2.6 Information on metrics and questionnaires

In order to identify and describe the behavior of these four metrics, and their relation to the arousal axis of Russell’s model of affect, an alignment sequence (Fig. 4) is implemented, corresponding to two different time windows from an experiment in which a subject was evaluated. First, the metrics must be collected when the subject is in a state of relaxation (from 3 to 10 minutes), the longer the recording, the more accurate the average from the metrics within the defined time window will be. From the average of the four metrics, both a positive and a negative threshold are calculated, adding and subtracting the standard deviation from the average.

Fig. 4

Flow diagram of the proposed sequence to calculate the arousal score (Russell’s model).

Then, the metrics are recorded at 60-second intervals within the time window in which the subject is playing the video game (from 15 to 20 minutes approximately). Each of these values will be compared to the average and to the thresholds obtained during the relaxation state time window. If the value from the metric is higher than the positive threshold, a + 1 to the arousal score is counted; if the value from the metric is lower than the negative threshold, a –1 to the arousal score is counted. The resulting score gives a reference value that indicates the arousal level of the subject.

In order to perform the correlation with the valence axis of Russel’s model of affect, the values of the PANAS questionnaire are gathered. The questionnaire is answered immediately after the application of the stimulus (video game) and the EMG, GSR and ECG signals are registered. These values will be compared with the result of the EPI questionnaire, which must be completed prior to the test.

2.7 Video game selection

Once the prototype metric detection system that will allow the identification of the emotion and its correlation with personality traits has been built, it is necessary to select the tool to be used to stimulate emotions in the individual. According to a study conducted by Nick Yee [19], the choice of a specific video game genre is directly related to the personality traits of the player. In this study, the personality model proposed by the Big Five is used as a reference to relate personality traits and video game genres. In the study carried out by Yee, video game genres are related to 3 personality types: extroversion, conscientiousness, and openness to experience. From these relationships, three different groups are created: action - social, mastery - achievement, immersion – creativity. From the results, the video game is selected based on two basic criteria: the first is that the video game should not belong to categories that are directly related to extroversion, so that the video game has no direct influence on extroverted or introverted personalities; the second criterion is that it should be a virtual environment that provides tasks or challenges that the individual must solve with low-medium difficulty and with a relatively short introduction to the game. This is so that the individual does not spend most of the test’s duration trying to understand the story of the virtual environment, but instead interacting with the tests.

From these criteria, a puzzle-type video game was chosen, which tests the ability of the player to solve problems that may be of a mathematical, spatial or logical nature. Within this genre, the video game Portal 2 was selected, which is a first-person, single-player video game of logic and consists of a series of puzzles that must be solved by teleporting the character as well as simple objects using a device that can create interspatial portals.

3 Results

The participants of the study were fifth and ninth semester students of the Electronic Engineering undergraduate program: 16 students from the fifth semester and 19 students from the ninth semester. The EPI test was applied by means of a Google form which was filled out by the 35 students, from which the three scores (Extroversion, Neuroticism and Lie) were obtained. A filter was made based on the students’ score in the Lie item, which registers how socially desirable a subject is trying to be with their answers, with values from 0 to 9. Those who received a score of 5 or higher in this field were not considered for the study. After applying the filter, the number of students was reduced to 20. Finally, prototype tests were performed on 11 students (those who wished to participate in the project) out of the 20 who were selected based on the result of the EPI. Of these 11 students, 6 were in fifth semester and 5 in ninth semester, 3 were women and 8 were men.

In order to perform the system tests, a series of tasks were defined as necessary, in which three important stages are summarized. The first stage is that of relaxation, which ensures that the metrics of the physiological signals of the individual reflect a state of calmness. This time window serves as a reference in order to compare the metrics recorded during gameplay. The second stage refers to the individual interacting with the video game. The signals are recorded and the metrics for each minute are registered, both during the period of relaxation and while playing the video game. The third stage consists of the entry of the previously obtained EPI values and those from the PANAS questionnaire provided by the individual being tested, as well as the data from a graph of the behavior of each of the metrics, which summarizes how the individual felt during the gaming session.

An inter-core i7 computer with 12GB of memory and an NVIDIA GTX 860M graphics card with 4GB of memory are used to run the emotion detection software, the video selected for the relaxation phase and the video game. As additional hardware, a monitor, a keyboard and a mouse are used as external peripherals to the main computer equipment, which the individual will manipulate during the test. The work space is adapted and the following activities are carried out:

Sensors are connected to the body of the subject by means of disposable electrodes to record the ECG and the EMG signals. Pads with electrodes are also attached for detecting the GSR.

Data acquisition tests are performed, evidencing that the signals are being registered correctly by the software.

Headphones are used to isolate ambient noise, and focus the subject’s attention on the sounds of the video game.

Once the subject is ready, with the sensors and other elements correctly placed on the body, the test protocol is applied (Fig. 7) with a duration of approximately 25 minutes, performing the following activities in sequential order:

Video playback with relaxing music (3 min): This step aims to ensure that the individual is in a state of passivity and relaxation. The signals are recorded and metrics extracted while the subject is in this state. These will serve as a basis for the comparison of the autonomic nervous system activity together with the metrics extracted while the individual is playing (Fig. 5).

Interaction of the subject with the video game (16 min): During this period of time the individual interacts with the video game. Depending on their performance, they are able to advance in the game, encountering challenges of greater difficulty. Throughout the process, the test supervisor can observe the signals being recorded in real-time, without interrupting the individual’s gaming session (Fig. 6).

Entering the EPI values: In this step, once the recording of the signals is completed, the person responsible for monitoring the test enters the values of the individual’s E, N and L scores, which were obtained earlier, into the software.

Entering the PANAS questionnaire answers: In this step, the test supervisor will administer 20 questions to the participant. These consist of a series of 20 adjectives to which the individual responds with a quantitative value, referring to the degree to which they had been experienced during the test. The responses are then entered into the software by the test supervisor.

Obtaining the visual values in the graphs displayed in the user interfaces for the analysis, correlation and detection of emotions: After entering all the necessary information into the software, the results will be displayed graphically so as to be interpreted by the test supervisor. Both the values taken from the EPI and Russell’s model of affect are shown, which displays the consolidation of the signal metrics and the PANAS questionnaire.

Producing an Excel report with the data of the signals and metrics: Finally, in order to have a record of the test, a report is created in Excel containing all the information generated by the software.

Fig. 5

Screen capture of the user interface of the signal and a video recording with relaxing music.

Fig. 6

Screen capture of the user interface of the signal register and video game.

Fig. 7

Subjects testing the prototype system.

4 Discussion

Table 1 shows a summary of the results of the two questionnaires (EPI and PANAS) applied to the 11 test participants, together with a comparison of them. In the case of subject 5, the registered GSR signal was not used as there was too much background noise and the respective metrics could not be recorded adequately.

Table 1
Results of the EPI - PANAS questionnaires of the 11 subjects

S EPI PANAS EPI personality group PANAS result Use

E Score N Score L Score Positive Score Negative Score Difference

01 16 6 4 31 16 15 Sanguine Positive Yes

02 6 17 3 22 40 –18 Melancholic Negative Yes

03 14 9 0 29 26 3 Sanguine Positive Yes

04 11 11 1 47 14 33 Phlegmatic Positive Yes

05 13 10 1 36 26 10 Sanguine Positive No

06 15 9 3 45 32 13 Sanguine Positive Yes

07 14 18 3 27 16 11 Choleric Positive Yes

08 12 12 4 17 16 1 Neutral Positive Yes

09 12 11 4 23 16 7 Stable - neutral Positive Yes

10 16 13 2 37 20 17 Choleric Positive Yes

11 6 8 3 33 23 10 Phlegmatic Positive Yes

S	EPI	PANAS	EPI personality group	PANAS result	Use
01	16	6	4	31	16	15	Sanguine	Positive	Yes
02	6	17	3	22	40	–18	Melancholic	Negative	Yes
03	14	9	0	29	26	3	Sanguine	Positive	Yes
04	11	11	1	47	14	33	Phlegmatic	Positive	Yes
05	13	10	1	36	26	10	Sanguine	Positive	No
06	15	9	3	45	32	13	Sanguine	Positive	Yes
07	14	18	3	27	16	11	Choleric	Positive	Yes
08	12	12	4	17	16	1	Neutral	Positive	Yes
09	12	11	4	23	16	7	Stable - neutral	Positive	Yes
10	16	13	2	37	20	17	Choleric	Positive	Yes
11	6	8	3	33	23	10	Phlegmatic	Positive	Yes

In Fig. 8, it is possible to see the results of a subject’s test and these are reflected in the EPI-RUSSELL interface. The results are shown graphically, so as to have a clear idea of the behavior of the metrics and questionnaires for each of the subjects.

Fig. 8

Visual comparison between the results of the signals of subject 1 with a sanguine personality.

Once the results were gathered for each of the 10 subjects, the values were entered into Table 2 in order to summarize the performance of the platform when identifying emotions and personality traits. To organize the information, the following values were taken into account: the PANAS questionnaire indicator, the indicators of the GSR, BPM, HRV and EMG metrics, the personality identified by each signal and, finally, the personality type resulting from the answers to the EPI.

Table 2

Relationship between the results of the signals and the PANAS questionnaire with respect to the personality type resulting from the EPI questionnaire

S	PANAS Indicator	GSR Indicator	Russel GSR	BPM Indicator	Russel BPM	HRV Indicator	Russel HRV	EMG Indicator	Russel EMG	EPI
1	Correct	Correct	Sanguine	Correct	Sanguine	Correct	Sanguine	Incorrect	Phlegmatic	Sanguine
2	Correct	Correct	Melancholic	Incorrect	Choleric	Incorrect	Choleric	Correct	Melancholic	Melancholic
3	Correct	Correct	Sanguine	Incorrect	Phlegmatic	Correct	Sanguine	Incorrect	Phlegmatic	Sanguine
4	Correct	Correct	Phlegmatic	Incorrect	Sanguine	Correct	Phlegmatic	Correct	Phlegmatic	Phlegmatic
6	Correct	Incorrect	Phlegmatic	Correct	Sanguine	Correct	Sanguine	Incorrect	Phlegmatic	Sanguine
7	Incorrect	Correct	Sanguine	Correct	Sanguine	Correct	Sanguine	Incorrect	Phlegmatic	Choleric
8	Correct	Incorrect	Relaxation	Incorrect	Activation	Incorrect	Relaxation	Incorrect	Relaxation	Neutral
9	Correct	Incorrect	Relaxation	Incorrect	Relaxation	Incorrect	Activation	Incorrect	Relaxation	Stable - Neutral
10	Incorrect	Correct	Sanguine	Incorrect	Phlegmatic	Correct	Sanguine	Incorrect	Phlegmatic	Choleric
11	Correct	Correct	Phlegmatic	Incorrect	Sanguine	Incorrect	Sanguine	Correct	Phlegmatic	Phlegmatic
%	80	70	50	30	10	60	40	30	10

In green, the values can be observed that coincide with the considered indicator of the Russel model: in this case, the activation axes (signals) and the valence axis (PANAS), and the value of the extraversion and neuroticism axes of the EPI model, respectively.

With regard to the PANAS questionnaire, an 80% success rate is seen between the valence axis of the Russel model and the axis of neuroticism of the EPI model. The two cases in which the values did not coincide refer to the subjects with choleric personalities.

With respect to the GSR signal indicator, there is a 70% accuracy between the activation axis of the Russel model and the extraversion axis of the EPI model.

Regarding the HRV signal indicator, there is a 60% success rate between the axis of activation of the Russel model and the axis of extraversion of the EPI model.

With respect to the indicators of the BPM and EMG signals, a correlation of only 30% is seen between the activation axis of the Russel model and the extraversion axis of the EPI model.

Finally, there is a column after each of the signal indicators which shows the personality type resulting from the model with respect to each of the signal metrics. As two of the results of the PANAS questionnaire are incorrect, this implies that the correct detection percentage of the model, with respect to the GSR and HRV signals, is 50 and 40 percent, respectively. More information in [20].

5 Conclusions

In the development of the research, once the test protocol had been applied to the 10 participants, the following conclusions were reached.

A system was designed for the recording, analysis and extraction of the metrics of the ECG, EMG and GSR signals associated with the activation of the sympathetic and parasympathetic nervous system that represent emotional activity during the interaction of an individual with the commercial video game PORTAL 2, without the use of artificial intelligence, as the first phase of the project.

The developed system does not depend on any particular emotional stimulus being applied to the individual; that is, the system would work with any type of audiovisual stimulus. The system is applicable to a wide variety of fields with the use of different stimuli in the form of multimedia files, such as audio, video, images, etc.

The system can work with different video game manipulation and viewing devices, since the positioning of the sensors and the recording of the signals are not affected when using keyboards, joysticks or control pads. In the study carried out, a keyboard, a mouse and an external screen were used for the manipulation and visualization of the game. These devices did not compromise the recording of the signals and gave the individual freedom of movement while playing the game.

Throughout the study and the systematic review of the literature, it was observed that the fields of emotion recognition and “personality computing” are relatively new. The investigations that are carried out begin with the registration and the correlation of the registered information in order to determine discrete emotions in an individual in a predetermined time window, leaving it open to the researchers to decide on the test protocols that must be used to provide conclusive data. From this information, a system was proposed for the identification of the emotional activation of an individual based on the statistical thresholds of the registered metrics. This allowed for the observation of how the signals of the individuals behaved during the testing period, achieving a measurement of the emotional state based on the three proposed signals.

Although the EMG and BPM signals are considered in the literature to be indicators of the behavior of the autonomic nervous system, they did not perform well when identifying the personality of the individual (30% correct), which implies a need to rethink, firstly, the use of the heart rate metric, and secondly, the location of the EMG sensor, so as to gather valid information about the manifestation of stress in the body of the individual who is being tested.

The next step of the research is to record a greater amount of data in order to achieve an effective correlation of the signals among individuals and design a classification model for each of the personalities using recognition techniques, such as metrics, classification algorithms and artificial intelligence. Likewise, the use of measurement tools for EEG signals is proposed in order to collect data on valence that does not depend on the application of questionnaires, such as PANAS.

References

Izard

C.E.

, The psychology of emotions. New York: Plenum. (1991).

Parkinson

, Intragroup Emotion Convergence: Beyond Contagion and Social Appraisal, Personality and Social Psychology Review 24(2) (2020), 121–140. DOI: 10.1177/1088868319882596

Smith

M.A.

, Thompson

, Hall

L.J.

, Allen

S.F.

and Wetherell

M.A.

, The physical and psychological health benefits of positive emotional writing: Investigating the moderating role of Type D (distressed) personality, British Journal of Health Psychology 23(4) (2018), 857–871. DOI: 10.1111/bjhp.12320

Eysenck

H.J.

and Eysenck

S.B.G.

, Manual of the Eysenck Personality Questionnaire, EdITS/Educational and Industrial Testing Service., editor. California; (1994).

Yusoff

, Anuar

N.N.

, Arifin

W.N.

, Begum

and Reza

, Emotional arousal substrate in relation to eysenck theory of extraversion, International Journal of Engineering and Technology (UAE) 7 (2018), 14–20. DOI: 10.14419/ijet.v7i3.22.17115

Matthews

, Traits, cognitive processes and adaptation: An elegy for Hans Eysenck’s personality theory, Personality and Individual Differences 103 (2016), 61–67. https://doi.org/10.1016/j.paid.2016.04.037

Subramanian

, Wache

, Abadi

M.K.

, Vieriu

R.-L.

, Sebe

and Winkler

, ASCERTAIN: Emotion and Personality Recognition Using Commercial Sensors, IEEE Trans Affect Comput 9 (2018), 147–160. DOI: 10.1109/TAFFC.2016.2625250

Stenberg

, Personality and the EEG: Arousal and emotional arousability, Pers Individ Dif (1992), 1097–113.

Kehoe

E.G.

, Toomey

J.M.

, Balsters

J.H.

and Bokde

A.L.W.

, Personality modulates the effects of emotional arousal and valence on brain activation, Soc Cogn Affect Neurosci 7 (2012), 858–70. doi: 10.1093/scan/nsr059

10.

Gupta

and Nicholson

, Simple visual reaction time, personality strength of the nervous system: theory approach, Pers Individ Dif 6(4) (1985), 461–469.

11.

Starostenko

, Cruz-Perez

, Alarcon-Aquino

and Rosas-Romero

, Real-time facial expression recognition using local appearance-based descriptors, Journal of Intelligent and Fuzzy Systems 36(5) (2019), 5037–5049. DOI: 10.3233/JIFS-179049

12.

Gervasi

, Franzoni

, Riganelli

and Tasso

, Automating facial emotion recognition, Web Intelligence 17(1) (2019), 17–27. DOI: 10.3233/WEB-190397

13.

Bahreini

, van der Vegt

and Westera

, A fuzzy logic approach to reliable real-time recognition of facial emotions, Multimedia Tools and Applications 78(14) (2019), 18943–18966. DOI: 10.1007/s11042-019-7250-z

14.

Rodríguez

F.M.

and Garza

S.E.

, Predicting emotional intensity in social networks, Journal of Intelligent and Fuzzy Systems 36(5) (2019), 4709–4719. DOI: 10.3233/JIFS-179020

15.

Tikhomirova

D.V.

, Chubarov

A.A.

and Samsonovich

A.V.

, Empirical and modeling study of emotional state dynamics in social videogame paradigms, Cognitive Systems Research 60 (2020), 44–56. DOI: 10.1016/j.cogsys.2019.12.001

16.

Wei

, Chen

L.-L.

, Song

Z.-Z.

, Lou

X.-G.

and Li

D.-D.

, EEG-based emotion recognition using simple recurrent units network and ensemble learning, Biomedical Signal Processing and Control 58 (2020), art. No. 101756. DOI: 10.1016/j.bspc.2019.101756

17.

Ahmed

, Bari

A.S.M.H.

and Gavrilova

M.L.

, Emotion Recognition from Body Movement, IEEE Access 8 (2020), art. no. 8945309, pp. 11761–11781. DOI: 10.1109/ACCESS.2019.2963113

18.

Rubin

D.C.

and Talerico

J.M.

, A comparison of dimensional models of emotion: Evidence from emotions, prototypical events, autobiographical memories, and words, Journal Memory 17(8) (2009), 802–808. DOI: 10.1080/09658210903130764

19.

Yee

, Gaming Motivations Align with Personality Traits [Internet], Quantic Foundry 1. Available from: http://quanticfoundry.com/2016/01/05/personality-correlates/

20.

Callejas-Cuervo

, Martinez-Tejada

L.A.

and Alarcon-Aldana

A.C.

, Tecnología para la identificación de patrones de comportamiento a partir de técnicas de reconocimiento emocional; Editorial Uptc: Tunja, Colombia, 2017, p. 92, ISBN: 9789586602518.