Abstract
Summary
The AUBADE system can be trained to classify a subject's feelings into six different emotional classes, derived from three of the basic emotions (happiness, disgust and fear). The performance of different classifiers was examined. Biosignals were recorded from 24 healthy subjects who viewed pictures designed to invoke different emotional responses. A psychologist evaluated the emotional status of the subjects by looking at their faces. During the training stage, information from 15 subjects was used to teach the system how to discriminate the emotional status of the subject based on the biosignals provided as input. A subset of the data was used for comparing the performance of four different classifiers. They were evaluated using three different metrics: sensitivity, positive predictive accuracy and accuracy. Using the SVM classifier, the AUBADE system provided sensitivities in the range 63–81%. The positive predictive accuracy was in the range 71–95%. The accuracy was in the range 63–83%, depending on the emotional class considered. The work paves the way for remote telemonitoring of patients suffering from neurological diseases.
Introduction
The AUBADE system for the detection and recognition of human emotions 1 is based on the processing of biomedical signals to determine the emotional state of subjects. The information used to classify human emotions comprises electromyogram (EMG) signals, which are acquired from the user's face. Other information, such as the ECG, respiration rate and skin conductivity, is also measured and is used to improve the accuracy of classification.
The acquired signals are collected and transmitted to a central workstation where a feature extraction module filters and conditions them. Feature reduction is then performed. The emotion recognition module uses classification techniques to detect the psychological state of the user. In addition, AUBADE implements a near real-time 3-D facial representation module, which animates a generic facial model with the specific user muscle movements (Figure 1).
Example of facial animation for a happiness emotional state
The AUBADE system can be trained to classify the subject's feelings into six different emotional classes, derived from three of the basic emotions (happiness, disgust and fear). The performance of different classifiers was examined.
Methods
The clinical validation of the AUBADE system was performed in a hospital environment, with subjects recruited from the University of Modena. Twenty-four healthy subjects (13 male; mean age 31 years) collaborated in the procedure. Participants provided their consent prior to the tests.
Data gathering
The EMG sensors were arranged in a mask and placed on the subject's face. A thin film multi-electrode grid, manufactured on a flexible carrier, was chosen for the facial EMG and ECG sensors. 2 A sensor mounted on a simple Velcro band was used for respiration rate. The band was strapped around the subject's diaphragm, to detect the expansion of the chest as the person breathed in and out. 2 The galvanic skin response was also measured using a Velcro strap that secured the sensor to the finger.
In order to evoke the appropriate emotion, a newly developed Emotions Inducing Test based on the International Affective Pictures System 3 was employed with the aim of evoking genuine emotions in the subjects. This test was based on pictures of 720 × 576 pixel dimension, randomly arranged in a Microsoft Power Point presentation (10 fear-inducing pictures, 10 happiness-inducing pictures, 10 disgust-inducing pictures and 5 neutral pictures), each one presented to the subject for four seconds. 2
For each stimulus, the subjects first said how they felt when visualizing the pictures (stimuli). Second, the biomedical signals were processed by the AUBADE system to indicate the emotional status of the subjects. Third, a psychologist evaluated the emotional status of the subjects by looking at their faces. The facial animation module was also used, since the facial mask with the sensors partly obscured the subjects' faces.
Training a classifier
During the training stage, data from 15 subjects were used. In total, 80% of the data-set from these 15 subjects was used to teach the system how to discriminate the emotional status of the subject, based on the biosignals provided as input. The corresponding emotional status associated with these biosignals was known from the three methods described above; if there was disagreement between the methods, the appropriate data-set was discarded.
Performance of the classifiers
The 20% of the data remaining from the 15 patients was used for comparing the performance of different classifiers. They were evaluated using three different metrics: sensitivity, positive predictive accuracy and accuracy. Four different classifiers were compared: Support Vector Machines (SVM), Decision Trees (C.45), K-Nearest Neighbour and Naïve Bayes classifiers. 4,5
Finally, nine subjects out of 24 were used to prove that the system was working correctly. The system classified the data into six emotional classes (low happiness, medium happiness, high happiness, high fear, medium disgust, high disgust), based only on the biosignals.
Results
The AUBADE system classified the subjective feeling of each subject into six different emotional classes. The results showed low performance values for the C.45 classifier for high fear, low and medium happiness emotional classes. The Naïve classifier outcome was not satisfactory for high disgust and medium happiness. The K-Nearest Neighbour classifier performed poorly for all emotions. The SVM classifier provided the best values for the three selected metrics (sensitivity, positive predictive accuracy and accuracy) and, therefore, it was selected for further system testing and included in the AUBADE system. 4,5
Using the SVM classifier, the AUBADE system provided sensitivities in the range 63–81%. This means that more than 60% of the emotional states classified by the system coincided with the psychologist's assessment. The positive predictive accuracy was in the range 71–95%. Finally, the number of emotional states correctly classified into a specific emotional class divided by the total number of emotional states classified (accuracy) was in the range 63–83%, depending on the emotional class considered. The accuracy was above 70% for all emotional classes except for high happiness. This exception was due to the lack of enough samples available during the training phase of the algorithms for this particular emotional class.
Discussion
The AUBADE system was designed for the diagnosis and follow-up of two particular progressive neurological diseases: Huntington's disease and Parkinson's disease. These illnesses are characterized by a deficit in emotional processing of fear and disgust. Thus the AUBADE system can be used to test the reaction — or lack of reaction — of subjects to specific emotions, helping health professionals to gain a better understanding of these disorders. The system was also designed to be scaleable and easy to modify, so that it can be adapted to detect additional sets of emotions that may be of interest in other neurological disorders or health-care specialities.
The present study showed that the system was able to detect six emotional classes and automatically classify them with high accuracy. Thus the AUBADE system appears suitable for the future assessment of the emotional state of humans suffering from neurological disorders.
The work paves the way for remote telemonitoring of patients suffering from neurological diseases. In future, an improved version of the current system will be capable of collecting and processing the recorded data from the patient and wirelessly transmitting the information to a centralized site, such as a hospital. There, further assessment, presentation in an interpretable format and storage will take place. This will provide health professionals with a decision support tool and patients with an ubiquitous and robust health evaluation and monitoring system.
Footnotes
Acknowledgements
We thank the AUBADE Project Consortium for their contributions to this work. The project was partially funded by the European Commission.