Automating facial emotion recognition

Abstract

The work described in this paper attempts to contribute to one of the most stimulating and promising sectors in the field of emotion recognition, which is health care. Multidisciplinary studies in artificial intelligence, augmented reality, and psychology stressed out the importance of emotions in communication and awareness. The intent is the recognition of human emotions, processing images streamed in real-time from a mobile device. The proposed techniques involve the use of open source libraries of visual recognition and machine learning approaches based on convolutional neural networks (CNN).

Keywords

Image recognition emotion recognition face detection machine learning convolutional neural networks

1. Introduction

The study proposes and tests an innovative technique for automated emotion recognition through face detection via Convolutional Neural Networks (CNN). The provided technique is meant to be applied for supporting people with health disorders that can reduce communication skills (e.g. muscle wasting, stroke, autism, or, more simply, pain) in order to recognize emotions and generate real-time feedback, or feeding more complex systems. Therefore, a user who cannot recognise emotions herself/himself (e.g. for a visual or cognitive impairment) can be assisted with augmented emotional stimuli; on the other hand, a user who has difficulties expressing emotions (e.g. speaking impairment or difficulty) can be monitored to infer expressions of suffering, and intervene accordingly. The proposed system, extension of the EmEx system, previous work of the authors [39], exploits the high precision of CNN processing, designed to process images. The added value of the proposed technique is to focus on visual micro-expressions for emotional classification: this strategy implies and aims to train the system on a specific person in order to recognise her facial particularities. Its capability is then exploited on the single user with a personalised training. A special classification of emotions is set up, divided into basic emotions inspired by the Ekmann classes [18]: anger, sadness, disgust, happiness, neutral. The system implementation includes a graphic user interface allowing to create a set of emotion events, i.e. a personalised training set. Finally, another part of the ad-hoc software makes use of the convolutional neural model to detect the type of emotion in a video stream, where the same user who trained the network appears. The particularity of our system is, in fact, to be user-centered: the user trains the network on her/his own facial expressions, the software supports personalized emotional feedback for each particular user. Personal traits, such as scars or flaws, or individual variations in emotional feeling and expression, will help the training to precise recognition.

Through an in-deep study of the elements characterising the problem, the system is capable of overcoming the barriers that have so far imposed such a problem. The proposed system has also robustness and scalability properties, verified with experiments. In order to obtain optimised results, the ambient light setting needs a proper setup:

Robustness: The algorithm must be able to operate even in the presence of low-quality data (e.g., low resolution, bad light conditions);

Scalability: The user position in front of the camera should not be conditioned by the fact that it has a fixed distance, in order to avoid constraining the person under recognition.

Luminosity: an important problem is precisely that of the variation of light. In computer vision [10], the variation of the lens involves an alteration of the information [13]. No complete control of the detected information is achieved, the system will be able to withstand variations in brightness without compromising the original information.

The proposed solution has been implemented in C++ and OpenCV graphics libraries; hence it is compatible with all operating systems. The use of these graphics libraries is due to their high reliability and constant support from an increasingly growing community of developers and users. Moreover, these libraries are widely used in many research projects and prototypes based on an open source approach. The open source approach greatly facilitated our work, thanks to the high number of examples available that simplified many developing phases, the accurate and extensive documentation, and the opportunities of sharing and improving our work with the community of developers.

2. Related work

2.1. Deep learning and image classification

The recent scientific focus on Deep Learning towards the end of the XX century has contributed to the rebirth of a major interest in neural networks. The true impact of Deep Learning began in the context of speech recognition around the year 2010, when two Microsoft Research employees, Lil Deng and Geoenix Hinton, realised that using large amounts of data for training a deep neural network resulted in lowering error rates far below the state of the art [25]. Discoveries in the field of hardware have certainly contributed to the rise of interest in Deep Learning. In particular, the ever-powerful GPUs seem to be able to perform the countless mathematical calculations of matrices and vectors in Deep Learning [21,31,45]. Actual GPUs allow reducing workout times from the weeks to a day.

Recently, deep learning has been used for several types of research aiming at the classification of images and learning, trying to solve the limitations of machine learning, which reside in overfitting and domain dependence, with image adaptation, kernel randomisation [32] and transfer learning [34].

Commitment has been dedicated by researchers to exploit domain dependence as a feature, where personalised classification can be easily done on a particular user or entity, especially for smart-home systems [29] and microblog sentiment tagging [11].

Alternative approaches consider evolutionary algorithms, random walks on semantic networks of images [16] and max-product neural networks.

Convolutional Neural Networks are among the most used methods for affective image classification [35] thanks to their flexibility for transfer learning, and easy tools available on the Web [6].

2.2. AI-assisted health care

For computerised health care assisting, multidisciplinary studies in Artificial Intelligence, Augmented Reality and Robotics stressed out the importance of computer science for automatizing real-life tasks for assistive and learning objects, such as detecting words from labial movements (i.e. automated lip detection) [19], Virtual reality for prosthetic training [36] or neural telerehabilitation of patients with stroke [20], vocal interfaces for robotics applications [3].

As an application of complex networks, it is possible to predict bacteria diffusion patterns [14], as well as epidemiology data [46], having a viral spread. To be mentioned, high advances are happening on medical image recognition and multi-stage feature selection for classification of cancer data [1], and of text corpora for medical or patient feedback in social networks.

One of the most promising advances of recent years for AI-assisted health care is the opportunity to have light-implementation Mobile Apps, that can be easily developed to be used in a friendly manner [15], to assist and support disabled users for communication and learning tasks. Such applications can be run directly on personal smartphones or wearables devices, for health monitoring and prognosis [33] as well as for interactive support for people with disabilities or conditions that can influence communication and learning, such as autism spectrum disorders [22]. Using cloud services or networks in the Internet of Things (IoT), makes possible both to connect such devices to high capability servers, both to collect data in a distributed collaborative perspective [44], in order to feed big knowledge-bases, and increase the capability of the single object, i.e. of its owner, as a member of a large interactive collective dynamic knowledge (i.e. a Big Data) network.

2.3. Affective computing and emotion recognition

Multidisciplinary approaches recently stressed out the importance of recognising and extracting affective and mental states, in particular emotions, for communication, understanding, and supporting humans in any task with automated detectors and artificial assistants having machine emotional intelligence [38]. In real-life problems, individuals transform overwhelming amounts of heterogeneous data in a manageable and personalized subset of classified items. The process of recognition of moods and sentiments is largely complex, but recent research underlines that basic emotional states such as happiness, sadness, anger, disgust, or neutral state [2] can be recognized based on physiological clues such as heart rate, skin conductance and face expression, differently from sentiment, moods and affect, which are more complex states and can be better managed with a multidimensional approach [17,18]. Since Rosalind Picard defined the challenges for Affective Computing in 2003 [37], numerous advances have been done in the task of emotion recognition, such as defining collective influence of emotions expressed online [8], stating that emotional expressiveness is the crucial fuel that sustains communities; studying cultural aspects of emotions in art [4] and its variations; create emotionally engaging experiences in games [7], where affective changes are crucial to the conscious experience of the world around us. Some of the more ethical and important challenges defined by Rosalind Picard, however, still remain open. For example, many of the modalities for emotion recognition (e.g. blood chemistry, brain activity, neurotransmitters) are not easily available, commercial tools are limited [9], datasets for training are not general [41] and people’s emotion is so idiosyncratic and variable, that there is difficult to accurately recognise individual’s emotional states from available data [37].

This paper aims to provide a solution to overcome such difficulties.

3. The proposed method for emotion extraction: EmEx2

The proposed method takes some insights from literature about lip reading [5,28,40], and the GUI-based EmEx software [39], ad-hoc implemented. EmEx2 will recognise the emotion of the user whose face is in the camera streaming [43]. For our work, we need to establish a subdivision of classes (see Table 1) as a representation of the Ekman basic emotions [18], which will be our multi-classification features [42].

Table 1
Identified classes to represent emotions

Fig. 1.

Various phases of the EmEx2 application: Acquisition of frames by a camera, identification of the main features (facial micro-expressions) and final classification by the convolutional neural network.

In Fig. 1 the various phases of the EmEx2 software are represented. The initial phase is based on the training of the system based on the specific user’s characteristics, and is realised acquiring a series of images associated to each of the considered emotions, summarised in Table 1. Each image is then processed by the Convolutional Neural Network to identify the features associated to the emotion expressions of the face and to classify the corresponding emotion.

In routine operations the frames captured by the camera will be processed and the emotions identified by the EmEx2 application in real-time. The information represented by the visual expression captured by the camera is processed to return a given output. The system will process the information, that can then be returned to the user as a real-time feedback (sounds, symbols, graphics, and elements in the Graphic User Interface) as shown in Fig. 2.A, or it can be further processed by the system, and the output saved in a file for further processing by external tools, as shown in Fig. 2.B. Actions may be carried out by the system to help people with impairments and/or disabilities, on the basis of the emotion expressions interpreted by the system.

In Fig. 3 the general scheme of the implemented Convolutional Neural Network is summarised, both for the initial training and the classification phases. The two phases are structured in a similar way, however the initial training phase, customised to the user, implies some more complex operations and the communication scheme among the various layers of the Convolutional Neural Networks is more complex. The layers are represented in Fig. 3 as rectangles (each type of layer has its own color associated), while the blobs (used to pass information among layers) are represented as hexagons.

Fig. 2.

Simple output scheme: The output can be returned as real-time feedback (sound, symbols, graphics, and elements in the GUI) (A), or in a file to be further processed by external tools (B). By the interpreted emotions, the system may carry out some actions to help people with disabilities or impairments.

The layers sequence is crucial to obtain good classification scores. In the present research, we used a series of Convolutional layers alternated to a Normalisation layer or a Pooling layer, considered when is necessary to improve the quality of the input data making transformations not altering the original information. For example, using the Normalization layer we can make a linear transformation of the image to improve its readability while, using the Pooling layer, we can reduce the size of the input data to speed-up the computation. Finally, there are Inner Product layers (particular Fully Connected layers, where the number of neurons in output is the number of neurons of the level itself), alternated to a Rectified Linear Units layer that quickly trains the convolutional neural network. Then, the Convolutional Neural Network is ended by the Output layer.

During the training phase, the output is represented by the Accuracy of the process and by the lost information; during the classification phase, we obtain the emotion recognition output, according to the classes defined in Table 1.

Fig. 3.

Representative diagram of CNN network to classify emotions. The training and classification phases.

4. System prototyping and settings

4.1. CNN training phase

Creating the dataset represents a crucial phase [23], where it is possible to train a neural network with a set of images expressing the emotions described in Table 1. Such a training phase will enable the user to train a personal network with the appropriate parameters for her/his case, and our algorithm will automatically recognise user’s emotions during the software training activity, i.e. in the frame acquisition phase.

A personalised automation software automatically detects the position of the person’s face [12,27,40] in the camera, recognising the emotions basing on the training set. The software interface is designed to be usable [15], to intuitively guide the user into the various facial expressions in the training phase, in order to form the dataset of images and files needed to train the neural network.

Besides the automated training, the software interface also allows the user to provide additional frames (e.g. photos) and to tag them with an emotional label, to improve the neural network and its recognition capabilities. Moreover, a usable list of frames of the training is always accessible by the user, to manually remove in an easy way eventual noisy frames.

4.2. Implementing the convolutional neural network

In order to have a direct approach to the world of conundrum neural networks, the proposed approach focuses on the use of an open source framework. Open source is in fact a powerful method to guarantee security and privacy, where it is possible for each version to check what exactly the code does, and to fix eventual bugs by a collective effort. For our proposal, we chose the Caffe framework [6] available online, supported by an extensive online documentation, usable, and of proven efficiency.

To create our user-centered personal dataset of emotional labels, we proceeded to set up the files for the proper convolutional neural training, in agreement with the CNN-based theory.

Network layers are thus set up to extract the specific information of the image data, accurately setting the parameters to have an effective recognition. Figure 3 shows the layers of the proposed neural network for emotion recognition [24] CNN, where training data are labeled with emotions, and the results of the layer computation is evaluated in terms of accuracy and loss.

The neural network consists of a number of layers connected to each other. The main level is for the appropriately converted dataset and the corresponding tags. The second layer is the convoluted layer: being directly linked to the dataset, in this layer the convolution operations are performed on the images, extracting specific information about each frame in each class. Subsequently, a pooling layer is used to reduce, in width and height, the volume of previously created data. Such a reduction leads to reduce also the parameters magnitude, and therefore the computational time, needed for the network to be computed. For scaling, a maximum and average calculation function of a variable set is used: in our tests we used a max function. After the reduction step, the network continues again with convolutional layers.

Subsequently, after the layer pooling is applied further, a layer of inner product (innerProduct) groups all the information obtained up to this point. This grouping step has the useful effect to express such information in a single numeric value, that can be processed again in the subsequent phases.

After the group layer, the system is able to return a vector representation of neurons (from n to 1). This suggests that, from here onwards, it will no longer be possible to apply unambiguous layers.

Another layer of innerProduct is then applied: it is useful to put these layers in sequence, because the last one will have an output parameter that will equal the number of classes needed for the classification. The K final values will be therefore the object of a probability function that will allow the final classification.

It is noticeable that in the TRAINING phase, the network ends with an Accuracy layer for network accuracy calculation, and with a Loss layer for the calculation of the error function needed for a correct and useful training phase. In the CLASSIFICATION phase, instead, we use the same file, but having a SoftMax layer as the final layer: the main purpose of the SoftMax layer is the classification of new images (test set), not present in the dataset. This type of layer calculates the likelihood of the most appropriate class in the grading phase, and therefore represents the final solution.

In order to reach the final solution, we created the network, set the parameters of each layer and prepared the various files needed for Caffe to start the training phase with the appropriate command. Each user will be able to setup a personalized network, in an easy and intuitive way, guided by the EmEx2 graphic interface.

It is also necessary to create a network for the classification phase. This will be identical, in the layer type, to the network used for training, but it will have as an input an unknown image captured by the camera, and as an output a layer capable of establishing a property for the most appropriate class corresponding to the detected image. The network in question is visible in the Fig. 3.

4.3. Image classification

It is possible to go to the classification step by creating the network template. A software has been designed to process the image by using the Opencv libraries, automatically pulling out the face position, after which the frames are captured and their rescue are controlled by Caffe, which will classify them, each in the corresponding class of membership, returning the proposed output value. To have a safer result, a check of occurrences is made: the consecutive frames belonging to the same class are counted, and if they exceed a predetermined threshold value, then it is likely that the expression will be expressed by the network. In this way it is possible to delete some false positives, therefore improving the accuracy of the network. The classification is carried out using the so-called ‘top-5 accuracy’ method. This means that the grading probabilities are reported for each of the first five classes in descending order, from the most probable to the least probable. In this way it is also possible to establish a classification of multiple emotions, that is represented by the visual expression of multiple expressions. The software, and the underlying convolutional neural network, are able to detect even if the user is expressing both sadness and seriousness at the same time, given the chances of the neural network for each dataset class.

To obtain an even more reliable final classification, a check of occurrences is carried out, in agreement with [30] and [26]. Observing how many times consecutively an emotion can detect noise by eliminating frame sequences below a predetermined threshold, it can be assumed that the emotions that occur a small number of times do not really belong to the emotion expressed by the person. With this step we get two benefits. The first is to significantly reduce the length of the carrier containing the data. Moreover, by observing how many times an emotion occurs, it is possible to detect noise, by eliminating emotion sequences below a predetermined threshold. In our experiments such a threshold has been found empirically. In some cases, the threshold may be too low, but it is considered preferable to leave some rare misconception, rather than risking to eliminate a sequence of correct emotions.

5. Evaluation of results and future works

5.1. Evaluation of results

In this paper we described EmEx2, a system designed to automate the identification of emotions from face expressions acquired through a Webcam. The system is based on Convolutional Neural Networks. To run tests, we used a personal computer equipped with an Intel core i5-3210M CPU, 2,50 GHz, 4,00 GB of RAM and a Webcam 640x480 px, 30 fps.

By using a specific training phase for each user, the recognition of emotions become more effective. It is advisable to create the dataset and the subsequent training phase for each user according to a personalised approach, although this method can also be used without a training phase, and introducing a complex and complete dataset of samples for each class. A significant advantage in using this system is also the fact that it allows a variable distance of the user from the Webcam.

The first step consists in creating an image dataset, in the appropriate input format, depicting the training set and the validation sets useful to Caffe for the training phase, which, thanks to the personalised layers, allows to calculate the accuracy of the model at certain intervals.

To facilitate the use of the system by the final user, the ability to interrupt the training phase in advance is provided, so that an expert user who wants to improve some parameters, without waiting the completion of the computational phase, is enabled to do it. The template created in such a way will be used for a Webcam rating. Finally, the classification program serves the system to classify emotions expressed by the user, based on the previously-prepared training data.

Final tests allow to identify the emotion with a given probability, depending on appropriate classes for the face acquired by the Webcam, as shown in Fig. 4.

Fig. 4.

Face image classification – 1.

Fig. 5.

Face image classification – 2.

In the two figures, we can see that the expression of happiness can be recognized easily (Fig. 4), while a small uncertainty is present for the trained data for the second classification (Fig. 5). This is due to the fact that it is easy for the system to recognize a smile by the movement of the lips, i.e. lips detection. A more accurate training for the other emotions may enhance the results.

Table 2 shows the number of correct classifications for each type of emotion, carried out in a consecutive image acquisition. In this case 5 consecutive acquisitions for the smile images have been carried out. Small defects are found only for very similar representations as visual expressions.

Table 2

Results

Class	Emotion	Images identified
0	anger	5/5
1	sad	5/5
2	happiness	5/5
3	disgust	2/5
4	fear	5/5
5	neut	3/5

A better match can be obtained with more examples for the training dataset. Ideal future works can include a revision of the structure of the neural convolutional network to better fit special applications for different user types.

5.2. Conclusions and future works

This work is focused on creating a method of recognizing emotions visually. Such a system can be useful to the user with difficulties in expressing or recognizing emotions e.g., users with autism spectrum disorders, sick users in pain, people assisting users to be monitored for health conditions.

This effort has been made possible thanks to recent developments in neural networks, more specifically CNN convolutional neural networks, able to process images accurately.

The use of a simple Webcam has led to good results, but a higher resolution camera may improve accuracy, both in training and grading. The experiments shown that this operation can be done with simple tools: good results can be achieved with inexpensive hardware. The proposed classification can be accurate, though having pictures of very similar emotions, visually speaking: this feature is considered an advantage for the proposed user-centered application, based on facial micro-expressions and small features.

The main disadvantage of the proposed approach is the same of all machine learning techniques: to depend in general on a good training phase.

Footnotes

Acknowledgements

The authors are partially supported by two projects financed by Basic Research Fund, 2015, University of Perugia: G-Lorep and EoL: interfacing between a repository of didactic-scientific material repositories and e-assessment environments (S. Tasso), Computational approaches for the efficient use of General Purpose GPU Computing (O. Gervasi).

References

Alkuhlani,

Nassef and

Farag, Multistage feature selection approach for high-dimensional cancer data, Soft Computing21(22) (2017), 6895–6906. doi:10.1007/s00500-016-2439-9.

Angelov,

Gu,

J.A.

Iglesias,

Ledezema,

Sanchis,

Sipele and

Ramezani, Cybernetics of the mind, learning individual’s perceptions autonomously, IEEE Systems, Man, and Cybernetics Magazine3(2) (2017), 6–17. doi:10.1109/MSMC.2017.2664478. ISSN: 2333-942X.

Bastianelli,

Nardi,

L.C.

Aiello,

Giacomelli and

Manes, Speaky for robots: The development of vocal interfaces for robotic applications, Applied Intelligence44(1) (2016), 43–66. doi:10.1007/s10489-015-0695-5.

Bertola and

Patti, Emotional responses to artworks in online collections, UMAP Workshops Proceedings997 (2013).

Biondi,

Franzoni,

Li and

Milani, Web-based similarity for emotion recognition in web objects, in: Proceedings of the 9th International Conference on Utility and Cloud Computing, UCC 2016, Shanghai, China, December 6–9, 2016, pp. 327–332. doi:10.1145/2996890.3007883.

Caffe Framework: Github: https://github.com/BVLC/caffe, last visited on Sept 12, 2018.

Canossa,

Badler,

M.S.

El-Nasr and

Anderson, Eliciting emotions in design of games – a theory driven approach, CEUR Workshop Proceedings1680 (2016), 34–40.

Chmiel,

Sienkiewicz,

Thelwall,

Paltoglou,

Buckley,

Kappas and

J.A.

Hołyst, Collective emotions online and their influence on community life, PLoS ONE6(7) (2011), e22207. doi:10.1371/journal.pone.0022207.

Cieliebak,

Dürr and

Uzdilli, Potential and limitations of commercial sentiment detection tools, in: CEUR Workshop Proceedings, Vol. 1096, 2013, pp. 47–58.

10.

T.F.

Cootes,

C.J.

Taylor,

D.H.

Cooper and

Graham, Active shape models-their training and application, Computer Vision and Image Understanding61(1) (1995), 38–59. doi:10.1006/cviu.1995.1004.

11.

Cui,

Du,

Shen,

Zhou and

Li, Personalized microblog recommendation using sentimental features, in: 2017 IEEE International Conference on Big Data and Smart Computing, BigComp 2017, Art. No. 7881756, 2017, pp. 455–456. doi:10.1109/BIGCOMP.2017.7881756.

12.

Eveno,

Caplier and

P.Y.

Coulon, Accurate and quasi-automatic lip tracking, IEEE Transactions on Circuits and Systems for Video Technology14(5) (2004), 706–715. doi:10.1109/TCSVT.2004.826754.. ISSN:1051-8215.

13.

Franzoni, Autonomous hexapod robot with artificial vision and remote control by myo-electric gestures, Cyber-Physical Systems for Next-Generation Networks (2018), 143–162. doi:10.4018/978-1-5225-5510-0.ch007.

14.

Franzoni,

Chiancone and

Milani, A multistrain bacterial diffusion model for link prediction, International Journal of Pattern Recognition and Artificial Intelligence31(11) (2017). doi:10.1142/S0218001417590248.

15.

Franzoni and

Gervasi, Guidelines for web usability and accessibility on the Nintendo Wii, in: Transactions in Computer Science VI, Lecture Notes in Computer Science, Vol. 5730, 2009, pp. 19–40. doi:10.1007/978-3-642-10649-1_2.

16.

Franzoni,

Milani,

Pallottelli,

C.H.C.

Leung and

Li, Context-based image semantic similarity, in: 12th International Conference on Fuzzy Systems and Knowledge Discovery, FSKD 2015, Art. No. 7382127, 2015, pp. 1280–1284.

17.

Franzoni,

Milani and

Vallverdu, Emotional affordances in human-machine interactive planning and negotiation, in: Proceedings of WI 2017, Workshop on Affective Computing and Emotion Recognition (ACER), 2017, pp. 924–930. doi:10.1145/3106426.3109421.

18.

Franzoni,

Poggioni and

Zollo, Automated classification of book blurbs according to the emotional tags of the social network, in: Zazie CEUR Workshop Proceedings, Vol. 1096, 2013, pp. 83–94. doi:10.13140/RG.2.1.3194.7689.

19.

Gervasi,

Magni and

Ferri, A method for predicting words by interpreting labial movements, in: Lecture Notes in Computer Science, ICCSA 2016, Vol. 9787, Springer, Beijing (China)2016, pp. 450–464. doi:10.1007/978-3-319-42108-7_34.

20.

Gervasi,

Magni and

Zampolini, Nu!RehaVR: Virtual reality in neuro tele-rehabilitation of patients with traumatic brain injury and stroke, Virtual Reality14(2) (2010), 131–141. doi:10.1007/s10055-009-0149-7.

21.

Gervasi,

Russo and

Vella, The AES implantation based on OpenCL for multi/many core architecture, in: 2010 International Conference on Computational Science and Its Applications, Fukuoka, ICCSA 2010, Washington, DC, USA, IEEE Computer Society, 2010, pp. 129–134. doi:10.1109/ICCSA.2010.44.

22.

G.R.

Hayes,

Hirano,

Marcu,

Monibi,

D.H.

Nguyen and

Yeganyan, Interactive visual supports for children with autism, Personal and Ubiquitous Computing14(7) (2010), 663–680. doi:10.1007/s00779-010-0294-8.

23.

Karayev,

Trentacoste and

Han, Recognizing Image Style, University of California Barkley Adobe, 2014, arXiv:1311.3715 [cs.CV].

24.

Krizhevsky,

Sutskever and

Hinton, Imagenet classification with deep convolutional neural networks, in: 25th International Conference on Advance in Neural Information Processing System, 2012, pp. 1106–1114.

25.

LeCun,

Bengio and

Hinton, Deep learning, Nature521 (2015), 436–444. doi:10.1038/nature14539.

26.

V.I.

Levenshtein, Binary codes capable of correcting deletions, insertions and reversals, Cibernetics and Control Theory, Soviet Physics Doklady10 (1966), 707–710.

27.

T.W.

Lewis and

D.M.W.

Powers, Lip contour detection techniques based on front view of face, Journal of Global Research in Computer Science2(5) (2011), 43–46ISSN: 2229-371X.

28.

Lombardi,

W.R.

Butt and

Grecuccio, Lip tracking towards an automatic lip reading approach, JMPT5 (2014), 1–11. doi:10.4236/jmp.2014.51001.

29.

Lou,

Wu,

R.-D.

Vatavu and

W.T.

Tsai, Personalized gesture interactions for cyber-physical smart-home environments, Science China Information Sciences60(7) (2017). doi:10.1007/s11432-015-1014-7.

30.

P.C.

Mahalanobis, On the generalised distance in statistics,in, Proceedings of the National Institute of Sciences of India2(1) (1936), 49–55.

31.

Mariotti,

Gervasi,

Vella,

Cuzzocrea and

Costantini, Strategies and systems towards grids and clouds integration: A DBMS-based solution, Future Generation Computer Systems88 (2018), 718–729. doi:10.1016/j.future.2017.02.047.

32.

Neumann,

Patricia,

Garnett and

Kersting, Efficient graph kernels by randomization, in: Lecture Notes in Computer Science, LNAI (PART 1), Vol. 7523, (2012), 378–393. doi:10.1007/978-3-642-33460-3_30.

33.

Pantelopoulos and

N.G.

Bourbakis, A survey on wearable sensor-based systems for health monitoring and prognosis, IEEE Transactions on Systems, Man and Cybernetics Part C: Applications and Reviews40(1) (2010), 5306098.

34.

V.M.

Patel,

Gopalan,

Li, Chellappa and

Visual, Domain adaptation: A survey of recent advances, IEEE Signal Processing Magazine32(3) (2015), 53–69. doi:10.1109/MSP.2014.2347059.

35.

K.-C.

Peng,

Chen,

Sadovnik and

Gallagher, A mixed bag of emotions: Model, predict, and transfer emotion distributions, in: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern, 2015, pp. 860–868.

36.

Phelan,

Arden,

Garcia and

Roast, Exploring virtual reality and prosthetic training, in: 2015 IEEE Virtual Reality Conference, VR 2015 – Proceedings, Art. No. 7223441, 2015, pp. 353–354.

37.

R.W.

Picard, Affective computing: Challenges, International Journal of Human Computer Studies59(1–2) (2003), 55–64. doi:10.1016/S1071-5819(03)00052-1.

38.

R.W.

Picard,

Vyzas and

Healey, Toward machine emotional intelligence: Analysis of affective physiological state, IEEE Transactions on Pattern Analysis and Machine Intelligence23(10) (2001), 1175–1191. doi:10.1109/34.954607.

39.

Riganelli,

Franzoni,

Gervasi and

Tasso, EmEx, a tool for automated emotive face recognition using convolutional neural networks, in: ICCSA 2017, Workshop on Emotion Recognition, Lecture Notes in Computer Science, Vol. 10406, 2017, pp. 692–704. doi:10.1007/978-3-319-62398-6_49.

40.

Saeed and

J.-L.

Dugelay, Combining edge detection and region segmentation for lip contour extraction, in: AMDO, 2010, 6th Conference on Articulated Motion and Deformable Objects, Andratx, Mallorca, Spain, July 7–9, 2010, Also published in LNCS, Volume 6169, pp.11–20, 2010.

41.

Saif,

Fernandez,

He and

Alani, Evaluation datasets for Twitter sentiment analysis a survey and a new dataset, the STS-Gold, in: CEUR Workshop Proceedings, Vol. 1096, 2013, pp. 9–21.

42.

Stephane and

Luc, The M2VTS multimodal face database (release 1.00) audio- and video-based biometric person authentication, in: First International Conference, AVBPA’97 Crans-Montana, Proceedings, Switzerland, March 12–14, 1997, Springer, Berlin Heidelberg1997, pp. 403–409. ISBN 978-3-540-68425-1. doi:10.1007/BFb0016021.

43.

Stiefelhagen,

Yang and

Waibel, A model-based gaze tracking system, International Journal of Artificial Intelligence Tools6(2) (1997), 193–209. doi:10.1142/S0218213097000116.

44.

Tasso,

Pallottelli,

Rui and

Laganá, Learning objects efficient handling in a federation of science distributed repositories, in: Lecture Notes in Computer Science, LNCS (PART 1), Vol. 8579, 2014, pp. 615–626. doi:10.1007/978-3-319-09144-0_42.

45.

Vella,

Neri,

Gervasi and

Tasso, A simulation framework for scheduling performance evaluation on CPU-GPU heterogeneous system, in: Lecture Notes in Computer Science, ICCSA 2012, Vol. 7336, Springer, 2012, pp. 457–469. doi:10.1007/978-3-642-31128-4_34.

46.

Voirin,

Payet,

Barrat,

Cattuto,

Khanafer,

Regis,

B.-A.

Kim,

Comte,

J.-S.

Casalegno,

Lina and

Vanhems, Combining high-resolution contact data with virological data to investigate influenza transmission in a Tertiary care hospital, Infection Control and Hospital Epidemiology36(3) (2015), 254–260. doi:10.1017/ice.2014.53.

Automating facial emotion recognition

Abstract

Keywords

1. Introduction

2. Related work

2.1. Deep learning and image classification

2.2. AI-assisted health care

2.3. Affective computing and emotion recognition

3. The proposed method for emotion extraction: EmEx2

Table 1 Identified classes to represent emotions

4.1. CNN training phase

4.2. Implementing the convolutional neural network

4.3. Image classification

5. Evaluation of results and future works

5.1. Evaluation of results

Footnotes

Acknowledgements

References

Table 1
Identified classes to represent emotions