Design of reliable virtual human facial expressions and validation by healthy people

Abstract

The level of realism that real-time virtual humans have reached in the last years enables their use as an alternative to pictures and videos in the remediation of social cognition deficits. This paper presents the engineering principles and tools used to design facial expressions on virtual humans to play basic emotions. The proposal is based on the Facial Action Coding System that makes it possible to easily represent facial expressions. Then, the paper describes how the designed virtual human facial emotions have been assessed by healthy people. For that purpose, 204 healthy participants have taken part in an experiment in which they had to recognize the six basic emotions (each of them with two levels of intensity) depicted by the virtual humans. The overall accuracy of the emotion identification task was 88.25%, which outperforms most results obtained by other authors using virtual humans and/or pictures. The best recognized emotions were neutral, happiness and anger. Remarkably striking was the high success rate gotten for disgust, far superior to previous studies based on virtual reality. Unlike other works, no significant differences were found between women and men in the recognition of emotions, probably due to an enhanced dynamism and realism of the designed human faces. However, age-related differences were found for some emotions in favor of the younger participants. In addition, higher emotion identification rates were detected for higher intensity representations of each emotion, for more dynamic avatars and for faces shown frontally compared to lateral ones. Therefore, the results of the evaluation experiment have demonstrated that virtual humans perfectly convey emotions using facial expressions.

Keywords

Virtual human facial expression facial affect recognition virtual reality validation

1. Introduction

Modeling social interactions is becoming increasingly important in the creation of interactive systems that apply Biomedical Engineering (BME) principles and design concepts [1, 2, 3]. BME pursues to close the gap between engineering and medicine to advance health care, including diagnosis, monitoring, and therapy [4]. Non-verbal communication (gaze, facial movements, gesture, interpersonal distance, etc.) is an essential dimension in the interaction of humans with humans and machines [5]. In this context, facial emotion (affect) recognition is the capacity to distinguish and perceive essential types of affective expressions in faces [6]. This daily life ability is critical for social interaction [7].

Indeed, the way an individual perceives emotional states in others influences his/her social success, which is important for assimilation to the community [8]. There is steady proof that patients with various neuropsychiatric disorders have noteworthy trouble in perceiving emotions communicated by others in a precise way. This creates a distortion of social circumstances that supports the presence of psychotic symptoms and a decrease of social functioning [9]. This deficit in facial affect recognition has been widely observed in psychotic disorders, particularly in schizophrenia and related disorders [10]. The disability is by all accounts stable throughout the disorder, not identified with psychopathology or pharmacological treatment, and autonomous of general cognitive deficits [11, 12, 13].

This is the reason why different mental interventions have been designed to improve facial affect recognition in patients with schizophrenia. In fact, recent meta-analyses have demonstrated promising results of psychotherapeutic methodologies improving facial affect recognition and functionality [9]. For this purpose, computer-based treatments appear to be fitting to date. Indeed, access to digital technology and the Internet to individuals experiencing mental disorders [14] empowers new ways to deal with their illness. Precisely, frameworks for emotion detection [15, 16, 17] and facial emotion detection [18] have been at the focal point of our research. The generation of frameworks based on virtual humans (VHs) for facial affect recognition [19, 20] is one of our current primary interests. In the most recent years, a few works on building multi-modal virtual characters for social cognition remediation programs [21, 22] and proposing the utilization of VHs to describe auditory hallucinations in schizophrenia patients [23, 24] have been presented.

The primary effort to reproduce human faces in 3D using computer graphics imagery is probably over 45 years old [25]. From that point forward, 3D human faces have been widely used in computer games and films. Current advances in graphics technologies are moving out of the uncanny valley, as real-time rendered virtual characters turn out to be increasingly realistic. Hence, an absence of human-like facial expressions portrayed by virtual characters increases the dimension of strangeness felt despite physical authenticity [26]. Additionally, physical authenticity ought to be accompanied by social realism [27], as the individuals’ assumptions regarding realistic movements or behaviors are raised by the level of physical authenticity [28].

Table 1
Action Units used to describe the six basic emotions (based on [32, 29, 34])

AU	Name	Surprise	Fear	Happiness	Sadness	Disgust	Anger
1	Inner Brow Raiser	(1.2)	(1.2)		(1.2)
2	Outer Brow Raiser	(1.2)	(1.2)
4	Brow Lowerer		(1.2)		(1.2)		(1.2)
5	Upper Lid Raiser	(1.2)	(1.2)				(1.2)
6	Cheek Raiser			(1.2)
7	Lid Tightener						(1.2)
9	Nose Wrinkler					(1.2)
12	Lip Corner Puller			(1.2)
15	Lip Corner Depressor				(1.2)	(1.2)
16	Lower Lip Depressor					(1.2)
17	Chin Raiser				(2)		(1)
20	Lip Stretcher		(1.2)
23	Lip Tightener						(1.2)
25	Lips Part						(2)
26	Jaw Drop	(1.2)	(1.2)

A few physical models depicting facial expressions have been proposed from the perspective of muscle activation and can be used to accomplish a higher level of authenticity. One of the most popular models in the literature is the one by Ekman and colleagues [29]. Notwithstanding, conveying emotions using virtual characters is not a simple task, and this might be identified with the absence of knowledge about the time course of facial movements and the affective content felt from those developments [30].

The present paper introduces the engineering principles and tools used to design facial expressions on virtual humans for showing basic emotions. At that point, an investigation is done with healthy individuals to assess that the designed facial emotions are accurately interpreted by people who have no social cognitive shortfalls. Thus, the objective of the present paper is the description of the design process of virtual human facial expressions and their validation by 204 healthy participants. The novelty of the article lies, first, in the description of the virtual human design process that combines several techniques and commercial software packages. Secondly, in the large sample size to validate the design, and finally, the designed facial expressions are shown to each participant with two levels of intensity (low and high), two levels of dynamism (low and high), and from different perspectives (frontal and lateral views). Should the outcomes be concluding, our virtual human emotional faces could be used as correlation with patients with schizophrenia. Conversely, if healthy people are not able to correctly detect facial expressions on virtual humans with a high accuracy, it would be nonsense to use the avatars in further studies with real patients. The current paper must be considered as an initial move towards planning a complete therapy to upgrade facial affect recognition in these patients.

2. Design of facial expressions in virtual humans

2.1 Facial action coding system

People are social creatures who need to convey emotions in order to socialize. Individuals express emotions in various ways and the face assumes a significant role in how emotions are transmitted both in verbal and non-verbal communication. Affect has been contemplated in the field of Psychology and Psychiatry. For example, Ekman and Friesen [31] concluded that there exist six universal basic emotions (Anger, Disgust, Fear, Happiness, Sadness, and Surprise). Facial muscles are used to demonstrate these emotions, changing that way the appearance in the face. A few studies attempted to discover a framework to evaluate the various changes in the face providing a few appro-ximations [32].

To reproduce the adjustments in the face, our proposal is not situated in an exact representation and investigation of the muscles, but in the well-known Facial Action Coding System (FACS). We picked this framework since it is the most generally used and has demonstrated its adequacy through a psychometric evaluation for assessing spontaneous expressions [33]. Subsequently, it gives more information, such as intensity, than other systems about the adjustments in the face [32]. In fact, FACS, designed in 1978 and modified and improved in 2002 [29], is a notable system that categorizes facial movements based on various Action Units (AUs), instructing how to perceive and score them.

Every AU characterizes a group of muscles that work together to provoke a modification on the facial appearance. The various AUs are gathered according to the location of the facial muscles that are separated in upper and lower face muscles. Upper face muscles incorporate eyebrows, forehead, eye cover fold, and upper and lower lids. The lower face incorporates the muscles around mouth and lips, and it is divided in different classifications as indicated by the muscles’ movement directions. There are other AUs based on muscles that move the neck and the gaze direction.

2.2 Facial expressions in FACS

Only fifteen AUs from the twenty-eight fundamental ones are required to describe the six basic emotions (Anger, Disgust, Fear, Happiness, Sadness, and Surprise) and the Neutral expression used in this research. The AUs used and their connection with every emotion is shown in Table 1.

The table shows that:

•
Twelve animations were modelled.
•
Two levels (1.2) of animations were designed for every emotion.
•
No AU was needed to model the Neutral expression.
•
Each level indicates the emotion intensity; a higher number shows greater emotional intensity, (2) $>$ (1).
•
New AUs were added for some emotions, indicating that more muscles are moved, and more facial changes appear.

2.3 Generation of affective virtual humans

The work flow for the generation of the affective VHs is one of the novelties of this proposal. The contribution of our approach lies in the combination of several well-known techniques in the game development and VR fields, using different software packages in order to obtain the desired result. This work flow process is depicted in Fig. 1 and described in the following paragraphs.

Figure 1.

Work flow of the generation of affective virtual humans.

We began by choosing and tuning two predefined characters accessible in Adobe Fuse CC [35]. Adobe Fuse CC is a software tool principally aimed at video game developers to create 3D characters using an advanced character creation editor. The characters gotten by using this tool exploit numerous visual features that are typical in recent high-end video games, specifically high-resolution textures, normal maps, ambient occlusion maps, etc. Besides, they are completely configurable, from the face to the length of the arms, torso or the garments they wear. For this work, predefined VHs were chosen (using the option “Assemble”) and not changed apart from picking garments and choosing a hair style (option “Clothing”) that did not occlude significant parts of the face like forehead and eyes.

By clicking the “Send to Mixamo” button from inside Adobe Fuse CC, the characters were then exported to Mixamo [36], a web platform that provides auto-rigging of humanoid 3D models. Rigging is a technique used in skeletal animation for representing a 3D character model using a series of interconnected digital bones. It refers to the process of creating the bone structure of a 3D model, which enables manipulating the 3D model like a puppet for animation. At that point, an idle animation was chosen from the available ones. This platform likewise provides some basic form of facial animation that can be used in other programs. In any case, it did not depend on FACS so that there is no immediate correspondence between them.

From that point onward, the resulting 3D character is brought into the 3D Studio Max [37] authoring tool to generate the AUs, always starting from the Neutral facial expression. We used blend shapes, also known as morph animation targets, which consists in altering the mesh likewise and storing the vertex positions for every AU. At that point, these AUs are smoothly morphed and combined to shape the wanted facial animation by using a “Morph Modifier”. Alternatives to implement AUs are the utilization of a muscle-based animation system or a hierarchy of bones to modify the geometry. A blend shape animation system was chosen because of its straightforwardness and the likelihood to combine AUs to generate complex facial expressions.

Once all the AUs were incorporated into the virtual human models, they were exported into Unity 3D [38], the real-time engine used to play the animations. Time is an essential information for the optimal planning and scheduling of interaction systems [39]. This software tool enables to use the initial idle animation incor-porated into Mixamo in the same manner as the blend shapes included into 3D Studio Max. At that point, the facial expressions were upgraded by adding wrinkles to the models. For this, a custom surface shader using several “Normal map” textures was generated in Unity 3D and used as a material parameter. Normal maps are used to simulate subtleties in an object’s surface by changing the vertices’ normal and, therefore, affecting light calculation on the surface. Everyone incorporated an alternate wrinkle pattern related to each emotion. This shader described all the visual appearance of the virtual human’s face, using textures for the skin color, normal mapping, reflections (specularity) and ambient occlusion. The standard lighting model was used, and shadows were enabled on all light types (fullforwardshadows option).

The textures generated by Adobe Fuse CC were used in the surface shader. For each facial expression, the Nvidia Normal Map Filter for Adobe Photoshop [40] was used to generate the normal maps based on photos of individuals delineating the facial emotion and on the wrinkle descriptions provided by Ekman and Friesen [34]. We used 7 different normal maps per virtual human, one per each facial expression plus the Neutral one. At last, the custom surface shader smoothly interpolated between the Neutral normal map and the normal map of every facial expression, simulating a dynamic generation of wrinkles. This interpolation was synchronized with the blend shape animation of the facial expression in a way that the wrinkles appeared gradually as the animation progressed. A value of 0 for a blend shape related to a given facial expression means that this facial expression is not displayed by the virtual human, while 100 means that the facial expression is completely displayed. This denotes that the vertices of the face depicting the expression have moved from the initial position (Neutral) to the final position for the given expression. Similarly, this numeric value ranging from 0 to 100 was used to blend the normal maps from the Neutral normal map to the corresponding normal map for the target facial expression. These two actions happening together gave the impression that the wrinkles are generated by the motion of the face when depicting each expression. For instance, Fig. 2 shows how the Surprise and Disgust facial expressions are enhanced by using wrinkles.

Figure 2.

Male virtual human demonstrating the surprise emotion (top images) and female virtual human showing disgust (bottom). Both pairs of pictures show how they look without (left) and with (right) normal maps. Please pay attention on the wrinkles on the forehead and on both sides of the mouth in the male VH, and the wrinkles around the nose and the mouth in the female.

As the fine-tuning of most parameters related to the AUs has been done by hand, the way decisions have been taken should be highlighted. Two engineers designed the avatars from scratch to a first version. Then, the other engineer and the psychiatrists discussed about the likeness of the virtual human’s emotions to the human’s ones. The final version of the avatars was obtained in an iterative manner.

Figure 3.

Different variations of the virtual humans used in the emotion recognition evaluation.

2.4 Improvements in facial expressions over a previous version

In a previous experiment [41], we detected troubles with some facial expressions. Originally, Happiness included a third variation of lower intensity, a very subtle smile which was mainly confused with the Neutral expression. It seems that the participants did not consider it to be a sign of happiness, even when they noticed the smile. Therefore, that variation was excluded from the present study. There were also some identification problems for Disgust and Sadness, which lead to their redesign.

3. Validation of facial expressions by healthy people

A validation of the system was performed to assess its suitability to propose a future clinical therapy for enhancing facial affect recognition in people with schizophrenia. Therefore, in first term it was determined as mandatory to evaluate if healthy people are capable of recognizing the virtual facial expressions generated. In addition, the acceptance and use of such avatar-based proposal has been positively assessed by 41 therapists in a recent previous work [42].

Figure 4.

Several variations of the virtual humans used in the emotion recognition evaluation.

3.1 Participants

The sample size was 204 healthy volunteers. The single inclusion criterion was to be aged between 20 and 79 years. The mean age of the participants was $M=$ 47.21 ( $\textit{SD}=$ 15.64). The participants were recruited from Madrid and Albacete (Spain). Exclusion criteria included having a diagnosis of mental illness, a personal history of medical illness, and a first-degree family history of psychosis. The sample was stratified by gender (50% male, 50% female). All the participants were volunteers and did not receive any compensation for their participation.

3.2 Experimental procedure

As explained before, this experiment aims at exam-ining whether the emotions shown by the designed virtual humans convey the same emotional meaning as real human expressions. Two variations of each of the original VHs previously used by our research team [41] were designed. This made a total amount of six VHs (see Fig. 3). Four were Caucasian (two males and two females) and two African (one male and one female). All six VHs were designed with two age representations, namely adult of about 30 years and old age. As said, seven emotions were portrayed for this experiment based on the FACS system: the six basic emotions and the Neutral expression. Two different intensities were implemented for each basic emotion, giving a total of thirteen intensity emotions, labeled as neutral, surprise1, surprise2, fear1, fear2, anger1, anger2, disgust1, disgust2, happiness1, happiness2, sadness1, and sadness2 (written in lower case).

Table 2
Emotion recognition rates for each of the 13 animations. Average successful recognition rate: 88.25%

	Neutral	Surprise	Fear	Anger	Disgust	Happiness	Sadness
Neutral	96.3%	0.4%	0.1%	0.2%	0.5%	0.0%	2.45%
Surprise1	0.5%	92.6%	6.3%	0.0%	0.1%	0.0%	0.5%
Surprise2	0.5%	90.0%	9.2%	0.0%	0.2%	0.1%	0.0%
Fear1	1.0%	9.9%	74.4%	0.1%	1.1%	0.0%	13.5%
Fear2	0.9%	18.9%	79.5%	0.2%	0.4%	0.0%	0.1%
Anger1	1.1%	1.8%	2.3%	87.7%	4.8%	0.0%	2.2%
Anger2	1.0%	1.0%	1.2%	92.9%	3.7%	0.1%	0.1%
Disgust1	1.5%	1.8%	1.1%	12.9%	82.0%	0.0%	0.7%
Disgust2	0.2%	0.4%	1.3%	8.3%	89.6%	0.0%	0.1%
Happiness1	2.7%	0.6%	0.2%	0.2%	0.1%	96.1%	0.0%
Happiness2	2.8%	0.7%	0.1%	0.6%	0.4%	95.2%	0.1%
Sadness1	5.4%	3.2%	2.0%	0.1%	0.7%	0.2%	88.4%
Sadness2	1.7%	3.8%	6.7%	1.0%	4.2%	0.0%	82.6%

Each facial expression was presented to the partici-pants four times (two frontal and two lateral views, one from each side). This made a total of fifty-two facial expression representations. Thirty-six of them were shown using the younger adult versions of the Caucasian VHs, eight using the aging adult version of the Caucasian VHs and eight using the African VHs. These fifty-two facial representations using different variations were randomly presented to the participants. Half of them were presented with less dynamism (only the most characteristic facial features presented movement, i.e. the AUs described in Table 1) and the other 50% with more dynamism, including movement of the head and the neck in order to bring more realism to the expressions.

The validation process is depicted in Fig. 4. It starts by requiring the participant to complete two questionnaires. The first one includes a collection of social, demographic and clinical data. The second is the Spanish version of the positive and nega-tive affect schedule (PANAS [43]), which is a 20-item self-report questionnaire that measures the individual’s positive and negative affect. This schedule is included in order to control the participant’s mood state or non-specific depression symptoms. After completing the questionnaires, the participant starts the evaluation test.

A short tutorial describing the task is presented to the participant. He/she must press a button to start the experiment. The validation process starts by displaying the first facial expression. Each time a new facial expression is shown, the character’s face is faded-in from a black background. A transition is made from the neutral expression to the new emotion (lasting 0.4 seconds), which is held for 1.5 seconds. Then, there is a new transition to the neutral expression (again, 0.4 seconds). This is in accordance with transition times studied in well-known works [44], as expression time lasts between 0.5 and 4 seconds.

Once this process has finished, a panel is presented to the participant asking him/her for the expression just offered by the VH. This panel also includes a button for each of the six basic emotions and the Neutral expression. Once the participant has selected an option, the character face is faded-out. This process is repeated for each of the fifty-two facial expressions. The experiment finishes once the system has presented all the facial expressions to the participant.

The same idle animation is used for all virtual humans. This animation is subtle enough not to distract the participant, while it adds a slight swing that provides more realism to the character. Similarly, a blinking animation is also added, but only during the time the system is waiting for a participant’s response, not during the actual visualization of the emotion.

4. Results and discussion

4.1 Accuracy in facial expression recognition

The number of correct answers did not follow a normal distribution (Kolmogorov-Smirnov test: $Z=$ 0.165, $p<$ 0.001). Therefore, non-parametric tests were used to compare the results.

As shown in Table 2, the percentage of successful recognition for each face expressing an emotion is high, all well above random chance (set at 14%), and all above 80% except fear1 and fear2. These results are consistent with previously published studies in which Fear and Disgust were the least identified emotions [45, 46]. In our study, although Fear is the least recognized emotion, it obtained a percentage of hits similar to previous studies [46, 20].

Table 3
Emotion recognition rate with (left) and without (right) VH movement. Average successful recognition rates are 90.12% and 85.03%, respectively

	Surprise	Fear	Anger	Disgust	Happiness	Sadness
Surprise1	92.1%	7.0%	0.0%	0.0%	0.0%	0.7%
Surprise2	91.7%	7.3%	0.0%	0.3%	0.0%	0.0%
Fear1	12.4%	82.5%	0.2%	0.9%	0.0%	3.7%
Fear2	18.7%	79.5%	0.3%	0.3%	0.0%	0.0%
Anger1	2.0%	2.5%	89.5%	3.0%	0.0%	1.5%
Anger2	0.7%	1.2%	93.4%	2.8%	0.2%	0.0%
Disgust1	2.0%	1.8%	3.5%	91.7%	0.0%	0.5%
Disgust2	0.3%	1.8%	2.3%	95.2%	0.0%	0.3%
Happiness1	0.5%	0.5%	0.3%	0.0%	96.3%	0.0%
Happiness2	1.0%	0.3%	0.5%	0.0%	95.4%	0.0%
Sadness1	4.6%	0.9%	0.2%	0.7%	0.0%	89.2%
Sadness2	3.9%	4.4%	1.4%	3.7%	0.0%	84.9%

	Surprise	Fear	Anger	Disgust	Happiness	Sadness
Surprise1	93.3%	5.5%	0.0%	0.3%	0.0%	0.3%
Surprise2	88.4%	10.9%	0.0%	0.2%	0.2%	0.0%
Fear1	7.2%	65.5%	0.0%	1.3%	0.0%	24.2%
Fear2	19.0%	79.6%	0.2%	0.5%	0.0%	0.2%
Anger1	1.7%	2.2%	86.1%	6.5%	0.0%	2.9%
Anger2	1.3%	1.3%	92.3%	4.6%	0.0%	0.3%
Disgust1	1.7%	0.5%	21.7%	72.8%	0.0%	1.0%
Disgust2	0.5%	1.0%	14.1%	84.2%	0.0%	0.0%
Happiness1	0.7%	0.0%	0.2%	0.2%	95.9%	0.0%
Happiness2	0.5%	0.0%	0.7%	0.7%	95.0%	0.2%
Sadness1	1.6%	3.1%	0.0%	0.8%	0.5%	87.4%
Sadness2	3.6%	9.4%	0.5%	4.7%	0.0%	80.0%

A closer look at the table shows that fear1 was confused with Surprise and Sadness, while fear2 was mainly confused with Surprise. This is astonishing as it differs from the results of our previous experiment using the same facial expressions [41], especially for Fear. With regards to disgust2 and sadness2, a great improvement is noticeable, as they improve from 68.5% to 89.6% for disgust2 and from 31.5% to 82.6% in the case of sadness2.

Regarding the rest of the emotions, the only one with a confusion percentage above 10% is disgust1, which was mainly confused with Anger. Remarkably, this is also consistent with previous literature [45, 47]. Nevertheless, the high percentage of successes around Disgust is striking in relation to previous outcomes [20, 45, 46, 47]. Previous studies have reported a limitation in the recognition of Disgust through virtual reality. This phenomenon may be due to the difficulty of authenticity recreating the nasolabial area [48]. For this reason, we paid special attention to this region during the redesign of emotions.

In general terms, the average percentage of successful recognition is high (88.25%), in this way improving in more than 5% our previously reported results (83.56%) [41]. This degree of accuracy is superior to that obtained in previous works with classical stimuli (natural faces) like the Ekman-60 Faces Test [49] or the Penn Emotion Recognition Test-96 Faces version [50]. With the first stimulus, the general percentage of success in two different populations was approximately 82% [49, 51]. With the second stimulus, a general percentage of success close to 70% was obtained [20]. In the latter study, the classic stimulus was compared with a set of virtual faces. With the virtual stimulus the overall accuracy was over 73%.

Similar studies based on virtual faces obtained an average percentage of success ranging from 62 to 78% [45, 47, 46, 19]. The last two papers included more emotions, namely contempt, embarrassment and pride. Thus, the results obtained in our study are in line with previous studies using virtual humans for the recognition of emotions. Only one previous study obtained a percentage of success superior to all other studies (91.7% [52]). Nonetheless, this study was only applied to 41 participants. Moreover, other findings of our experiment are consistent with previous studies using virtual humans to recognize emotions, where the Neutral expression and the Happiness emotion were the most easily recognized, followed by Anger and Surprise.

4.2 Influence of gender, age and emotion intensity

There was no significant difference in the overall number of correct answers for all emotions per gender (Mann-Whitney: $U=$ 5085, $p=$ 0.781). We also tested the difference for individual emotions, showing all of them non-significant results in a similar way. Curiously, previous studies have reported differences in the recognition of facial emotions in terms of gender in favor of women. While most studies used static stimuli and/or stimuli that represent extreme examples of facial emotions [51], recent studies with more subtle and dynamic facial expressions have obtained similar results to ours [53].

Table 4
Emotion recognition rates for frontal (left) and lateral (right) views of the VHs. Average successful recognition rates are 9.06% and 87.45%, respectively

	Neutral	Surprise	Fear	Anger	Disgust	Happiness	Sadness
Neutral	97.1%	0.7%	0.0%	0.0%	0.0%	0.0%	2.19%
Surprise1	1.0%	92.4%	5.9%	0.0%	0.2%	0.0%	0.5%
Surprise2	0.2%	92.4%	7.4%	0.0%	0.0%	0.0%	0.0%
Fear1	1.0%	8.5%	76.1%	0.2%	1.2%	0.0%	12.9%
Fear2	0.7%	16.3%	81.7%	0.5%	0.5%	0.0%	0.2%
Anger1	1.0%	1.7%	2.5%	87.1%	5.2%	0.0%	2.5%
Anger2	1.0%	1.2%	1.4%	92.3%	3.9%	0.0%	0.2%
Disgust1	1.5%	1.2%	1.0%	12.8%	82.5%	0.0%	1.0%
Disgust2	0.2%	0.5%	2.2%	9.3%	87.5%	0.0%	0.2%
Happiness1	2.9%	0.5%	0.0%	0.2%	0.2%	96.1%	0.0%
Happiness2	3.0%	0.2%	0.0%	0.5%	0.2%	96.0%	0.0%
Sadness1	3.7%	2.0%	1.5%	0.0%	0.2%	0.5%	92.2%
Sadness2	1.2%	2.7%	6.3%	1.0%	4.4%	0.0%	84.5%

	Neutral	Surprise	Fear	Anger	Disgust	Happiness	Sadness
Neutral	95.6%	0.0%	0.2%	0.5%	1.0%	0.0%	2.72%
Surprise1	0.0%	92.9%	6.7%	0.0%	0.0%	0.0%	0.5%
Surprise2	0.7%	87.6%	11.0%	0.0%	0.5%	0.2%	0.0%
Fear1	1.0%	11.3%	72.7%	0.0%	1.0%	0.0%	14.0%
Fear2	1.0%	21.4%	77.4%	0.0%	0.2%	0.0%	0.0%
Anger1	1.2%	1.9%	2.2%	88.4%	4.4%	0.0%	1.9%
Anger2	1.0%	0.7%	1.0%	93.5%	3.5%	0.2%	0.0%
Disgust1	1.5%	2.4%	1.2%	12.9%	81.5%	0.0%	0.5%
Disgust2	0.2%	0.2%	0.5%	7.3%	91.7%	0.0%	0.0%
Happiness1	2.4%	0.7%	0.5%	0.2%	0.0%	96.1%	0.0%
Happiness2	2.7%	1.2%	0.2%	0.7%	0.5%	94.4%	0.2%
Sadness1	7.1%	4.4%	2.5%	0.2%	1.2%	0.0%	84.5%
Sadness2	2.2%	5.0%	7.2%	1.0%	4.0%	0.0%	80.7%

Table 5

Emotion recognition rates with virtual human movement and using front cameras (left), and without virtual human movement and using lateral cameras (right). Average successful recognition rates are 91.18% and 84.42%, respectively

	Surprise	Fear	Anger	Disgust	Happiness	Sadness
Surprise1	92.6%	6.4%	0.0%	0.0%	0.0%	0.5%
Surprise2	94.5%	5.5%	0.0%	0.0%	0.0%	0.0%
Fear1	11.2%	83.6%	0.5%	0.5%	0.0%	3.7%
Fear2	15.6%	81.9%	0.5%	0.5%	0.0%	0.0%
Anger1	2.0%	3.1%	88.3%	4.1%	0.0%	1.0%
Anger2	1.4%	1.4%	92.5%	3.3%	0.0%	0.0%
Disgust1	1.0%	1.5%	2.0%	94.6%	0.0%	1.0%
Disgust2	0.5%	3.0%	3.0%	92.9%	0.0%	0.5%
Happiness1	0.0%	0.0%	0.0%	0.0%	97.8%	0.0%
Happiness2	0.5%	0.0%	0.0%	0.0%	96.5%	0.0%
Sadness1	2.8%	0.0%	0.0%	0.5%	0.0%	93.4%
Sadness2	3.3%	4.7%	1.4%	3.7%	0.0%	85.6%

	Surprise	Fear	Anger	Disgust	Happiness	Sadness
Surprise1	94.4%	5.6%	0.0%	0.0%	0.0%	0.0%
Surprise2	86.0%	13.0%	0.0%	0.5%	0.5%	0.0%
Fear1	8.9%	63.0%	0.0%	0.5%	0.0%	25.5%
Fear2	20.8%	77.8%	0.0%	0.5%	0.0%	0.0%
Anger1	1.9%	2.4%	86.1%	6.7%	0.0%	1.9%
Anger2	1.6%	1.0%	92.7%	4.7%	0.0%	0.0%
Disgust1	1.9%	0.5%	20.0%	74.9%	0.0%	0.9%
Disgust2	0.5%	0.5%	13.0%	86.1%	0.0%	0.0%
Happiness1	0.5%	0.0%	0.0%	0.0%	97.1%	0.0%
Happiness2	0.9%	0.0%	0.5%	0.9%	94.5%	0.5%
Sadness1	2.2%	3.2%	0.0%	1.6%	0.0%	83.8%
Sadness2	5.3%	10.6%	0.5%	4.3%	0.0%	76.6%

Regarding age, we created two groups (younger and older adults) using the median (47 years) to split the two classes. Similarly, no significant differences were found in the total number of correct answers. However, there were differences for some individual emotions. The results for anger1, joy2 and sadness1 were significantly better for the younger group ( $U=$ 4405.5, $p=$ 0.017, $U=$ 4656, $p=$ 0.041, and $U=$ 4329.5, $p=$ 0.012, respectively). In this regards, some authors have reported that there is a decline in facial emotion recognition ability as people grow older [54, 55], suggesting that older adults pay less attention to socially relevant areas such as eyes or mouth [56]. Another factor that could influence our results is the greater difficulty that older people present when using digital technological devices.

As discussed earlier, each emotion was depicted by the VHs with two different intensities. In this case, the Wilcoxon Signed Rank test revealed a significant difference for the intensity parameter ( $Z=$ $-$ 2.317, $p=$ 0.020). As expected, higher intensity emotions obtained a higher number of correct answers. Again, this result is in line with a previous work [46].

A deeper study focusing on individual emotions revealed that fear1, anger1 and disgust1 obtained a higher number of correct answers for higher intensity emotions ( $Z=$ 2.349-, $p=$ 0.019, $Z=$ $-$ 3.586, $p<$ 0.001, and $Z=$ $-$ 4.149, $p<$ 0.001), while it was the opposite for surprise1 and sadness1 ( $Z=$ $-$ 1.994, $p=$ 0.046, and $Z=$ $-$ 3.165, $p=$ 0.002). surprise2 and sadness2 got fewer hits than surprise1 and sadness1, respectively, because they were often confused with Fear. Fear is also often mistaken with Surprise in other studies [45]. Social psychology states that the expression of Fear and Surprise often occurs simul-taneously, such as when Fear is suddenly felt due to an unexpected threat [34].

4.3 Influence of the virtual human movement and the camera angle

A subtle movement of the VHs’ head and upper body was included in 50% of the emotions presented to the participants in order to study its effect in the recognition rate. Our hypothesis was that it would increase the rates of successful recognition because the number of hints were increased. The Wilcoxon Signed Rank test ( $Z=$ $-$ 5.666, $p<$ 0.001) confirmed the hypothesis. A further study of the impact on the indivi-dual emotions showed that the improvement was significant for fear1, disgust1, sadness1 and sadness2 ( $Z=$ $-$ 3.817, $p<$ 0.001, $Z=$ $-$ 2.667, $p=$ 0.008, $Z=$ $-$ 2.168, $p=$ 0.030, and $Z=$ $-$ 1.993, $p=$ 0.046, respectively).

Table 3 shows the confusion matrices for both conditions. The average successful identification rate with movement was 90.12% while it was 85.03% for no movement. A general improvement is noticeable, for example, taking a look at fear1. Using the less dynamic virtual humans, this emotion was confused with Sadness almost one-fourth of the number of times it was presented to the participants. This was reduced to 3.7% when more dynamic VHs were used. It makes sense that a greater dynamism in the area of the neck and face is related to a better identification of emotions due to the increasing realism.

In addition, three different camera angles were randomly used to present the VHs to the participants (50% from the front and 50% from both sides). Our hypothesis was that front views would obtain a higher number of correct answers than lateral views. This was confirmed for the total number of correct answers by a Wilcoxon Signed Rank test ( $Z=$ $-$ 2.829, $p=$ 0.005). Focusing on individual emotions, the test revealed that there was a significant difference for sadness1 and sadness2 ( $Z=$ $-$ 3.166, $p=$ 0.002, and $Z=$ $-$ 2.129, $p=$ 0.033). In both cases, the results were better for frontal views in regards to lateral views. We could not find significant results for the other emotions. The aver-age successful identification rate for front views was 89.06%, while for the lateral views it was 87.45%. This time, even though it is true that there is an improvement due to the use of frontal cameras only, this improvement is less ostensible. The most significant improvement is related to the confusion of fear2 with Surprise, which experiences 5% reduction when only front cameras are used (see Table 4).

Figure 5.

Histogram showing number of participants in terms of ranges of bad responses.

Figure 6.

Identification errors per each of the 52 faces presented to the participants. The trend line shows a negative slope of 18%.

This confirmed that the combination of virtual human movement and frontal cameras obtains the best possible results. Table 5 shows this enhanced combination side-by-side with the worst combination, which does not use virtual human movement and only lateral cameras. This comparison maximizes the differences. Apart from the ones mentioned before (fear1 and fear2), there are other noticeable differences between both tables. sadness2 is confused with Fear 10% of the times it is presented to the participants in the worst combination (see left side of Table 5). This percentage is reduced to 5% for the best combination (see right side of Table 5). Moreover, this issue is of great importance, as there are no studies published to date that include different camera angles and compare the identification of emotions in the avatars presented with frontal and lateral views.

4.4 Errors per participant

The average number of errors per participant (measured as the mistakes made when identifying an emotion performed by a virtual human) was 6.11 ( $\textit{SD}=$ 5.03). It is a reduction from our previous study, where this number was 9.7 ( $\textit{SD}=$ 3.7). The histogram in Fig. 5 groups the participants by number of recognition errors. The largest group gathers those participants who made a total of 5 or less errors (115 volunteers). In addition, it is also apparent that the number of participants decreases as the intervals of errors get bigger, especially for the number of people making 15 or more mistakes, which is only 13 people. These results demonstrate that only a few participants made many recognition errors. The maximum number of errors per session registered is 26, which were made by a single participant.

4.5 Influence of the number of faces presented on the number of errors made

We wanted to study whether the number of emotion identification errors increased or decreased during the progression of the test. Figure 6 plots in the $X$ axis the faces to be identified (from 1 to 52) by the participants, while the $Y$ axis shows the number of people failing to recognize the emotion presented (from 0 to 204). Notice that the order of presentation of emotions differed from one participant to another.

Two situations would be possible: (1) more errors are made at the beginning, which would mean that the participants learn and improve along the test, or, (2) more errors are made at the end, which would mean that they become tired. The trend line (colored in red in Fig. 6) shows a reduction in the number of errors as the test progresses. The slope of the line is $-$ 0.1781, which is a reduction of approximately 18% in the number of errors. This result enforces the hypothesis that the participants improve along the test.

The loss of progressive attention during the performance of a task is known, as well as the improvement in cognitive performance after the repetition of an instruction. However, no previous studies of facial recognition of emotions that evaluate the “fatigue effect” or “learning effect” have been published. Therefore, our results cannot be compared with any previous work.

5. Conclusions

The objective of this paper was twofold. On the one hand, it aimed at describing the complete engineering design process of virtual humans capable of expressing facial emotions. Following the Facial Action Coding System, six facial expressions with two levels of intensity were designed to convey the six basic emotions (Anger, Disgust, Fear, Happiness, Sadness, and Surprise), plus the Neutral one. On the other, this paper has described in detail the assessment by 204 healthy people of the avatar expressions designed.

The work described in this paper is a follow-up of a previous pilot study [41]. The current work has improved the design of the virtual humans, introducing variations for age and race, and increasing the number of participants in the evaluation from 23 to 204. This has enabled a further analysis of the results and allowed us to reach out wider and more precise conclusions.

The overall success recognition rate was 88.25%, which is consistent with (and mostly outperforms) the results obtained by other authors in previous works using virtual humans and/or pictures [19, 20, 46, 53].

Moreover, age and gender were found to have no statistically significant influence on the overall recognition rate. However, the difference of two age groups proved to be significant for some emotions, being better for the younger group. Another interesting result is that the intensity of each emotion was found statistically significant, being the more intense emotional expression easier to recognize by the participants in respect to the less intense one. Other technical aspects have also been evaluated, such as the camera angle in which the faces were presented to the participants, and the level of dynamism of the VH. The results showed that frontal cameras (versus lateral cameras) and more dynamic VHs (in relation to less dynamic VHs) provided the best results in terms of overall successful recognition rates.

In summary, the results show that the virtual faces designed in the experiment are valid for accurately recreating human facial affect expressions. Current findings show that virtual reality environments allow the design of virtual faces that can be controlled externally and in real time, overcoming some of the limitations associated with the use of static faces. This has clinical implications, since the advances provided by virtual reality for sure help to design therapies for patients with difficulties in identifying basic emotions.

The next objective of our research team is the design of a facial affect recognition therapy for patients with schizophrenia. The lessons learned through the current research can also be used to adapt and personalize the therapies to each patient. For example, diff-erent camera angles and even more dynamic VHs can be used to increase or decrease the learning curve of emotion recognition tasks. In future work, the idea and methodology proposed in this paper could be easily extended to various science and engineering applications [57, 58, 59].

Footnotes

Acknowledgments

This work was partially supported by Spanish Ministerio de Ciencia, Innovación y Universidades, Agencia Estatal de Investigación (AEI) / European Regional Development Fund (FEDER, UE) under PID2019-106084RB-I00, DPI2016-80894-R and TIN2015-72931-EXP grants, and by Biomedical Research Networking Centre in Mental Health (CIBERSAM) of the Instituto de Salud Carlos III.

References

Gorbunov

Rauterberg

Barakova

. A cognitive model of social preferences in group interactions. Integrated Computer-Aided Engineering. 2018; 10;26: 1-12.

García-Martínez

Martínez-Rodrigo

Alcaraz

Fernández-Caballero

. A Review on Nonlinear Methods Using Electroencephalographic Recordings for Emotion Recognition. IEEE Transactions on Affective Computing. 2018, p. 1.

Sánchez-Reolid

García

Vicente-Querol

Fernández-Aguilar

López

Fernández-Caballero

, et al. Artificial Neural Networks to Assess Emotional States from Brain-Computer Interface. Electronics. 2018; 7(12).

Reisman

Ritter

Hazelwood

Michniak

Valdevit

Ascione

. Biomedical Engineering Principles. CRC Press, 2018.

Almagro-Cádiz

Fresno

De la Paz López

. Speech gestural interpretation by applying word representations in robotics. Integrated Computer Aided Engineering. 2018; 26: 97-109.

Russell

. Is there universal recognition of emotion from facial expression? A review of the cross-cultural studies. Psychological Bulletin. 1994; 115(1): 102.

Sachs

Winklbaur

Jagsch

Lasser

Kryspin-Exner

Frommann

, et al. Training of affect recognition (TAR) in schizophrenia – impact on functional outcome. Schizophrenia Research. 2012; 138(2-3): 262-267.

Wölwer

Brinkmeyer

Stroth

Streit

Bechdolf

Ruhrmann

, et al. Neurophysiological correlates of impaired facial affect recognition in individuals at risk for schizophrenia. Schizophrenia Bulletin. 2011; 38(5): 1021-1029.

Bordon

O’Rourke

Hutton

. The feasibility and clinical benefits of improving facial affect recognition impairments in schizophrenia: Systematic review and meta-analysis. Schizophrenia Research. 2017; 188: 3-12.

10.

Marwick

Hall

. Social cognition in schizophrenia: a review of face processing. British Medical Bulletin. 2008; 88(1): 43-58.

11.

Fernández-Sotos

Torio

Fernández-Caballero

Navarro

González

Dompablo

, et al. Social cognition remediation interventions: a systematic mapping review, PLoS One, 2019.

12.

Fernández-Sotos

Navarro

Torio

Dompablo

Fernández-Caballero

Rodriguez-Jimenez

. Pharmacological interventions in social cognition deficits: A systematic mapping review. Psychiatry Research. 2018; 270: 57-67.

13.

Barkhof

de Sonneville

Meijer

de Haan

. Specificity of facial emotion recognition impairments in patients with multi-episode schizophrenia. Schizophrenia Research: Cognition. 2015; 2(1): 12-19.

14.

Fernández-Sotos

Fernández-Caballero

González

Aparicio

Martínez-Gras

Torio

, et al. Digital technology for Internet access by patients with early stage schizophrenia in Spain: A multicenter research study. Journal of Medical Internet Research. 2019.

15.

Sokolova

Fernández-Caballero

. A review on the role of color and light in affective computing. Applied Sciences. 2015; 5(3): 275-293.

16.

Castillo

Fernández-Caballero

Castro-González

Salichs

López

. A framework for recognizing and regulating emotions in the elderly. In: Ambient Assisted Living and Daily Activities, 2014, pp. 320-327.

17.

Fernández-Caballero

Latorre

Pastor

Fernández-Sotos

. Improvement of the elderly quality of life and care through smart emotion regulation. In: Ambient Assisted Living and Daily Activities, 2014, pp. 348-355.

18.

Lozano-Monasor

López

Fernández-Caballero

Vigo-Bustos

. Facial expression recognition from webcam based on active shape models and support vector machines. In: Ambient Assisted Living and Daily Activities, 2014, pp. 147-154.

19.

Amini

Lisetti

Ruiz

. HapFACS 30: FACS-based facial expression generator for 3D; speaking virtual characters. IEEE Transactions on Affective Computing. 2015; 6(4): 348-360.

20.

Gutiérrez-Maldonado

Rus-Calafell

González-Conde

. Creation of a new set of dynamic virtual reality faces for the assessment and training of facial emotion recognition ability. Virtual Reality. 2014; 18(1): 61-71.

21.

García

Navarro

Fernández-Caballero

González

. Towards the design of avatar-based therapies for enhancing facial affect recognition. In: Ambient Intelligence – Software and Applications, 9th International Symposium on Ambient Intelligence; 2019. pp. 306-313.

22.

García-Sánchez

Teruel

Navarro

González

Fernández-Caballero

. A distributed tool to perform dynamic therapies for social cognitive deficit through avatars. In: Ubiquitous Computing and Ambient Intelligence, 2017, pp. 731-741.

23.

Fernández-Caballero

Navarro

Fernández-Sotos

González

Ricarte

Latorre

, et al. Human-avatar symbiosis for the treatment of auditory verbal hallucinations in schizophrenia through virtual/augmented reality and brain-computer interfaces. Frontiers in Neuroinformatics. 2017, 11: 64.

24.

Fernández-Caballero

Fernández-Sotos

Navarro

González

Ricarte

Ros

, et al. Human-avatar symbiosis in cognitive cybertherapies: Proof of concept for auditory verbal hallucinations. In: Ubiquitous Computing and Ambient Intelligence, 2017, pp. 742-753.

25.

Parke

. Computer generated animation of faces. Proceedings of the ACM Annual Conference. vol. 1, ACM; 1972, pp. 451-457.

26.

Tinwell

Grimshaw

Williams

. Uncanny behaviour in survival horror games. Journal of Gaming & Virtual Worlds. 2010; 2(1): 3-25.

27.

Guadagno

Swinth

Blascovich

. Social evaluations of embodied agents and avatars. Computers in Human Behavior. 2011; 27(6): 2380-2385.

28.

Slater

Steed

. Meeting people virtually: Experiments in shared virtual environments. In: Schroeder

, editor. The Social Life of Avatars: Presence and Interaction in Shared Virtual Environments. Springer, 2002, pp. 146-171.

29.

Ekman

Friesen

Hager

. Facial Action Coding System: The Manual on CD ROM. A Human Face. 2002.

30.

Tessier

Gingras

Robitaille

Jackson

. Toward dynamic pain expressions in avatars: Perceived realism and pain level of different action unit orders. Computers in Human Behavior. 2019.

31.

Ekman

Friesen

. Constants across cultures in the face and emotion. Journal of Personality and Social Psychology. 1971; 17(2): 124.

32.

Ekman

Friesen

Hager

. Facial Action Coding System Investigator’s Guide. Research Nexus. 2002.

33.

Sayette

Cohn

Wertz

Perrott

Parrott

. A psychometric evaluation of the facial action coding system for assessing spontaneous expression. Journal of Nonverbal Behavior. 2001; 25(3): 167-185.

34.

Ekman

Friesen

. Unmasking the face: A guide to recognizing emotions from facial clues. ISHK, 2003.

35.

Adobe. Fuse; 2019. Available from: https://www.adobe.com/es/products/fuse.html.

36.

Mixamo Adobe. Mixamo, 2019. Available from: https://www.mixamo.com.

37.

Autodesk. 3D Studio Max; 2019. Available from: https//www.autodesk.es/products/3ds-max/overview.

38.

Unity3d. Unity 3D; 2019. Available from: https//unity3d.com.

39.

Pellegrinelli

Pedrocchi

. Estimation of robot execution time for close proximity human-robot collaboration. Integrated Computer-Aided Engineering. 2017; 25: 1-16.

40.

NVIDIA. Texture Tools for Adobe Photoshop; 2019. Available from: https://developer.nvidia.com/nvidia-texture-tools-adobe-photoshop.

41.

Vicente-Querol

MÁ

García

Fernández-Sotos

Rodriguez-Jimenez

Fernández-Caballero

. Development and validation of basic virtual human facial emotion expressions. In: Ferrández Vicente

Álvarez-Sánchez

de la Paz López

Toledo Moreo

Adeli

. (eds). Understanding the Brain Function and Emotions, Proceedings of the 8th International Work-Conference on the Interplay Between Natural and Artificial Computation, Part I. Springer; 2019, pp. 222-231.

42.

García

Fernández-Sotos

Fernández-Caballero

Navarro

Latorre

Rodriguez-Jimenez

, et al. Acceptance and use of a multi-modal avatar-based tool for remediation of social cognition deficits. Journal of Ambient Intelligence and Humanized Computing. 2019.

43.

Watson

Clark

Tellegen

. Development and validation of brief measures of positive and negative affect: The PANAS scales. Journal of Personality and Social Psychology. 1988; 54(6): 1063-1070.

44.

Ekman

. Emotions revealed: recognizing faces and feelings to improve communication and emotional life. Times Books. 2003.

45.

Fabri

Moore

Hobbs

. Mediating the expression of emotion in educational collaborative virtual environments: an experimental study. Virtual Reality. 2004; 7(2): 66-81.

46.

Krumhuber

Tamarit

Roesch

Scherer

. FACSGen 20. animation software: Generating three-dimensional FACS-valid facial expressions for emotion research. Emotion. 2012; 12(2): 351.

47.

Dyck

Winbeck

Leiberg

Chen

Gur

Mathiak

. Recognition profile of emotions in natural and virtual faces. Plos One. 2008; 3(11): 1-8.

48.

Spencer-Smith

Wild

Innes-Ker

ÂH

Townsend

Duffy

Edwards

, et al. Making faces: Creating three-dimensional parameterized models of facial expression. Behavior Research Methods, Instruments, & Computers. 2001; 33(2): 115-123.

49.

Ekman

Friesen

. Pictures of Facial Affect. Consulting Psychologists Press, 1976.

50.

Kohler

Turner

Bilker

Brensinger

Siegel

Kanes

, et al. Facial Emotion Recognition in Schizophrenia: Intensity Effects and Error Pattern. American Journal of Psychiatry. 2003; 160(10): 1768-1774.

51.

Dodich

Cerami

Canessa

Crespi

Marcone

Arpone

, et al. Emotion recognition from facial expressions: a normative study of the Ekman 60-Faces Test in the Italian population. Neurological Sciences. 2014; 35(7): 1015-1021.

52.

Joyal

Jacob

Cigna

Guay

Renaud

. Virtual faces expressing emotions: An initial concomitant and construct validity study. Frontiers in Human Neuroscience. 2014; 8: 787.

53.

Wingenbach

Ashwin

Brosnan

. Validation of the Amsterdam Dynamic Facial Expression Set–Bath Intensity Variations (ADFES-BIV): A set of videos expressing low, intermediate, and high intensity emotions. PloS One. 2016; 11(1): e0147112.

54.

Calder

Keane

Manly

Sprengelmeyer

Scott

Nimmo-Smith

, et al. Facial expression recognition across the adult life span. Neuropsychologia. 2003; 41(2): 195-202.

55.

Mill

Allik

Realo

Valk

. Age-related differences in emotion recognition ability: A cross-sectional study. Emotion. 2009; 9: 619-630.

56.

Wong

Cronin-Golomb

Neargarder

. Patterns of visual scanning as predictors of emotion identification in normal aging. Neuropsychology. 2005; 19: 739-749.

57.

Chen

. A parallel and robust object tracking approach synthesizing adaptive Bayesian learning and improved incremental subspace learning. Frontiers of Computer Science. 2019; 13(5): 1116-1135.

58.

Pan

. A novel segmentation model for medical images with intensity inhomogeneity based on adaptive perturbation. Multimedia Tools and Applications. 2019; 78(9): 11779-11798.

59.

Gómez-Silva

Izquierdo

Escalera Adl Armingol

. Transferring learning from multi-person tracking to person re-identification. Integrated Computer-Aided Engineering. 2019; 26(4): 329-344.

Design of reliable virtual human facial expressions and validation by healthy people

Abstract

Keywords

1. Introduction

Table 1 Action Units used to describe the six basic emotions (based on [32, 29, 34])

2.1 Facial action coding system

2.2 Facial expressions in FACS

3. Validation of facial expressions by healthy people

3.2 Experimental procedure

Table 2 Emotion recognition rates for each of the 13 animations. Average successful recognition rate: 88.25%

4.1 Accuracy in facial expression recognition

Table 3 Emotion recognition rate with (left) and without (right) VH movement. Average successful recognition rates are 90.12% and 85.03%, respectively

Table 4 Emotion recognition rates for frontal (left) and lateral (right) views of the VHs. Average successful recognition rates are 9.06% and 87.45%, respectively

4.5 Influence of the number of faces presented on the number of errors made

5. Conclusions

Footnotes

Acknowledgments

References

Table 1
Action Units used to describe the six basic emotions (based on [32, 29, 34])

Table 2
Emotion recognition rates for each of the 13 animations. Average successful recognition rate: 88.25%

Table 3
Emotion recognition rate with (left) and without (right) VH movement. Average successful recognition rates are 90.12% and 85.03%, respectively

Table 4
Emotion recognition rates for frontal (left) and lateral (right) views of the VHs. Average successful recognition rates are 9.06% and 87.45%, respectively