Facial Emotion Recognition: Virtual Reality Program for Facial Emotion Recognition—A Trial Program Targeted at Individuals With Schizophrenia

Abstract

People with severe mental illness (SMI), schizophrenia in particular, show considerable functional impairment in emotional recognition and social perception, which negatively affects interpersonal relationships and social functioning. Owing to its ecological validity, virtual reality (VR) has been observed to improve both assessment and training of emotional recognition skills of people with SMI. This article includes two studies: (a) a descriptive study on the Virtual Reality program for Facial Emotion Recognition (VR-FER) and (b) an empirical study that presents the results of the application of the VR-FER’s first module. For the second study, data were collected using two samples: a group of 12 people with schizophrenia and a reference group of 12 people who were mentally healthy. Data analysis comprised descriptive (mean, standard deviation) and inferential statistics (Mann–Whitney U test). Results showed that the first group presented a lower number of correct answers and a higher number of incorrect answers compared with the second group regarding facial emotion recognition (FER), thereby confirming the need to develop strategies to improve emotional recognition and social perception in people with schizophrenia. VR-FER is regarded a strategic training program for FER, using latest technology and following rehabilitation guidelines for SMI.

Keywords

severe mental illness virtual reality program 3D avatar facial emotion recognition schizophrenia

Introduction

Severe mental illness (SMI) is defined as “a mental, behavioral, or emotional disorder resulting in serious functional impairment, which substantially interferes with or limits one or more major life activities. The burden of mental illnesses is particularly concentrated among those who experience disability due to SMI” (National Institute of Mental Health, 2019). People with SMI (schizophrenia [1.1%], severe bipolar disorder classified [2.2%], and severe major depression [2.0%]) are estimated to comprise more than 5% of adult population (Mental Illness Policy Organization, 2019). SMI, which is characterized by chronicity and relevant functional limitations, notably affects an individual’s quality of life (QoL; World Health Organization, 2014).

Previous studies (Barkhof, de Sonneville, Meijer, & de Haan, 2015; Behere, 2015; Campos et al., 2016; Feingold et al., 2016; Fisher et al., 2017; Kohler, Walker, Martin, Healey, & Moberg, 2010; Kring & Elis, 2013; Kring, Germans Gard, & Gard, 2011; Richard, More, & Joy, 2015) demonstrated that people with one of the SMIs—schizophrenia—present difficulties in social cognition, that is, emotional recognition and social perception. Social life and everyday communication largely depend on the ability of an individual to infer emotional states from small signals emitted by interlocutors, and recognition of facial expression of emotions is an instrumental component of nonverbal communication despite being undervalued in some research that relates social competence to emotional perception (Kohler et al., 2010; Penn, Roberts, Combs, & Stern, 2007; Vauth, Rusch, Wirtz, & Corrigan, 2004). Emotional recognition is also a predictor of professional functioning and of the ability to live independently (Kee, Qreen, Mintz, & Brekke, 2003). Among individuals diagnosed with schizophrenia, social functioning and skills are compromised, thereby affecting their communication and interpersonal relationships and contributing to their social exclusion (Bellack, Mueser, Gingerich, & Agresta, 2004). Psychoeducational intervention and individualized training of emotional recognition have proven to be promoting psychological, social, and/or emotional empowerment of such an individual (Strohmer & Chan, 2015), considering the possibility of approaching the processes and/or strategies involved in emotion recognition by addressing aspects related to facial expression and respective emotions. Well-established serious games have been commercialized for improving psychoeducation or specific behavioral changes in individuals with different medical illnesses (Danilina, Cox, Fonseca, & Johnson, 2017). However, such kind of intervention should preserve the playful aspect of the processes involved, respecting the required therapeutic component and simultaneously reinforcing individual’s motivation to continue the task. This belief is supported by Ravyse, Seugnet Blignaut, Leendertz, and Woolner (2017), stating that participants want to have fun before they value the subsequent learning benefit that serious games can offer them. Isleyen et al. (2014) designed and developed an interactive web-based educational program and conducted a study about its effectiveness on the ability of individuals with SMI in perceiving emotions. They proposed that a combination of heuristic evaluation and think-aloud method has an effective result in the evaluation of serious games designed for specific clinical groups.

Since the 1970s, the quality of clinical intervention was almost exclusively related to psychopharmacology; however, nowadays, rehabilitative approaches intended to enable people with functional limitations to achieve the highest level of personal, familiar, professional, and social functioning in the least restrictive environment possible (Ministry of Health, 2004; Phillips, Deiches, Morrison, & Kaseroff, 2016). Future interventions should be directed toward social cognition aspects such as empathy, which are undervalued in research, as the ultimate goal of sociocognitive intervention lies in enabling people with SMI to achieve a longer and more lasting improvement with regard to functionality. In other words, interventions that go “beyond the traditional perspective of the management of symptoms, promote a social functioning more effective and satisfactory in daily life” are required (Horan, Kern, Green, & Penn, 2008, p. 245).

Intervention by rehabilitation counselors (RCs) plays a key role in improving the QoL of individuals with schizophrenia and promoting their social inclusion as well as participation. RCs support such individuals in overcoming challenges such as adjusting to their disability (physically and emotionally) and responding to “the demands of today’s diverse practice environments effectively; thus, RCs must be well prepared in both traditional and emerging knowledge areas” (Chan et al., 2003, p. 83). Developments in rehabilitation include exercising in a gaming or virtual reality (VR) environment (Giggins, Persson, & Caulfield, 2013), which conveys the important role of VR in rehabilitation. According to Rizzo and Kim (2005), VR strengths in respect to rehabilitation are enhanced ecological validity; stimulus control and consistency; real-time performance feedback; cuing stimuli to support “error-free learning”; self-guided exploration and independent practice; back-end data extraction, management, analysis, and visualization; interface modification contingent on user’s impairments; complete naturalistic performance record; safe testing and training environment; gaming factors to enhance motivation; and low-cost environments that can be duplicated and distributed.

Intervention programs that include facial emotion recognition (FER) oriented to ameliorating social interaction are among the important contributions to reducing social exclusion. According to Tcherkassof, Bollon, Dubois, Pansu, and Adam (2007), recognition of facial expressions has been one of the most discussed topics with regard to emotions. Face is a fundamental element to express and/or recognize emotions, encouraging psychology researchers to discuss what kind of facial emotion stimuli is useful, artificial versus spontaneous, or static versus dynamic facial expressions (Coan & Allen, 2007). Varying results of studies that try to teach how to recognize emotions exist (Golan et al., 2007); however, none of these studies used real-time face synthesis, preferring—as a tool—photographs of facial expressions, such as those developed by Ekman and Friesen (1978). According to Freeman et al. (2017), there are significant gaps when it comes to meaningful applications regarding mental health. However, VR appears as an important resource to study emotions (Kim et al., 2007), using virtual realistic faces or avatars (Gutierrez-Maldonado, Rus-Calafell, & González-Conde, 2014; Raffard et al., 2016; Wrzesien et al., 2015). Moreover, during the last 20 years, mental health has faced new challenges (Tortella-Feliu et al., 2016), such as the increased use of technology and eHealth movement (Barello et al., 2016; Riva, 2015).

Effectively, VR has gained importance, having the potential to simulate real world as well as social and communicational interactions by using avatars (Bekele et al., 2014; McGloin, Nowak, & Watt, 2014). VR is beneficial in that individuals realize that computer environment is not real, although they behave as if it were (Freeman et al., 2017). Borrego, Latorre, Llorens, Alcañiz, and Noé (2016) have proved the utility of VR for rehabilitation and the importance of ecological settings (Cipresso & Riva, 2016). Thus, application of VR to SMI has become promising, with researchers trying to understand the mental process or develop training programs for people with SMI (Didehbani, Allen, Kandalaft, Krawczyk, & Chapman, 2016; Gayer-Anderson, 2016; Uvais, 2015; Valmaggia, Day, & Rus-Calafell, 2016), including emotion recognition (Rice, Wall, Fogel, & Shic, 2015).

Considering the importance of both emotional recognition and application of VR technology to SMI, in this research, a certain intervention program to evaluate emotion recognition has been elaborated, targeting individuals diagnosed with schizophrenia. The program uses validated stimuli that can be applied to diverse technology-based media and presents stimuli, in experimental conditions, which are as close to reality as possible and which occur in real time to ensure ecological validity. Next, it presents (a) a description of the conception and function of the Virtual Reality program for Facial Emotion Recognition (VR-FER) and (b) preliminary results obtained from the application of the first module of the VR-FER.

Study 1: VR-FER

This study aimed at discussing VR-FER, a modular program using several therapeutic options provided by VR tools that evaluate and train emotion recognition in individuals with SMI.

The program was encapsulated in a format called “serious games.” Serious games are video games with a primary purpose to promote well-being and to train and educate rather than provide pure entertainment. VR-FER was developed on the basis of two concerns: the need for an individualized or personalized intervention (after assessing the participant’s functionality level and symptoms of SMI) and a modular structure promoting generalization and transference of acquisitions to an individual’s daily life context (Freeman et al., 2017), respecting his or her evolution rate in the task. Regarding methodology, an option was made available for a combined strategy ranging from a psychoeducational component to interaction, considering observation/imitation as an instrumental complement.

Development of VR-FER included two phases—assessment and training—of emotion recognition, supporting a personalized acquisition of knowledge about emotions and their recognition in addition to a programmed training for individualized emotion recognition, which aimed at promoting a decrease in the functional limitations of the participants and its negative effects. Therefore, the program’s structure is based on VR to present emotional stimuli. The means of assessing emotion recognition comprised three-dimensional (3D) stimuli in a virtual environment as a novel methodology to study emotion recognition, ensuring ecological validity by maximum likelihood with real life. The assessment stage (where the presentation of emotional facial stimulus occurred) resorted to the use of an avatar only showing the head inserted in a virtual environment. It involved a recognition task of neutral facial emotion stimuli and six basic emotions (anger, happiness, disgust, sadness, surprise, and fear) proposed by Ekman and Friesen (1978). It was intended to provide both a quantitative (number of correct/incorrect emotions recognized) and a qualitative analysis (different error patterns corresponding to the wrongly identified emotions; see Figure 5) of the functional limitations in emotion recognition of each participant (Kohler et al., 2010). According to an individualized plan for training emotion recognition, the therapist will rely on the available tools: multiple avatars (varying in gender, age, and degree of familiarity of the user), virtual environments (neutral, familiar, and representatives of public spaces), and “parameterize” a set of “exercises” for the task of emotion recognition, allowing to create differentiation. Implementation can be done either within the therapeutic context or, following a therapeutic decision, in the context of the individual’s real life (using ordinary portable equipment) and eventually involving family members or significant others. This training aims to trigger generalization (Phillips et al., 2016) and to promote transference of emotion recognition competence to daily life interactions.

Technology Involved in the VR-FER Program

To create the virtual experience, a set of software tools and hardware equipment was used. To construct the “main avatar,” a panel of experts from the Laboratory of Psychosocial Rehabilitation of the University of Porto (to which some of the authors are affiliated) selected a photograph of a young male adult from a validated picture database (Langner et al., 2010). The photo was subsequently transformed into a 3D model using a widely used software (FaceGen Modeller^® v3.5 software program) in the design of realistic faces for various mainstream video games (Roesch et al., 2011), as well as in both commercial and academic research (Sloan, Cook, & Robinson, 2009). The basic emotions were obtained through a “morph” technique that allows working out the replication of basic emotions or expressive peculiarities (e.g., blink) in a range of values from a neutral stimulus to a 100% of the stimulus. After obtaining all the required data, they were exported to a format that allows a 3D model to be worked on a wide range of 3D modeling programs (Figure 1).

Figure 1.

Summary of some of the steps in the construction of the three-dimensional avatar.

After exporting, the avatar model was animated, respecting the settings of the relevant facial expressions; then, a virtual environment was developed to receive the avatar. A chosen environment was created to be perceived as neutral and familiar both for individuals diagnosed with SMI and for those without any psychiatric disorders. It consisted of a large living room with static and dynamic objects such as furniture, decor, plants, and toys scattered on the floor. Furthermore, another character was filmed in a green set (later replaced through the chroma-key technique) and was subsequently integrated into the virtual environment in a full-body avatar. This add-on had two purposes: to serve as a guide in the application and to facilitate the participant’s familiarization with the 3D environment (Figure 2). The last step involved exporting the data set and animating the video in a format for high definition (full HD 1080p), using the side-by-side (SbS) stereo method, which improves immersion and allows the participant to “feel” being in the environment.

Figure 2.

Virtual environment and full-body avatar.

Next steps of the VR-FER are being developed, which rely on more complex 3D technology and VR hardware to deploy an even more realistic experience. The software will be easy to use and will run on different hardware platforms such as phablets and tablet computers to enable the participant to perform tasks at home, as requested by the therapist.

Modular Structure of VR-FER

The practical strategy proposed for this intervention program included two specific and complementary dimensions that consider the rehabilitation process of individuals diagnosed with SMI: assessment of the ability to recognize emotions based on facial expression because emotions influence social interaction (Reeve, 2009) and individualized intervention program designed as per the initial assessment of the participant’s level of emotion recognition, integrating quantitative and qualitative information. The intervention program is based on a modular design considering differential aspects of emotions expressed by individuals (unknown, known, himself) and different contexts (strange, neutral, familiar). Figure 3 presents a synthesis of the VR-FER program, briefly describing each module.

Figure 3.

Scheme of the modular structure of the program VR-FER.

Regarding Module 1, “Assessment of facial emotion recognition of individuals diagnosed with SMI,” the evaluation was done using the presentation of facial expressions on a VR technical support. First, it considered the presentation of the six basic emotions (anger, fear, sorrow, disgust, joy, and surprise) proposed by Ekman and Friesen (1978). Next, these 3D facial stimuli, along with a neutral stimulus, were presented in the face of an avatar specifically created for this purpose; the participant could see only its head. At this stage, the option of including a neutral stimulus contemplates recognition errors occurring in such stimuli, as reported in previous studies (Bediou et al., 2005; Couture, Penn, & Roberts, 2006).

The task of emotion recognition was processed in the laboratory in the following sequence: (a) reception–general information about the task (the participant watched a 3D movie on the screen, having to use polarized glasses and was monitored by electroencephalography [EEG] throughout the viewing) and informed consent authorization; (b) placement of glasses and EEG electrodes; (c) task prelude (the participant watched and listened to rain falling in the sea, for a few seconds, accompanied by music). The virtual environment was shown using 360° panoramic images by different parallaxes (when objects in a 3D film appear to exist between you and the screen, they are in negative parallax; the inverse of this effect is positive parallax that happens when objects in a 3D appear to exist behind the screen). Last is the (d) emotional recognition task—in the first scene, a full-body avatar appears, presenting the task and providing instructions; stimuli were presented by another avatar. The display of the stimulus lasted 7 s between neutral stimulus, elaboration of the stimulus, and return to neutral stimulus. Furthermore, a list with emotions was shown for 10 s, following which the avatar remained in a neutral stimulus. The total time of the session for collecting data per individual was approximately 25 min.

In the second phase of the evaluation, a new presentation of seven 3D facial stimuli occurred, with an avatar inserted in the virtual environment. This module consisted of two sessions that were conducted 1 week apart. Thus, assessment becomes possible for obtaining and comparing indicators of emotion recognition stimuli, considering the number of hits at different intensity levels and the context that affects the recognition (Chung & Barch, 2011).

Module 2, “Design of an individualized program,” was based on previously obtained data, quantitative as well as qualitative, to estimate the limitations in recognizing facial expressions. This information was obtained from a database or file resulting from Module 1 and represented a comprehensive procedure of assessing emotion recognition competence, allowing more rigorous analysis of the overall level of emotional recognition owing to its inclusion of both quantitative and qualitative parameters. Modules 3 and 4, “Implementation” and “Reassessment,” respectively, were jointly presented because they are interdependent and interfere in two related processes—generalization and transference. Table 1 shows the format of the sessions of only Modules 3 and 4, considering their structure, content, and methods/strategies. The predefined 16 sessions, with an estimated duration of 45 to 60 min, were scheduled once a week. The sessions were allowed to have an individual or small group format (with two to three participants), depending on the option chosen by the therapeutic team. The review sessions were to be conducted as required by the individual and the therapist (Figure 4).

Table 1.

Sessions per Unit and Their Contents.

Sessions per unit	Contents
Sessions 1 to 4	Learning the characteristics of basic emotions and neutral components and their mimics
Sessions 5 to 8	Training recognition and differentiation of emotions (preparation of avatars and virtual environments)
Sessions 9 to 12	Understanding others’ facial expressions (known, unknown)
Sessions 13 to 16	Understanding one’s facial expressions and interpretation of the context

Figure 4.

VR-FER sessions composite.

Study 2: Empirical Results of Emotion Recognition Obtained From the Application of VR-FER Module 1

This study aimed at presenting the preliminary results of emotion recognition assessment (first-stage VR-FER).

Method

Participants

The sample included 24 participants, divided into two groups: (a) 12 individuals with a diagnosis of schizophrenia (Diagnostic and Statistical Manual of Mental Disorders (5th ed.; DSM-5; American Psychiatric Association, 2013) from a Portuguese private institution of mental health (group of individuals with schizophrenia—A) and (b) 12 individuals with no psychiatric disorder from the general population (reference group—B) (Table 2). Group A comprised a convenience sample, and the criteria for participant selection included (a) being stable from the psychopathological point of view for at least 1 year (no crises or other major psychotic symptoms), (b) having an effective control of compliance with prescribed psychopharmacological intervention by a psychiatrist (supervised medication), and (c) having no cognitive impairment screened by the Mini-Mental State Examination (MMSE; Folstein, Folstein, & McHugh, 1975; Portuguese version Guerreiro et al., 1994). The results of the MMSE application were obtained by consulting the concerned individual’s medical file. Group B comprised 12 volunteers with no psychiatric disorder, sorted through an intentional sampling procedure, and to match the independent variables of gender and age of the individuals in Group A, healthy subjects were recruited.

Table 2.

Participants’ Sociodemographic Data.

Variables	Total (N = 24)	Group of individuals with schizophrenia (N = 12)	Reference group (N = 12)
Gender, n (%)
Female	6 (25)	3 (25)	3 (25)
Male	18 (75)	9 (75)	9 (75)
Age, M (SD); [Minimum–Maximum]; Mode	36.25 (5.75); [28–47]; 32	36.25 (5.75); [28–47]; 32	36.25 (5.75); [28–47]; 32
Marital status, n (%)
Single	16 (66.7)	12 (100)	4 (33)
Married	5 (20.8)	0 (0)	5 (42)
Divorced	3 (12.5)	0 (0)	3 (25)
Education level, n (%)
4–6 years	4 (16.6)	3 (25)	1 (8)
9–12 years	13 (54.2)	7 (58)	6 (50)
University or technology course	7 (29.2)	2 (17)	5 (42)
Professional situation, n (%)
Unemployed	13 (54.2)	9 (75)	4 (33.3)
Retired	3 (12.5)	3 (25)	0 (0)
Employed	8 (33.3)	0 (0)	8 (66.7)

Procedures and Instruments

For both groups, a sociodemographic questionnaire (gender, age, marital status, education, and present professional situation) was used, and for Group A, additional data were collected (condition, hospitalization, medication, and psychosocial functioning) in their own institution. An index of emotional recognition was obtained during the first stage of the VR-FER program (assessment phase, considering the number of correct answers and the number and type of data errors). All the participants gave their informed consent to this study and followed the ethical principles—privacy and confidentiality—recommended by American Psychological Association (APA, 2016).

Results

The following results exclusively relate to the first module (assessment of emotion recognition) of the program described above (see Figure 3).

The Emotion Recognition Task

A comparative statistical analysis of the responses (hits, errors, and no answers), for both the groups, was conducted using the Mann–Whitney U test, a nonparametric test to compare medians for independent samples. Significant differences were found between the groups with regard to the number of correct responses for the six facial emotion stimuli (see Table 3).

Table 3.

Mann–Whitney U Test Comparing Median of Hits and Errors of Facial Emotion Recognition Between Groups.

Variable	Group of individuals with schizophrenia (A)	Reference group (B)	Mann–Whitney U	Significance
Variable	Mdn	Mdn	Mann–Whitney U	Significance
Hits	3.00	3.50	67.50	.799
Errors	2.50	2.00	68.00	.843
No answers	1.06	1.27	55.00	.347

Although Group A presented a lower number of correct answers and a higher number of incorrect answers, the difference between the two groups was not significant at p < .5. In other words, on analyzing each emotion, no statistical significance was found between the groups (Table 4).

Table 4.

Mann–Whitney U Test Comparing Means of Responses for Each Emotion.

Emotion	Group of individuals with schizophrenia (A)		Reference group (B)		Mann–Whitney U	Significance
Emotion	M	SD	M	SD	Mann–Whitney U	Significance
Anger	1.42	0.79	1.25	0.45	49.50	.088
Happiness	1.17	0.58	1.00	0.00	60.00	.296
Disgust	1.92	0.79	1.50	0.52	56.00	.324
Surprise	1.50	0.52	1.50	0.52	66.00	.699
Sadness	1.58	0.67	1.67	0.65	71.00	.951
Fear	1.83	0.39	1.92	0.52	65.00	.568

For Group A, of the six facial emotions presented, happiness and anger were recognized more accurately than the others, and inversely, fear and disgust were recognized less accurately. Even for Group B, happiness and anger were recognized more accurately, but the emotions recognized less accurately were sadness and fear (Figure 5).

Figure 5.

Number of responses (hits and errors) for group of individuals with schizophrenia (A) and reference group (B) for each emotion.

Discussion

The two studies in this research intended to describe the significance of VR-FER and to present the results of the emotion recognition assessment module. Regarding the structure of the VR-FER (Study 1), we found that its modular structure as suggested by Naslund, Aschbrenner, and Bartels (2016) when they stated that different features of technologies could support the design of tailored interventions as a complement and an extension to reach and improve the quality of services delivered through community mental health settings. Based on the literature and the description of the remaining modules of the program, which have not yet been tested (Module 2: Design of an individual program, Module 3: Implementation, and Module 4: Reassessment), we inferred that the design of a customized program using immersive VR competency training systems will allow the development of skills required in emotion recognition. Moreover, the use of such immersive systems will facilitate transference and generalization of the learned skills in other contexts (in everyday life and in social, recreational, and professional contexts)

As Study 1 was a descriptive one, the results and discussion refer only to Study 2. Two samples (12 individuals diagnosed with schizophrenia and 12 individuals without any psychiatric disorders) performed emotion recognition tasks. Results showed differences between the groups with regard to the number of correct responses for the six facial emotional stimuli: The group of individuals with schizophrenia presents a lower number of correct answers and a higher number of incorrect answers. Besides, facial emotions of happiness and anger were recognized more accurately by both groups, whereas major difficulties were related to fear and disgust recognition (group of individuals with schizophrenia) and sadness and fear (reference group). Our results led us to conclude that Module 1 of the VR-FER (assessment) allows evaluation of facial emotion recognition skills.

Our results are consistent with those of other studies involving individuals diagnosed with schizophrenia, where facial recognition of negative emotions was comparatively less accomplished than that of the positive ones, which might explain the performance of Group A with regard to facial stimuli of fear and disgust (Bediou et al., 2005; Couture et al., 2006; Kim et al., 2007; Kohler et al., 2010; Marques-Teixeira et al., 2005; Tsoi et al., 2008; Van’t Wout et al., 2007). Thus, there is a less achieved performance in the recognition of negative facial emotions by individuals with schizophrenia; however, the recognition of positive emotions appears to be less compromised (Johnston, Devir, & Karayanidis, 2006; Johnston, McCabe, & Schall, 2003). According to Tsoi et al. (2008), individuals with a diagnosis of schizophrenia can adopt a stricter criterion for recognizing facial emotions of sadness or fear.

Thus, the use of VR will be beneficial to RCs from a clinical point of view, owing to its flexibility that can be used to train emotional expression. Another advantage is that virtual faces can be manipulated by the therapist, allowing training emotion recognition, thereby minimizing the impact of the condition on social interaction and integration (Nicolaou, Gunes, & Pantic, 2011). This will also enable reinforcing the idea conveyed in previous research, acknowledging that people with SMI can acquire skills despite their symptoms and that several technologies have been developed in the field of mental health to facilitate skill training (Anthony & Mizock, 2014). Virtual environments can also improve emotionally relevant experiences and social interactions (Parsons et al., 2017). Accordingly, for mental health intervention and rehabilitation programs, social skill training comprises the interventions required in primary skills training. Thus, the effectiveness of applying VR is a contribution to facilitate better communication in a community and improve daily life activities (Anthony & Mizock, 2014). This approach also presents other opportunities such as rehabilitation through VR with a widespread intuitive appeal to the public, academic and professional acceptance, close-knit VR rehabilitation in the scientific and clinical community, and integration of VR with physiological monitoring, brain imaging, and telerehabilitation (Rizzo & Kim, 2005).

Currently, the impact of virtual environments is relevant as an initial approach for an individual’s actual exposure in the real-life context. It is an opportunity to train individuals in a secure environment, with a corrective factor performed according to the options chosen by the therapist: The intensity of emotions and the type of environment and avatars can be manipulated (Pallesen, Andersen, Hansen, Lundquist, & Brunner, 2018). The combination of these three variables will enable the therapist to customize a wide range of difficulty levels involved in rehabilitation. Regarding the program itself, future studies should test the impact of a full-body avatar in a virtual environment and in a meaningful social context. According to Rus-Calafell, Garety, Sason, Craig, and Valmaggia (2018), “Future research includes demonstrating generalizability to real life settings, examining potential negative effects, larger sample sizes and long-term follow-up studies” (p. 3). In addition, Tieri, Morone, Paolucci, and Iosa (2018) have stated the need for further research in cognitive and motor rehabilitation to better understand the role of Present and Virtual Embodiment.

Limitations

This study has the following limitations: both the groups were classified by an intentional sampling method and cannot be considered representative either of the clinical population (Group A) or of the general population (Group B), and both samples were small (N = 12), which implies that the findings are not generalizable but limited to the sample study. It was also difficult to find this clinical sample, considering the restricted inclusion criteria: not having a cognitive deficit, having a diagnosis of schizophrenia (DSM-5), and being sufficiently stable (medically) to participate in this study. Finally, it was only possible to implement the first phase of the program, and there are no other comparative studies.

Implications for Rehabilitation Research and Intervention Strategies

To the best of our knowledge, VR-FER is one of the first attempts to use VR as a methodological resource to evaluate emotion recognition, and this approach has potential for a future effective tool, as it includes attributes that may support other situations and research contexts, such as learning and rehabilitation. As the systematic review of Valmaggia et al. (2016) reveals, VR can be used to investigate psychological processes associated with SMI and its implications for experimental research, assessment, and clinical practice; however, these subjects need to be further discussed.

The use of VR-FER is conceived only for experts, such as RCs, which comprise one of the professional categories with training in mental health that can benefit from this tool, after certain training. Because the program is modular and personalized, currently, the training focuses on adjusting the program’s functionalities as per the participants’ needs (e.g., with the objective of finding a job, it is possible to simulate interviews with the employer). Thus, VR-FER is exclusively intended for certified therapists and requires purchasing equipment by either a single therapist or a rehabilitation institution. According to the therapeutic/rehabilitation objectives, personal use could be made available, thereby promoting generalization to daily routine and increasing autonomy.

RCs interested in eMental health could benefit from understanding the possibilities offered by the said technology and incorporate these in their research and development practice (Schueller, 2018). Rus-Calafell et al. (2018) have suggested that VR can be applied to the “delivery of cognitive rehabilitation, social skills training interventions and virtual reality-assisted therapies for psychosis” (p. 3).

Footnotes

Declaration of Conflicting Interest

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

References

American Psychiatric Association. (2013). Diagnostic and statistical manual of mental disorders (5th ed.). Arlington, VA: American Psychiatric Publishing.

American Psychological Association. (2016). Revision of ethical standard 3.04 of the “ethical principles of psychologists and code of conduct” (2002, as amended 2010). American Psychologist, 71, Article 900. doi:10.1037/amp0000102

Anthony

W. A.

Mizock

(2014). Evidence-based processes in an era of recovery: Implications for rehabilitation counseling and research. Rehabilitation Counseling Bulletin, 57, 219–227. doi:10.1177/0034355213507979

Barello

Triberti

Graffigna

Libreri

Serino

Hibbard

Riva

(2016). eHealth for patient engagement: A systematic review. Frontiers in Psychology, 6, Article 2013. doi:10.3389/fpsyg.2015.02013

Barkhof

de Sonneville

Meijer

de Haan

(2015). Specificity of facial emotion recognition impairments in patients with multi-episode schizophrenia. Schizophrenia Research: Cognition, 2, 12–19. doi:10.1016/j.scog.2015.01.001

Bediou

Krolak-Salmon

Saoud

Henaff

M. A.

Burt

Dalery

D’Amato

(2005). Facial expression and sex recognition in schizophrenia and depression. Canadian Journal of Psychiatry, 50, 525–533. doi:10.1177/070674370505000905

Behere

R. V.

(2015). Facial emotion recognition deficits: The new face of schizophrenia. Indian Journal of Psychiatry, 57, 229–235. doi:10.4103/0019-5545.166641

Bekele

Crittendon

Zheng

Swanson

Weitlauf

Warren

Sarkar

(2014). Assessing the utility of a virtual environment for enhancing facial affect recognition in adolescents with autism. Journal of Autism and Developmental Disorders, 44, 1641–1650.

Bellack

Mueser

Gingerich

Agresta

(2004). Social skills training for schizophrenia: A step by step guide (2nd ed.). New York, NY: Guilford Press.

10.

Borrego

Latorre

Llorens

Alcañiz

Noé

(2016). Feasibility of a walking virtual reality system for rehabilitation: Objective and subjective parameters. Journal of NeuroEngineering and Rehabilitation, 13, 68–77. doi:10.1186/s12984-016-0174-1

11.

Campos

Santos

Gagen

Machado

Rocha

Kurtz

Rocha

(2016). Neuroplastic changes following social cognition training in schizophrenia: A systematic review. Neuropsychology Review, 26, 310–328. doi:10.1007/s11065-016-9326-0

12.

Chan

Leahy

M. J.

Saunders

J. L.

Tarvydas

V. M.

Ferrin

J. M.

Lee

(2003). Training needs of certified rehabilitation counselors for contemporary practice. Rehabilitation Counseling Bulletin, 46, 82–91. doi:10.1177/00343552030460020201

13.

Chung

Barch

(2011). The effect of emotional context on facial emotion ratings in schizophrenia. Schizophrenia Research, 13, 235–241. doi:10.1016/j.schres.2011.05.028

14.

Cipresso

Riva

(2016). Personality assessment in ecological settings by means of virtual reality. In Kumar

(Ed.), The Wiley handbook of personality assessment (pp. 240–248). New York, NY: John Wiley.

15.

Coan

J. A.

Allen

J. B.

(2007). Handbook of emotion elicitation and assessment. Oxford, UK: Oxford University Press.

16.

Couture

Penn

Roberts

(2006). The functional significance of social cognition in schizophrenia: A review. Schizophrenia Bulletin, 23, S44–S63. doi:10.1093/schbul/sbl029

17.

Danilina

Cox

A. L.

Fonseca

Johnson

(2017). Serious video games as psychosocial interventions for psychosis. CHI 2017, Denver, CO, USA.

18.

Didehbani

Allen

Kandalaft

Krawczyk

Chapman

(2016). Virtual reality social cognition training for children with high functioning autism. Computers in Human Behavior, 62, 703–711. doi:10.1016/j.chb.2016.04.033

19.

Ekman

Friesen

(1978). The facial action coding system: A technique for the measurement of facial movement. San Francisco, CA: Consulting Psychologists Press.

20.

Feingold

Hasson-Ohayon

Laukka

Vishne

Dembinsky

Kravets

(2016). Emotion recognition deficits among persons with schizophrenia: Beyond stimulus complexity level and presentation modality. Psychiatry Research, 240, 60–65. doi:10.1016/j.psychres.2016.04.015

21.

Fisher

Nahum

Howard

Rowlands

Brandrett

Kermott

. . . Vinogradov

(2017). Supplementing intensive targeted computerized cognitive training with social cognitive exercises for people with schizophrenia: An interim report. Psychiatric Rehabilitation Journal, 40, 21–32. doi:10.1037/prj0000244

22.

Folstein

M. F.

Folstein

S. E.

McHugh

P. R.

(1975). “Mini-mental state”: A practical method for grading the cognitive state of patients for the clinician. Journal of Psychiatric Research, 12, 189–198. doi:10.1016/0022-3956(75)90026-6

23.

Freeman

Reeve

Robinson

Ehlers

Clark

Spanlang

Slater

(2017). Virtual reality in the assessment, understanding, and treatment of mental health disorders. Psychological Medicine, 47, 2393–2400. doi:10.1017/S003329171700040X

24.

Gayer-Anderson

(2016). The application of virtual reality technology to understanding psychosis. Social Psychiatry and Psychiatric Epidemiology, 51, 937–939. doi:10.1007/s00127-016-1245-0

25.

Giggins

O. M.

Persson

U. M.

Caulfield

(2013). Biofeedback in rehabilitation. Journal of NeuroEngineering and Rehabilitation, 10(1), Article 60. doi:10.1186/1743-0003-10-60

26.

Golan

Baron-Cohen

Ashwin

Day

McClintock

Leggett

(2008). Facilitating emotional understanding and face-processing in young children with autism spectrum conditions using animations of vehicles with faces. In British Psychological Society Annual Conference. University of Bath.

27.

Guerreiro

Silva

A. P.

Botelho

M. A.

Leitão

Castro-Caldas

Garcia

(1994). Adaptação à população portuguesa da tradução do Mini Mental State Examination (MMSE) [Adaptation to the Portuguese population of the translation of the Mini Mental State Examination (MMSE)]. Revista Portuguesa De Neurologia, 1(9), 9–10.

28.

Gutiérrez-Maldonado

Rus-Calafell

González-Conde

(2014). Creation of a new set of dynamic virtual reality faces for the assessment and training of facial emotion recognition ability. Virtual Reality, 18, 61–71. doi:10.1007/s10055-013-0236-7

29.

Horan

W. P.

Kern

R. S.

Green

M. F.

Penn

D. L.

(2008). Social cognition training for individuals with schizophrenia: Emerging evidence. American Journal of Psychiatric Rehabilitation, 11, 205–252. doi:10.1080/15487760801963652

30.

Isleyen

Gulkesen

K. H.

Cinemre

Samur

M. K.

Zayim

Sen Kaya

(2014). Evaluation of the usability of a serious game aiming to teach facial expressions to schizophrenic patients. Studies in Health Technology and Informatics, 205, 662–666. doi:10.3233/978-1-61499-432-9-662

31.

Johnston

Devir

Karayanidis

(2006). Facial emotion processing in schizophrenia: No evidence for a deficit specific to negative emotions in a differential deficit design. Psychiatry Research, 143, 51–61. doi:10.1016/j.psychres.2005.08.006

32.

Johnston

McCabe

Schall

(2003). Differential susceptibility to performance degradation across categories of facial emotion—A model confirmation. Biological Psychology, 63, 45–58. doi:10.1016/S0301-0511(03)00026-7

33.

Kee

Qreen

Mintz

Brekke

(2003). Is emotion processing a predictor of functional outcome in schizophrenia? Schizophrenia Bulletin, 29, 487–497. doi:10.1093/oxfordjournals.schbul.a007021

34.

Kim

J. J.

Kim

Park

D. E.

Jang

H. J.

. . . Kim

S. I.

(2007). Characteristics of social perception assessed in schizophrenia using virtual reality. Cyberpsychology & Behavior, 10, 215–219. doi:10.1089/cpb.2006.9966

35.

Kohler

Walker

Martin

Healey

Moberg

(2010). Facial emotion perception in schizophrenia: A meta-analytic review. Schizophrenia Bulletin, 36, 1009–1019. doi:10.1093/schbul/sbn192

36.

Kring

A. M.

Elis

(2013). Emotion deficits in people with schizophrenia. Annual Review of Clinical Psychology, 9, 409–433. doi:10.1146/annurev-clinpsy-050212-185538

37.

Kring

A. M.

Germans Gard

Gard

D. E.

(2011). Emotion deficits in schizophrenia: Timing matters. Journal of Abnormal Psychology, 120, 79–87. doi:10.1037/a0021402

38.

Langner

Dotsch

Bijlstra

Wigboldus

D. J.

Hawk

S. T.

van Knippenberg

(2010). Presentation and validation of the Radboud faces database. Cognition and Emotion, 24, 1377–1388. doi:10.1080/02699930903485076

39.

Marques-Teixeira

Pereira

Silva

Marieiro

Humberto

Guerreiro

(2005). Caracterização de doentes portugueses com esquizofrenia: Resultados da avaliação do estudo observacional SOHO no início do estudo [Characterization of Portuguese patients with schizophrenia: Results from the evaluation of the SOHO observational study at baseline]. Saúde Mental, 7(4), 13–21.

40.

McGloin

Nowak

K. L.

Watt

(2014). Avatars and expectations: Influencing perceptions of trustworthiness in an online consumer setting. PsychNology Journal, 1(1–2), 7–28.

41.

Mental Illness Policy Organization. (2019). What is “serious mental illness” and what is not? Retrieved from https://mentalillnesspolicy.org/serious-mental-illness-not/

42.

Ministry of Health. (2004). National health plan 2004 to 2010. Vol. I. Priorities. Lisbon, Portugal: Author.

43.

Naslund

J. A.

Aschbrenner

K. A.

Bartels

S. J.

(2016). How people with serious mental illness use smartphones, mobile apps, and social media. Psychiatric Rehabilitation Journal, 39, 364–367. doi:10.1037/prj0000207

44.

National Institute of Mental Health. (2019). Mental illness. Retrieved from https://www.nimh.nih.gov/health/statistics/mental-illness.shtml

45.

Nicolaou

Gunes

Pantic

(2011). Output-associative RVM regression for dimensional and continuous emotion prediction. Retrieved from http://ibug.doc.ic.ac.uk/people/mnicolaou

46.

Pallesen

Andersen

M. B.

Hansen

G. M.

Lundquist

C. B.

Brunner

(2018). Patients’ and health professionals’ experiences of using virtual reality technology for upper limb training after stroke: A qualitative substudy. Rehabilitation Research and Practice. Advance online publication. doi:10.1155/2018/4318678

47.

Parsons

T. D.

Riva

Parsons

S. J.

Mantovani

Newbutt

Lin

. . . Hall

(2017). Virtual reality in pediatric psychology. Pediatrics, 140, S86–S91. doi:10.1542/peds.2016-1758I

48.

Penn

Roberts

Combs

Sterne

(2007). The development of social cognition and interaction training program for schizophrenia spectrum disorders. Psychiatric Services, 58, 449–451. doi:10.1176/ps.2007.58.4.449

49.

Phillips

B. N.

Deiches

Morrison

Kaseroff

A. A.

(2016). Social effectiveness: A 30-year review of the rehabilitation counseling literature. Rehabilitation Counseling Bulletin, 60, 16–26. doi:10.1177/0034355215601073

50.

Raffard

Bortolon

Khoramshahi

Salesse

R. N.

Burca

Marin

. . . Capdevielle

(2016). Humanoid robots versus humans: How is emotional valence of facial expressions recognized by individuals with schizophrenia? An exploratory study. Schizophrenia Research, 176, 506–513. doi:10.1016/j.schres.2016.06.001

51.

Ravyse

W. S.

Seugnet Blignaut

Leendertz

Woolner

(2017). Success factors for serious games to enhance learning: A systematic review. Virtual Reality, 21, 31–58. doi:10.1007/s10055-016-0298-4

52.

Reeve

(2009). Understanding motivation and emotion. Hoboken, NJ: John Wiley.

53.

Rice

L. M.

Wall

C. A.

Fogel

Shic

(2015). Computer-assisted face processing instruction improves emotion recognition, mentalizing, and social skills in students with ASD. Journal of Autism and Developmental Disorders, 45, 2176–2186. doi:10.1007/s10803-015-2380-2

54.

Richard

D. A.

Joy

S. P.

(2015). Recognizing emotions: Testing an intervention for children with autism spectrum disorders. Art Therapy, 32, 13–19. doi:10.1080/07421656.2014.994163

55.

Riva

(2015). Embodied medicine: What human-computer confluence can offer to health care. In Gaggioli

Ferscha

Riva

Dunne

Viaud-Delmon

(Eds.), Human computer confluence: Transforming human experience through symbiotic technologies (pp. 55–79). Warsaw, Poland: De Gruyter Open.

56.

Rizzo

A. S.

Kim

G. J.

(2005). A SWOT analysis of the field of virtual reality rehabilitation and therapy. Presence: Teleoperators & Virtual Environments, 14, 119–146. doi:10.1162/1054746053967094

57.

Roesch

E. B.

Tamarit

Reveret

Grandjean

Sander

Scherer

K. R.

(2011). FACSGen: A tool to synthesize emotional facial expressions through systematic manipulation of facial action units. Journal of Nonverbal Behavior, 35, 116. doi:10.1007/s10919-010-0095-9

58.

Rus-Calafell

Garety

Sason

Craig

T. J. K.

Valmaggia

L. R.

(2018). Virtual reality in the assessment and treatment of psychosis: A systematic review of its utility, acceptability and effectiveness. Psychological Medicine, 48, 362–391. doi:10.1017/S0033291717001945

59.

Schueller

S. M.

(2018). Mental health and eHealth technology. In van Gemert-Pijnen

Kelders

S. M.

Kip

Sanderman

(Eds.), eHealth research, theory and development (pp. 111–130). New York, NY: Routledge.

60.

Sloan

Cook

Robinson

(2009). Considerations for believable emotional facial expression animation. 2nd International Conference on Visualization, Barcelona, Spain.

61.

Strohmer

D. C.

Chan

(2015). Introduction to the special issue on international rehabilitation counseling research. Rehabilitation Counseling Bulletin, 58, 67–69. doi:10.1177/0034355214561208

62.

Tcherkassof

Bollon

Dubois

Pansu

Adam

J. M.

(2007). Facial expressions of emotions: A methodological contribution to the study of spontaneous and dynamic emotional faces. European Journal of Social Psychology, 37, 1325–1345. doi:10.1002/ejsp.427

63.

Tieri

Morone

Paolucci

Iosa

(2018). Virtual reality in cognitive and motor rehabilitation: Facts, fiction and fallacies. Expert Review of Medical Devices, 15, 107–117. doi:10.1080/17434440.2018.1425613

64.

Tortella-Feliu

Baños

R. M.

Barrantes

Botella

Fernández-Aranda

García-Campayo

. . . Soler

(2016). Retos de la investigación psicológica en salud mental [Challenges of psychological research in mental health]. Clínica Y Salud, 27, 37–43. doi:10.1016/j.clysa.2016.02.001

65.

Tsoi

Lee

Khokhar

Mir

Swalli

Gee

. . . Woodruff

(2008). Is facial emotion recognition impairment in schizophrenia identical for different emotions? A signal detection analysis. Schizophrenia Research, 99, 263–269. doi:10.1016/j.schres.2007.11.006

66.

Uvais

N. A.

(2015). Virtual reality applications in Schizophrenia. Industrial Psychiatry Journal, 24(2), Article 210. doi:10.4103/0972-6748.181755

67.

Valmaggia

L. R.

Day

Rus-Calafell

(2016). Using virtual reality to investigate psychological processes and mechanisms associated with the onset and maintenance of psychosis: A systematic review. Social Psychiatry and Psychiatric Epidemiology, 51, 921–936. doi:10.1007/s00127-016-1245-0

68.

Van’t Wout

Aleman

Kessels

R. P. C.

Cahn

de Haan

Kahn

R. S

. (2007). Exploring the nature of facial affect processing deficits in schizophrenia. Psychiatry Research, 150, 227–235. doi:10.1016/j.psychres.2006.03.010

69.

Vauth

Rusch

Wirtz

Corrigan

(2004). Does social cognition influence the relation between neurocognitive deficits and vocational functioning in schizophrenia? Psychiatry Research, 128, 155–165. doi:10.1016/j.psychres.2004.05.018

70.

World Health Organization. (2014). Global reference list of 100 core health indicators. Retrieved from http://www.who.int/healthinfo/indicators/2015/en/

71.

Wrzesien

Rodriguez Ortega

Rey

Alcañiz Raya

M. L.

Banos

Vara

(2015). How the physical similarity of avatars can influence the learning of emotion regulation strategies in teenagers. Computers in Human Behavior, 43, 101–111. doi:10.1016/j.chb.2014.09.024