Recognizing facially expressed emotions in videos of people with visual impairments in online settings

Abstract

BACKGROUND

Facial expressions are critical for conveying emotions and facilitating social interaction. Yet, little is known about how accurately sighted individuals recognize emotions facially expressed by people with visual impairments in online communication settings.

OBJECTIVE

This study aimed to investigate sighted individuals’ ability to understand facial expressions of six basic emotions in people with visual impairments during Zoom calls. It also aimed to examine whether education on facial expressions specific to people with visual impairments would improve emotion recognition accuracy.

METHODS

Sighted participants viewed video clips of individuals with visual impairments displaying facial expressions. They then identified the emotions displayed. Next, they received an educational session on facial expressions specific to people with visual impairments, addressing unique characteristics and potential misinterpretations. After education, participants viewed another set of video clips and again identified the emotions displayed.

RESULTS

Before education, participants frequently misidentified emotions. After education, their accuracy in recognizing emotions improved significantly.

CONCLUSIONS

This study provides evidence that education on facial expressions of people with visual impairments can significantly enhance sighted individuals’ ability to accurately recognize emotions in online settings. This improved accuracy has the potential to foster more inclusive and effective online interactions between people with and without visual disabilities.

Keywords

Cyberspace non-verbal communication visual disabilities emotion recognition facial expression

1. Introduction

Facial expression plays a critical role in communication while conveying a rich, non-verbal information, specifically emotional states which significantly contribute to promoting social interaction [1]. The human face has a complex network of muscles on each side, working together to enable chewing, speech, and a wide range of facial expressions [2]. Despite various channels available to humans for expressing emotions, such as non-verbal vocalizations, language, prosody, and chemosensory signals, humans tend to heavily rely on facial expressions [3]. While humans have the capacity to perceive a limitless number of emotions [4], Ekman [5] suggests six basic emotions (happiness, sadness, anger, fear, surprise, and disgust) that are deemed innate and expressed through universal facial expressions. Ekman’s view has been widely embraced by numerous scholars [6, 7, 8, 9, 10].

The Facial Action Coding System (FACS) is a widely adopted method for analyzing facial muscle movements involved in expressing facial expressions [11, 12]. These movements are segmented into individual components, termed Action Units (AUs). Each AU corresponds to distinct facial actions; for instance, AU 6 denotes Cheek Raiser, AU 27 represents Mouth Stretch, AU 45 signifies Blink, and AU 64 indicates Eyes Down [13]. For example, Scherer et al. [14] observed individuals expressing various emotions and noted specific combinations of AUs: AU 5 $+$ 27 $+$ 57 for anger, AU 4 $+$ 5 $+$ 1 $+$ 4 for fear, 1 $+$ 15 $+$ 64 $+$ 4 for sadness, AU 6 $+$ 7 $+$ 12 for happiness, and AU 4 $+$ 7 $+$ 9 $+$ 10 $+$ 17 $+$ 20 for disgust. Interestingly, FACS AUs have also proven applicable in analyzing facial expressions among individuals with visual impairments. Kim [15] observed that those with visual impairments displayed AU 4 $+$ 15 $+$ 44 for anger; AU 4 for disgust, AU 1 $+$ 5 $+$ 25 for fear AU 6 $+$ 12 $+$ 25 for happiness; AU 17 $+$ 41 $+$ 54 $+$ 64 for sadness; and AU 1 $+$ 5 $+$ 25 $+$ 27 for surprise.

In addition to expressing emotions effectively, accurately perceiving and understanding the emotions of others is a crucial aspect of emotional intelligence that contributes to social interaction [16, 17, 18, 19]. Even infants possess the ability to perceive facial emotions [20], typically beginning to recognize emotional expressions after six months of age [21]. Ichikawa et al. [22] discovered that infants aged 6 to 7 months were capable of recognizing subtle facial expressions of anger. It has been well documented that an individual’s capacity to recognize emotions through facial expressions can be significantly improved through education [23, 24]. For instance, Patel et al. [25] conducted a study involving social cognition training aimed at enhancing the recognition of facial emotions. Their findings revealed an increase in the accuracy of recognizing facial expressions depicting anger and neutrality. Moreover, they observed a decrease in reaction time, measured as the duration between stimulus presentation and the subsequent mouse-click response for recognizing neutral, fearful, and sad facial expressions. Furthermore, Ekman et al. [26] proposed the potential establishment of a universal expression and perception of emotion through culture-constant learning. This learning encompasses acquiring knowledge about facial expressions that are commonly shared across various cultures. It could be facilitated through diverse means, such as watching movies and television shows, reading books, and engaging with individuals from different cultural backgrounds.

Learning about facial expressions can be challenging for individuals with visual impairments, as they have limited ability to observe how others use facial muscles to convey emotions. This difference in visual experience might lead to differences in facial expressions between individuals with and without visual impairments, depending on their visual acuity levels. Valente et al. [3] argued that prior visual experience might not significantly impact spontaneous facial expressions but could influence voluntary expressions. Spontaneous facial expressions occur involuntarily in response to an emotional state, while voluntary expressions are intentionally produced to convey specific emotions [27]. Several studies [28, 29, 30, 31] support this notion, reporting no significant differences in spontaneous facial expressions between individuals with and without visual impairments. Galati et al. [32] proposed that significant distinctions exist in voluntary facial expressions, based on varying visual acuity levels. This findings suggest that individuals with and without visual impairments may present different voluntary facial expressions. This raises the critical need for further research to investigate the extent to which sighted individuals recognize voluntary facial expressions among people with visual impairments, particularly within cyberspace contexts.

In today’s interconnected world, people extensively leverage diverse information and communication tools for remote communication. Remarkably, individuals with visual impairments have been using various assistive technologies (e.g., screen readers and voice user interfaces) [33] to leverage mainstream technologies including video conferencing applications [34, 35]. These video conferencing applications enable individuals, both with and without visual impairments, to communicate while expressing emotional states through facial expressions. Several studies have delved into understanding of facial expressions shared via cyberspace, which would be beneficial to various stakeholders. For instance, in computer-mediated psychotherapy, understanding emotional interactions between healthcare providers and consumers is pivotal for delivering high quality services [36]. Furthermore, online-collected facial expressions have shown promise in detecting Parkinson’s disease [37] and assessing online users’ personalities [38]. Real-time analysis of facial expressions has been instrumental in promoting communication between online users [39] and designing online recommender systems for e-commerce [40]. User satisfaction in virtual museums has been measured through facial expressions [41]. Moreover, facial expressions play a critical role in assessing online students’ attentiveness and engagement in virtual teaching and learning environments [42, 43, 44].

However, understanding facial expressions of individuals with visual impairments in online settings remains a relatively unexplored area that requires further exploration, particularly when compared to their sighted peers. Various systems have been developed to help people with visual impairments interpret facial expressions of sighted individuals in face-to-face interactions [45, 46]. For instance, engineers developed innovative solutions such as wearable vibrotactile devices and smartphone applications, enabling individuals with visual impairments to understand facial expressions when interacting with others in person. Similarly, systems have also been created to assist sighted individuals in interpreting facial expressions of general populations during face-to-face interactions [47]. Recent studies [15, 48] have contributed to a deeper understanding of how individuals with visual impairments express facial emotions in online interaction. However, despite these efforts, a significant gap remains in understanding the degree to which sighted individuals can accurately discern the emotions expressed through facial expressions of people with visual impairments in online communication settings.

To bridge this knowledge gap, this study investigated the accuracy of sighted individuals’ ability to identify emotions by having them watch videos featuring basic emotions expressed facially by people with visual impairments via Zoom. We further investigated whether their perception could be improved through education focusing on facial expressions specific to people with visual impairments.

2. Methods

2.1. Participants

This study recruited a convenience sample of 43 participants (mean age: 28.86 $\pm$ 8.51 years old), consisting of 17 males, and 26 females. Inclusion criteria required participants to be English-speaking, aged 18 years or older, and free from severe visual impairments/blindness (i.e., visual acuity greater than 20/70) [49]. Individuals expressing interest, seeking information, or having questions about the study directly contacted the research team. All participants provided informed consent before participating in the study. Approval for this study was granted from the Institutional Review Board (IRB).

2.2. Materials

Facial expression videos were randomly chosen from a prior study (citation omitted for anonymized peer review). The videos depicted voluntary facial expressions of six basic emotions (happiness, sadness, anger, fear, surprise, and disgust) displayed by people with visual impairments during online interaction via Zoom. Each video sequence started with a neutral expression, transitioned to a peak emotional expression, and concluded with a return to a neutral expression.

Table 1

Emotions perceived by sighted participants when viewing facial expression videos of anger displayed by people with visual impairments

Perceived emotions	Composition (%) of perceived emotions, given the angry facial expression		$\chi^{2}$ test between before and after
	Before education	After education
Anger	16.10	67.51	$\chi^{2}$ (1) $=$ 60.24, $p<$ 0.001, $v=$ 0.60
Disgust	20.00	17.26	$\chi^{2}$ (1) $=$ 0.65, $p=$ 0.42, $v=$ 0.09
Fear	16.10	2.03	$\chi^{2}$ (1) $=$ 22.73, $p<$ 0.001, $v=$ 0.78
Happiness	21.46	1.02	$\chi^{2}$ (1) $=$ 38.35, $p<$ 0.001, $v=$ 0.91
Sadness	16.10	4.06	$\chi^{2}$ (1) $=$ 15.24, $p<$ 0.001, $v=$ 0.61
Surprise	10.24	8.12	$\chi^{2}$ (1) $=$ 0.68, $p=$ 0.41, $v=$ 0.14
$\chi^{2}$ test between before and after	$\chi^{2}$ (5) $=$ 137.79, $p<$ 0.001, $v=$ 0.26

Table 2

Emotions perceived by sighted participants when viewing facial expression videos of disgust displayed by people with visual impairments

Perceived emotions	Composition (%) of perceived emotions, given the disgust facial expression		$\chi^{2}$ test between before and after
	Before education	After education
Anger	18.63	26.92	$\chi^{2}$ (1) $=$ 3.45, $p=$ 0.06, $v=$ 0.02
Disgust	22.06	44.71	$\chi^{2}$ (1) $=$ 16.70, $p<$ 0.001, $v=$ 0.35
Fear	5.39	4.81	$\chi^{2}$ (1) $=$ 0.048, $p=$ 0.83, $v=$ 0.05
Happiness	21.08	2.40	$\chi^{2}$ (1) $=$ 30.08, $p<$ 0.001, $v=$ 0.79
Sadness	24.51	14.42	$\chi^{2}$ (1) $=$ 5.00, $p=$ 0.03, $v=$ 0.25
Surprise	8.33	6.73	$\chi^{2}$ (1) $=$ 0.29, $p=$ 0.59, $v=$ 0.10
$\chi^{2}$ test between before and after	$\chi^{2}$ (5) $=$ 55.53, $p<$ 0.001, $v=$ 0.16

2.3. Procedures

Phase I (pre-education). Participants were presented with a set of 30 facial expression videos (five videos for each of the six basic emotions) via Zoom. The order of video was randomized. Participants were instructed to watch each video carefully and identify the emotion being displayed. Their responses were recorded for later analysis.

Education. Following the pre-education phase, participants received a 30-minute education delivered remotely through Zoom. The education aimed to familiarize participants with the findings of the earlier study (citation omitted for anonymized peer review), highlighting the Facial Action Coding System (FACS) and FACS-based models that account for the nuances of facial expressions displayed by people with visual impairments in online settings.

Phase II (post-education). After the education session, participants completed another set of 30 facial expression videos (different from the videos used in the pre-education phase). The videos were again presented in a randomized order. Participants completed the same emotion recognition task as in Phase I.

2.4. Data analysis

Emotion recognition performance was assessed for both phases. Chi-square testing was conducted to determine if there was a significant difference in the perceived emotions before and after education. Statistical analyses were carried out using the IBM SPSS Statistics for Macintosh, version 24 [50].

3. Results

In summary, the study revealed significant differences in participants’ ability to perceive emotions displayed by individuals with visual impairments in online settings. Prior to education (baseline), participants exhibited inaccurate interpretations for all facial expressions, except for happiness. Post-education, their accuracy significantly improved for all emotions displayed, except for fear. This suggests that participants might have encountered challenges in selectively attending to and interpreting subtle facial cues in individuals with visual impairments. Detailed results are available below.

3.1. Anger

Chi-square testing revealed a significant difference in perceived emotions before and after education when participants viewed facial expression videos depicting anger (see Table 1). Before education, there was no specific emotion that was significantly perceived as all the six basic emotions were equally likely to be perceived, $\chi^{2}$ (5) $=$ 9.39, $p=$ 0.09, $v=$ 0.10. In contrast, after education, anger was the dominant emotion perceived while the other emotions were perceived less often, $\chi^{2}$ (5) $=$ 386.32, $p<$ 0.001, $v=$ 0.20.

3.2. Disgust

Chi-square testing revealed a significant difference in perceived emotions before and after education by participants when they were presented with facial expression videos depicting disgust (see Table 2). Before education, the most frequently perceived emotion was sadness followed by disgust, $\chi^{2}$ (5) $=$ 38.00, $p<$ 0.001, $v=$ 0.19. However, after education, disgust became the most perceived emotion while sadness was perceived less frequently, $\chi^{2}$ (5) $=$ 167.17, $p<$ 0.001, $v=$ 0.40.

3.3. Fear

Table 3

Emotions perceived by sighted participants when viewing facial expression videos of fear displayed by people with visual impairments

Perceived emotions	Composition (%) of perceived emotions, given the fear facial expression		$\chi^{2}$ test between before and after
	Before education	After education
Anger	7.28	3.88	$\chi^{2}$ (1) $=$ 2.13, $p=$ 0.14, $v=$ 0.30
Disgust	16.50	9.71	$\chi^{2}$ (1) $=$ 3.63, $p=$ 0.06, $v=$ 0.26
Fear	12.14	17.96	$\chi^{2}$ (1) $=$ 2.32, $p=$ 0.13, $v=$ 0.19
Happiness	11.65	0.97	$\chi^{2}$ (1) $=$ 18.62, $p<$ 0.001, $v=$ 0.85
Sadness	18.93	4.85	$\chi^{2}$ (1) $=$ 17.16, $p<$ 0.001, $v=$ 0.59
Surprise	33.50	62.62	$\chi^{2}$ (1) $=$ 18.18, $p<$ 0.001, $v=$ 0.30
$\chi^{2}$ test between before and after	$\chi^{2}$ (5) $=$ 62.04, $p<$ 0.001, $v=$ 0.17

Chi-square testing revealed a significant difference in perceived emotions before and after education when participants viewed facial expression videos depicting fear (see Table 3). Before education, the most frequently perceived emotion was surprise, $\chi^{2}$ (5) $=$ 52.18, $p<$ 0.001, $v=$ 0.23. After education, surprise remained the most frequently perceived emotion, $\chi^{2}$ (5) $=$ 335.11, $p<$ 0.001, $v=$ 0.57. Although the perception of fear increased post-education, this change was not statistically significant.

3.4. Happiness

Table 4

Emotions perceived by sighted participants when viewing facial expression videos of happiness displayed by people with visual impairments

Perceived	Composition (%) of perceived emotions, given the happy facial expression		$\chi^{2}$ test between before and after
emotions	Before education	After education
Anger	0.49	0.97	$\chi^{2}$ (1) $=$ 0.33, $p=$ 0.56, $v=$ 0.33
Disgust	2.91	2.42	$\chi^{2}$ (1) $=$ 0.09, $p=$ 0.76, $v=$ 0.09
Fear	0.97	19.32	$\chi^{2}$ (1) $=$ 34.38, $p<$ 0.001, $v=$ 0.09
Happiness	81.55	69.57	$\chi^{2}$ (1) $=$ 1.85, $p=$ 0.17, $v=$ 0.08
Sadness	2.91	2.42	$\chi^{2}$ (1) $=$ 0.09, $p=$ 0.76, $v=$ 0.09
Surprise	11.17	5.31	$\chi^{2}$ (1) $=$ 4.24, $p=$ 0.04, $v=$ 0.35
$\chi^{2}$ test between before and after	$\chi^{2}$ (5) $=$ 40.98, $p<$ 0.001, $v=$ 0.14

Chi-square testing revealed a significant difference in perceived emotions before and after education when participants viewed facial expression videos depicting happiness (see Table 4). Before education, happiness was the most perceived emotion, $\chi^{2}$ (5) $=$ 633.71, $p<$ 0.001, $v=$ 0.78. After education, happiness remained the predominantly perceived emotion, $\chi^{2}$ (5) $=$ 445.49, $p<$ 0.001, $v=$ 0.66.

3.5. Sadness

Table 5

Emotions perceived by sighted participants when viewing facial expression videos of sadness displayed by people with visual impairments

Perceived emotions	Composition (%) of perceived emotions, given the sad facial expression		$\chi^{2}$ test between before and after
	Before education	After education
Anger	23.41	11.11	$\chi^{2}$ (1) $=$ 8.80, $p=$ 0.003, $v=$ 0.35
Disgust	25.85	17.39	$\chi^{2}$ (1) $=$ 3.25, $p=$ 0.07, $v=$ 0.19
Fear	8.29	2.90	$\chi^{2}$ (1) $=$ 5.26, $p=$ 0.02, $v=$ 0.48
Happiness	4.88	3.38	$\chi^{2}$ (1) $=$ 0.53, $p=$ 0.47, $v=$ 0.18
Sadness	21.95	61.84	$\chi^{2}$ (1) $=$ 39.82, $p<$ 0.001, $v=$ 0.48
Surprise	15.61	3.38	$\chi^{2}$ (1) $=$ 16.03, $p<$ 0.001, $v=$ 0.64
$\chi^{2}$ test between before and after	$\chi^{2}$ (5) $=$ 73.68, $p<$ 0.001, $v=$ 0.19

Chi-square testing revealed a significant difference in perceived emotions before and after education when participants viewed facial expression videos depicting sadness (see Table 5). Before education, the most frequently perceived emotion was disgust, followed by sadness, $\chi^{2}$ (5) $=$ 45.27, $p<$ 0.001, $v=$ 0.21. However, after education, sadness became the predominantly perceived emotion, while disgust was perceived less frequently $\chi^{2}$ (5) $=$ 324.68, $p<$ 0.001, $v=$ 0.56.

3.6. Surprise

Table 6

Emotions perceived by sighted participants when viewing facial expression videos of surprise displayed by people with visual impairments

Perceived emotions	Composition (%) of perceived emotions, given the surprise facial expression		$\chi^{2}$ test between before and after
	Before education	After education
Anger	3.88	1.91	$\chi^{2}$ (1) $=$ 1.33, $p=$ 0.25, $v=$ 0.33
Disgust	5.83	2.39	$\chi^{2}$ (1) $=$ 2.88, $p=$ 0.09, $v=$ 0.41
Fear	6.80	11.96	$\chi^{2}$ (1) $=$ 3.10, $p=$ 0.08, $v=$ 0.28
Happiness	53.40	2.87	$\chi^{2}$ (1) $=$ 93.24, $p<$ 0.001, $v=$ 0.90
Sadness	6.31	0.96	$\chi^{2}$ (1) $=$ 8.07, $p=$ 0.005, $v=$ 0.73
Surprise	23.79	79.9	$\chi^{2}$ (1) $=$ 64.46, $p<$ 0.001, $v=$ 0.55
$\chi^{2}$ test between before and after	$\chi^{2}$ (5) $=$ 173.08, $p<$ 0.001, $v=$ 0.29.

Chi-square testing revealed a significant difference in perceived emotions before and after education when participants viewed facial expression videos depicting surprise (see Table 6). Before education, the predominantly perceived emotion was happiness, followed by surprise, $\chi^{2}$ (5) $=$ 233.05, $p<$ 0.001, $v=$ 0.48. However, after education, surprise became the predominantly perceived emotion followed by fear while the perception of happiness decreased significantly, $\chi^{2}$ (5) $=$ 611.91, $p<$ 0.001, $v=$ 0.77.

4. Discussion

Table 7

The predominant emotions perceived by participants between before and after education

	The most perceived emotion
Given facially expressed emotions	Before education	After education
Anger	Happiness	Anger
Disgust	Sadness	Disgust
Fear	Surprise	Surprise
Happiness	Happiness	Happiness
Sadness	Disgust	Sadness
Surprise	Happiness	Surprise

This study contributes to advancing the understanding of sighted individuals’ capacity to accurately recognize facially expressed emotions in people with visual impairments in online settings. As presented in Table 7 (a summary of Tables 1 to 6), participants exhibited incongruent perception for all facially expressed emotions except happiness, suggesting difficulty in recognizing emotions accurately. However, after education they demonstrated congruent perception for all facially expressed emotions except fear, indicating a significant improvement in recognizing emotions accurately. The following sections offer a detailed discussion of these findings, providing further insights and interpretations.

4.1. Anger

Angry facial expressions presented a unique challenge for participants before education. Participants perceived a range of emotions, including anger, disgust, fear, happiness, sadness, and surprise. However, no specific emotion stood out significantly among these varied emotions. This suggests that sighted individuals are likely to experience challenges interpreting angry facial expressions exhibited by people with visual impairments, potentially confusing them with other emotions such as disgust, fear, happiness, sadness, or surprise. The presentation of angry expressions among individuals with visual impairments primarily involved Action Units (AU) 4 $+$ 15 $+$ 44 (brow lowerer $+$ lip corner depressor $+$ squint), AU 4 $+$ 14 (brow lowerer $+$ dimpler), or a neutral facial expression. More specifically, AU 4 played a substantial role in expressing anger, accounting for nearly 30% of all AU codes engaged in conveying this emotion. In contrast, other individual AU codes contributed less than 10%, with a recurring contribution rate as low as 1%. It was well documented that AU 4 has versatile roles in expressing various emotions among sighted people; for instance, sadness and fear [7, 51], disgust [52], sad [53], happiness [54], and surprise [55]. This versatility might have influenced participants in perceiving the presented angry facial expressions not solely as anger, but also other emotions.

Following education, a significant shift occurred in the perception of emotions. Anger emerged as the prominently perceived emotion, while happiness – which previously held that position – became the least perceived emotion. This substantial change underscores the effectiveness of education in reshaping participants’ interpretations of facial expressions among individuals with visual impairments in online communication settings.

4.2. Disgust

Before education, participants primarily perceived sadness followed by disgust when presented with facial expressions of disgust displayed by individuals with visual impairments. The primary facial expression used to convey both sadness and disgust was a neutral face (AU), potentially contributing to confusion between these emotions. Before education, participants also associated happiness with the facial expressions of disgust, despite the fundamental distinction between these emotions – one positive and the other negative. Wegrzyn et al. [56] found that sighted individuals tended to focus on lower facial muscles (e.g., the mouth) when perceiving such emotion as happiness and disgust. It can therefore be argued that participants in this study might have focused more on the lower part of the face. This selective attention might have led to perceiving similarities between the lower facial expressions of disgust and happiness, causing confusion between the two emotions. A more comprehensive attention encompassing both upper and lower facial areas might have helped participants to avoid such confusion. After education, the perception of happiness significantly diminished in response to the facial expression of disgust, indicating the effectiveness of education in improving the accurate identification of disgust.

4.3. Fear

Despite having received education, sighted participants tended to consistently misinterpret the facial expressions of fear as surprise. Jack et al. [57] proposed a reclassification of human emotions into four categories – (i) anger/disgust, (ii) fear/surprise, (iii) happiness, and (iv) sadness – instead of Ekman’s six categories [5]. According to Jack et al., fear and surprise are not distinctively expressed via facial muscles, resulting in potential confusion between the two emotions. Neta et al. [58]similarly reported that fear expressions were often misinterpreted as surprise due to shared facial muscle movements, particularly widened eyes, in the upper face. This study suggests that education should emphasize the importance of examining the lower parts of the face. This emphasis would enable sighted individuals to develop comprehensive analysis skills, leading to accurate interpretation of facial expressions of fear displayed by people with visual impairments.

4.4. Happiness

The predominantly perceived emotion was happiness both before and after education. Happiness was the only positive emotion among the six basic emotions, making it consequently easier for participants to identify happiness when viewing facial expressions. Hess et al. [59] found that the accuracy of recognizing facially expressed emotions is positively related with the intensity of the facial expression (i.e., the degree of facial muscle activity). Hess et al. also found that happy facial expressions even with a relatively low level of physical intensity are likely to be recognized with high accuracy. Leppänen and Hietanen [60] argued that a greater degree of facial muscle movements is typically observed in happy expressions compared to negative ones, which would make happy facial expressions more visually distinctive than other emotions. Adolphs [61] also noted that happiness could be easily recognized via even a single facial cue, $a$ smile whereas negative expressions might be harder to distinguish, as they may share several facial cues in common. In contrast to Ekman’s basic emotion model (where happiness is the only positive emotion), Fredrickson [62] identified ten distinct positive emotions: joy, gratitude, serenity, interest, hope, pride, amusement, inspiration, awe, and love. If this study had allowed participants to freely report any positive emotions they perceived, a lower rate of congruence might have been observed among participants.

4.5. Sadness

Before education, disgust was the predominantly perceived emotion but it shifted to sadness after education, highlighting the effectiveness of education. Prior to education, participants primarily perceived disgust and anger, followed by sadness. It is well documented [57, 63] that anger and disgust are often confused due to similarities in their facial expressions. The sad facial expression in this study was mainly conveyed through neutral face (AU 0). AU 0 was also prominently engaged in the facial expressions of disgust and anger. Such similarities in facial expressions might have resulted in confusion among the three emotions: disgust, anger, and sadness. However, after education, participants were less likely to perceive anger and disgust, which is evidence of the educational effect.

4.6. Limitations

A few limitations might have influenced the findings. Participants viewed a series of 60 video clips, which could have induced fatigue potentially impacting their ability to accurately perceive emotions (cognitive fatigue [64]). However participants spent only a few seconds per video clip and were provided a break time after the initial set of 30 video clips, mitigating the expected impact of fatigue. The videos featured background images (e.g., household goods), which might have influenced participants’ emotions [65].

5. Conclusion

This study investigated the ability of sighted individuals to recognize facial expressions of emotions in people with visual impairments in online settings. Initially, participants frequently missed, misinterpreted and/or misclassified critical facial cues in the videos featuring facial expressions of individuals with visual impairments. However, after education, participants showed significantly improved performance in recognizing these facially expressed emotions. These findings suggest that proper education can effectively enhance the ability of sighted individuals to accurately interpret facial expressions conveyed by people with visual impairments in cyberspace.

Footnotes

Acknowledgments

This material is based upon work supported by the National Science Foundation under Grant No. 1831969.

Conflict of interest

The author has no conflicts of interest to report.

Ethical considerations

Ethical approval for this study was obtained from the Institutional Review Board. Informed consent was obtained for all study participants.

References

Kunz

Faltermeier

Lautenbacher

. Impact of visual learning on facial expressions of physical distress: A study on voluntary and evoked expressions of pain in congenitally blind and sighted individuals. Biological Psychology. 2012; 89(2): 467-76.

Westbrook

Nessel

Varacallo

. Anatomy, Head and Neck, Face: Treasure Island (FL): StatPearls Publishing; 2022. [updated April 20, 2024; cited 2024 August 19]. Available from: https://www.ncbi.nlm.nih.gov/books/NBK493209/.

Valente

Theurel

Gentaz

. The role of visual experience in the production of emotional facial expressions by blind people: A review. Psychonomic Bulletin and Review. 2018; 25(2): 483-97.

Kemper

. How many emotions are there? Wedding the social and the autonomic components. American Journal of Sociology. 1987; 93(2): 263-89.

Ekman

. Facial expressions. In: D’algleish

Power

, editors. Handbook of Cognition and Emotion. Susses, UK: John Wiley and Sons, Ltd. 1999; 301-20.

Prkachin

. The consistency of facial expressions of pain: a comparison across modalities. Pain. 1992; 51(3): 297-306.

Waller

Cray

Jr. Burrows

. Selection for universal facial emotion. Emotion. 2008; 8(3): 435.

Yeasin

Bullot

Sharma

. Recognition of facial expressions and measurement of levels of interest from video. IEEE Transactions on Multimedia. 2006; 8(3): 500-8.

Reyes

Meza

Pineda

. Robotics facial expression of anger in collaborative human–robot interaction. International Journal of Advanced Robotic Systems. 2019; 16(1): 1729881418817972.

10.

Matsumoto

. More evidence for the universality of a contempt expression. Motivation and Emotion. 1992; 16(4): 363-8.

11.

Ekman

Friesen

. Measuring facial movement. Environmental Psychology and Nonverbal Behavior. 1976; 1(1): 56-75.

12.

Ekman

Friesen

. Facial action coding system: a technique for the measurement of facial movement. California, CA: Consulting Psychologists Press, Palo Alto; 1978.

13.

Tian

Y-L

Kanade

Colin

. Recognizing action units for facial expression analysis. Multimodal interface for human-machine communication: World Scientific. 2002; 32-66.

14.

Scherer

Ellgring

Dieckmann

Unfried

Mortillaro

. Dynamic facial expression of emotion and observer inference. Frontiers in Psychology. 2019; 10: 508.

15.

Kim

. Facial expressions of emotions by people with visual impairment and blindness via video conferencing. British Journal of Visual Impairment. 2023.

16.

Swain

O’Hare

Brandley

Gardner

. Individual differences in social intelligence and perception of emotion expression of masked and unmasked faces. Cognitive Research: Principles and Implications. 2022; 7(1): 1-9.

17.

Elfenbein

Marsh

Ambady

. Emotional intelligence and the recognition of emotion from facid expressions. The Wisdom in Feeling: Psychological Processes in Emotional Intelligence. The Guilford Press: New York, NY, USA. 2002.

18.

Chen

. Face recognition as a predictor of social cognitive ability: Effects of emotion and race on face processing. Asian Journal of Social Psychology. 2014; 17(1): 61-9.

19.

Gao

Maurer

Nishimura

. Altered representation of facial expressions after early visual deprivation. Frontiers in Psychology. 2013; 4: 878.

20.

Field

Woodson

Greenberg

Cohen

. Discrimination and imitation of facial expression by neonates. Science. 1982; 218(4568): 179-81.

21.

Walker-Andrews

. Emotions and social development: Infants’ recognition of emotions in others. Pediatrics. 1998; 102(5 Suppl E): 1268-71.

22.

Ichikawa

Yamaguchi

. Infants’ recognition of subtle anger facial expression. Japanese Psychological Research. 2014; 56(1): 15-23.

23.

Barrett

Adolphs

Marsella

Martinez

Pollak

. Emotional expressions reconsidered: Challenges to inferring emotion from human facial movements. Psychological Science in the Public Interest. 2019; 20(1): 1-68.

24.

Döllinger

Laukka

Högman

Bänziger

Makower

Fischer

, et al. Training emotion recognition accuracy: Results for multimodal expressions and facial micro expressions. Frontiers in Psychology. 2021; 12: 708867.

25.

Schoeneman Patel

Haut

Guty

Dodell-Feder

Saxena

Nahum

, et al. Social cognition training improves recognition of distinct facial emotions and decreases misattribution errors in healthy individuals. Frontiers in Psychiatry. 2022; 13: 1026418.

26.

Ekman

Sorenson

Friesen

. Pan-cultural elements in facial displays of emotion. Science. 1969; 164(3875): 86-8.

27.

Namba

Makihara

Kabir

Miyatani

Nakao

. Spontaneous facial expressions are different from posed facial expressions: Morphological properties and dynamic sequences. Current Psychology. 2017; 36(3): 593-605.

28.

Galati

Sini

Schmidt

Tinti

. Spontaneous facial expressions in congenitally blind and sighted children aged 8–11. Journal of Visual Impairment and Blindness. 2003; 97(7): 418-28.

29.

Galati

Miceli

Sini

. Judging and coding facial expression of emotions in congenitally blind children. International Journal of Behavioral Development. 2001; 25(3): 268-78.

30.

Tracy

Matsumoto

. The spontaneous expression of pride and shame: Evidence for biologically innate nonverbal displays. Proceedings of the National Academy of Sciences. 2008; 105(33): 11655-60.

31.

Matsumoto

Willingham

. Spontaneous facial expressions of emotion of congenitally and noncongenitally blind individuals. Journal of Aersonality and Social Psychology. 2009; 96(1): 1-10.

32.

Galati

Scherer

Ricci-Bitti

. Voluntary facial expression of emotion: Comparing congenitally blind with normally sighted encoders. Journal of Personality and Social Psychology. 1997; 73(6): 1363-79.

33.

Kim

. User Experience of Mainstream and Assistive Technologies for People with Visual Impairments. Technology and Disability. 2018; 30(3): 127-33.

34.

Doush

Al-Jarrah

Alajarmeh

Alnfiai

. Learning features and accessibility limitations of video conferencing applications: are people with visual impairment left behind. Universal Access in the Information Society. 2022; 1-16.

35.

Acosta

Nso-Mangue

Luján-Mora

, editors. Accessibility barriers of video conferencing tools focused on people with visual disabilities. INTED2023; Proceedings; 2023; IATED.

36.

Alvandi

Van Doorn

Symmons

. Emotional awareness and decision-making in the context of computer-mediated psychotherapy. Journal of Healthcare Informatics Research. 2019; 3: 345-70.

37.

Ali

Myers

Wagner

Ratnu

Dorsey

Hoque

. Facial expressions can detect Parkinson’s disease: preliminary evidence from videos collected online. npj Digital Medicine. 2021; 4(1): 129.

38.

Biel

J-I

Teijeiro-Mosquera

Gatica-Perez

, editors. Facetube: predicting personality from facial expressions of emotion in online conversational video. Proceedings of the 14th ACM international conference on Multimodal interaction; 2012. Santa Monica, California.

39.

Chandrasiri

Naemura

Ishizuka

Harashima

Barakonyi

. Internet communication using real-time facial expression analysis and synthesis. IEEE MultiMedia. 2004; 11(3): 20-9.

40.

Bandyopadhyay

Thakur

Mandal

, editors. Online recommendation system using human facial expression based emotion detection: a proposed method. Proceedings of International Conference on Advanced Computing Applications: ICACA 2021; 2022; Springer.

41.

Nubani

Öztürk

. Measuring the impact of museum architecture, spaces and exhibits on virtual visitors using facial expression analysis software. Buildings. 2021; 11(9): 418.

42.

Wang

Niu

Miao

. Emotion recognition of students based on facial expressions in online education based on the perspective of computer simulation. Complexity. 2020; 2020: 1-9.

43.

Zhang

Liu

Cao

Liu

. Data-driven online learning engagement detection via facial expression and mouse behavior recognition technology. Journal of Educational Computing Research. 2020; 58(1): 63-86.

44.

Savchenko

Makarov

. Classifying emotions and engagement in online learning based on a single facial expression recognition neural network. IEEE Transactions on Affective Computing. 2022; 13(4): 2132-43.

45.

Ashok

John

. Facial expression recognition system for visually impaired. 2018. In: International Conference on Intelligent Data Communication Technologies and Internet of Things (ICICI) 2018 ICICI 2018 Lecture Notes on Data Engineering and Communications Technologies [Internet]. Springer, Cham; 244-50.

46.

Zhao

Reynolds

Azenkot

, editors. A face recognition application for people with visual impairments: Understanding use beyond the lab. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems; 2018. Montréal, Canada.

47.

Den Uyl

Van Kuilenburg

, editors. The FaceReader: Online facial expression recognition. Proceedings of Measuring Behavior 2005 – 5th International Conference on Methods and Techniques in Behavioral Research; 2005. Wageningen, The Netherlands.

48.

Kim

. The frequency of facial muscles engaged in expressing emotions in people with visual disabilities via cloud-based video communication. Theoretical Issues in Ergonomics Science. 2023; 24(3): 267-80.

49.

World Health Organization. Visual impairment and blindness 2015 [cited 2024 August 19]. Available from: www.who.int/mediacentre/factsheets/fs282/en/#.

50.

IBM Corp. IBM SPSS Statistics for Macintosh, Version 24.0. Armonk, NY: IBM Corp; 2016.

51.

Clark

Kessinger

Duncan

Bell

Lahne

Gallagher

, et al. The facial action coding system for characterization of human affective response to consumer product-based stimuli: a systematic review. Frontiers in Psychology. 2020; 11: 920.

52.

Kunz

Peter

Huster

Lautenbacher

. Pain and disgust: The facial signaling of two aversive bodily experiences. PloS One. 2013; 8(12): e83277.

53.

Pampouchidou

Simos

Marias

Meriaudeau

Yang

Pediaditis

, et al. Automatic assessment of depression based on visual cues: A systematic review. IEEE Transactions on Affective Computing. 2017; 10(4): 445-70.

54.

Crivelli

Carrera

Fernández-Dols

J-M

. Are smiles a sign of happiness? Spontaneous expressions of judo winners. Evolution and Human Behavior. 2015; 36(1): 52-8.

55.

Noordewier

van Dijk

. Surprise: unfolding of facial expressions. Cognition and Emotion. 2019; 33(5): 915-30.

56.

Wegrzyn

Vogt

Kireclioglu

Schneider

Kissler

. Mapping the emotional face. How individual face parts contribute to successful emotion recognition. PloS One. 2017; 12(5): e0177239.

57.

Jack

Garrod

Schyns

. Dynamic facial expressions of emotion transmit an evolving hierarchy of signals over time. Current Biology. 2014; 24(2): 187-92.

58.

Neta

Tong

Rosen

Enersen

Kim

Dodd

. All in the first glance: first fixation predicts individual differences in valence bias. Cognition and Emotion. 2017; 31(4): 772-80.

59.

Hess

Blairy

Kleck

. The intensity of emotional facial expressions and decoding accuracy. Journal of Nonverbal Behavior. 1997; 21241-57.

60.

Leppänen

Hietanen

. Positive facial expressions are recognized faster than negative facial expressions, but why? Psychological Research. 2004; 69(1-2): 22-9.

61.

Adolphs

. Recognizing emotion from facial expressions: psychological and neurological mechanisms. Behavioral and Cognitive Neuroscience Reviews. 2002; 1(1): 21-62.

62.

Fredrickson

. Positive emotions broaden and build. Advances in Experimental Social Psychology. 2013; 47: 1-53.

63.

Ventura

Palmisano

Innamorato

Tedesco

Manippa

Caffò

, et al. Face memory and facial expression recognition are both affected by wearing disposable surgical face masks. Cognitive Processing. 2022; 1-15.

64.

Ioannucci

Chirokoff

Dilharreguy

Ozenne

Chanraud

Zénon

. Neural fatigue by passive induction: repeated stimulus exposure results in cognitive fatigue and altered representations in task-relevant networks. Communications Biology. 2023; 6(1): 142.

65.

Chang

W-Y

M-T

Huang

C-F

. The influence of emotional environmental pictures on behavior intentions: the evidence of neuroscience technology. International Journal of Environmental Research and Public Health. 2019; 16(24): 5142.