Facial expressions of emotions by people with visual impairment and blindness via video conferencing

Abstract

Many people including those with visual impairment and blindness take advantage of video conferencing tools to meet people. Video conferencing tools enable them to share facial expressions that are considered as one of the most important aspects of human communication. This study aims to advance knowledge of how those with visual impairment and blindness share their facial expressions of emotions virtually. This study invited a convenience sample of 28 adults with visual impairment and blindness to Zoom video conferencing. The participants were instructed to pose facial expressions of basic human emotions (anger, fear, disgust, happiness, surprise, neutrality, calmness, and sadness), which were video recorded. The facial expressions were analyzed using the Facial Action Coding System (FACS) that encodes the movement of specific facial muscles called Action Units (AUs). This study found that there was a particular set of AUs significantly engaged in expressing each emotion, except for sadness. Individual differences were also found in AUs influenced by the participants’ visual acuity levels and emotional characteristics such as valence and arousal levels. The research findings are anticipated to serve as the foundation of knowledge, contributing to developing emotion-sensing technologies for those with visual impairment and blindness.

Keywords

Empathy individual differences online communication visual disability Zoom

Introduction

Humans perceive and express various emotions in everyday life, which facilitates effective social interactions (Matsumoto & Willingham, 2009). Yet, no consensus exists on the definition of emotion. Emotion is often exemplified in the literature as anger, fear, disgust, happiness, surprise, neutrality, calmness, and sadness (Cabanac, 2002). More specifically, emotion is defined as a complex state of feeling, leading to physical and psychological changes that affect thoughts and behaviors. Myers (2004) argued that human emotion involves physiological arousal, expressive behaviors, and conscious experience. Theories of emotion are grouped into three categories: physiological, neurological, and cognitive theories (R. Barrett, 2016). According to physiological theories, physiological changes in the body lead to the experience of emotions (e.g., James-Lange Theory of Emotion (Cannon, 1927)); for example, an external stimulus leads to a physiological response, and an emotional reaction is influenced by how an individual interprets the physiological response. Cognitive theories argue that thoughts and other mental activities play a critical role in forming emotions (e.g., Cognitive Appraisal Theory (Lazarus et al., 1970)); for example, the brain first analyzes a situation, and the resulting response is an emotion. Neurological theories suggest that activities within the brain lead to emotional responses (e.g., Facial-Feedback Theory of Emotion (Buck, 1980)); for example, emotions are directly linked to changes in facial muscles, such that a facial expression is a means to display emotion but also influences the emotional experience (Söderkvist et al., 2018). The perceived emotions are typically expressed via a range of channels such as prosody (Adolphs et al., 2002), nonverbal vocalizations (Sauter et al., 2010), posture (Aviezer et al., 2012), chemosensory signals (Mujica-Parodi et al., 2009), and language (Pell et al., 2009). Ekman (1993) argued that humans tend to depend significantly on facial expressions among those various channels. Facial expressions convey rich emotional information and are considered as one of the most important forms of interpersonal communication (Riggio & Riggio, 2012; Song, 2021). An individual’s ability to understand the mental states of others by reading facial expressions is a critical factor that contributes to social cognition (Behere, 2015), that is, the ability to process, remember, and use information in social contexts to explain and predict one’s own behavior or that of others (Fiske & Taylor, 2013).

Sighted people tend to learn and imitate other people’s facial expressions through observation from the first day of life, while people with visual impairment and blindness are less likely to benefit from such vision-based learning (Meltzoff & Moore, 1983). Although people with visual impairment and blindness have a limited ability to perceive emotions via facial expressions of other people due to visual challenges, they can still perceive various emotions through their remaining sensory modalities (e.g., hearing) and are capable of revealing their emotions via facial expressions. Iverson and Goldin-Meadow (1998) compared the gestures between sighted children/adolescents and their peers who were blind since birth. They found that those with blindness still used gestures when they spoke to sighted people and their peers with blindness although they never saw other people using gestures as they were born blind. The visual challenges would not stop those with visual impairment and blindness from perceiving different emotions and displaying a variety of facial expressions to fully engage in affective communication with other people. It is important for sighted people to correctly understand facial expressions displayed by people with visual impairment and blindness in the home, workplace, and community to facilitate diverse and inclusive social interactions. For example, if sighted people pay attention to what people with visual impairment and blindness say but ignore what their face tells, they are less likely to engage deeply in communication with those with visual impairment and blindness. Yet, sighted people with a comprehensive understanding of facial expressions posed by people with visual impairment and blindness can adequately interpret emotional states, and such a high level of ability to read facial expressions is likely to lead to successfully building a strong empathy, contributing to interpersonal relationships (Dimberg et al., 2011).

There is evidence that an individual difference exists in facial expressions between people with different visual acuity levels. Thompson (1941) found that two groups of children with and without blindness showed similar facial expressions for smiling, laughter, physical discomfort, and anger; however, their longitudinal observations revealed that children with blindness tended to smile less as time passed. Cole et al. (1989) pointed out that even if people have lost vision, they can still control their facial expressions when they perceive negative emotions. In their study, children with blindness were engaged in showing positive facial expressions as much as their sighted peers. However, those with blindness were more likely to engage in neutral facial expressions when they received a disappointing prize as compared with sighted peers. Peleg et al. (2009) measured the total number of times a facial movement was observed (i.e., frequency) and found individual differences in the frequency of facial expressions between sighted people and people with congenital blindness. Sighted people showed a higher frequency of expressing anger as compared with their peers with congenital blindness, while people with congenital blindness showed a higher frequency of expressing other emotions such as disgust and surprise than did sighted people. Kunz et al. (2012) suggested that the repertoire of facial muscles engaged in experiencing pain is biologically prepared regardless of visual ability and disability. Nonetheless, they observed that people with blindness were poor at voluntarily modulating their pain expression as compared with sighted people. They claimed that visual learning is a prerequisite to encode adequately different intensities of physical distress (e.g., pain).

The facial expressions can be analyzed using a Facial Action Coding System (FACS), that is, a comprehensive anatomically based system to distinguish all visible facial movements (Ekman & Friesen, 1978; Ekman & Friesen, 1976). The FACS helps to break down all visually discernible facial expressions into individual components of muscle movements, called Action Units (AUs). A total of 44 AUs cover various facial components (e.g., brows, eyelids, lips, head, nose, and cheeks) (Tian et al., 2002). AUs are mapped onto facial expressions to present emotions, for example, AU 1 refers to Inner Brow Raiser, AU 9 refers to Nose Wrinkler, and AU 12 refers to Lip Corner Puller. Many efforts have been made to observe how general populations make facial expressions (L. F. Barrett et al., 2019). Benitez-Quiroz et al. (2016) coded facial expressions using the FACS and reported that AU 12 + 25 display happiness; AU 4 + 15 display sadness; AU 1 + 4 + 20 + 25 display fear; AU 4 + 7 + 24 display anger; AU 1 + 2 + 25 + 26 display surprise; and AU 9 + 10 + 17 display disgust. However, research studies on facial expressions showed a lack of unanimity on facial expression modeling. For example, Kohler et al. (2004) observed that sadness was facially expressed via a wide range of AUs, that is, AU 1 + 4 + 7 + 10 + 15 + 17 + 20 + 25. Keltner et al. (2019) reported that AU 6 + 7 + 12 + 25 + 26 represents happiness, while Matsumoto et al. (2008) reported that AU 6 + 12 represents happiness. In addition, Cordaro et al. (2018) coded over 2600 facial and body expressions of 22 emotions in China, India, Japan, Korea, and the United States. They found cultural variants in emotional expression. Mesquita and Frijda (1992) conducted a comprehensive literature review on cultural emotional variations. They suggested that cultural differences in emotions are induced by differences in event types, culture-specific appraisal propensities, behavior repertoires, and regulation processes.

Unfortunately, there are still only a handful of published reports on “voluntary” facial expressions of emotions perceived by people with visual impairment and blindness. People visualize their emotions via facial expressions either spontaneously or voluntarily. When people are exposed to an emotional stimulus (e.g., watching video clips), their natural emotions may unconsciously be expressed. That is referred to as spontaneous expression. When people are asked to make an effort to express emotions, it is referred to as voluntary expression (Kang et al., 2019). Valente et al. (2018) conducted a systematic literature review of 21 published articles that examined facial expressions posed by people with blindness. Their review indicated that there had been a lack of attention paid to voluntary emotion expressions, as only eight articles (38%) investigated voluntary emotion expressions while the majority of studies investigated spontaneous expressions. Among the few studies on voluntary expressions, detailed analyses of the engagement of facial muscles were not fully reported. For example, Peleg et al. (2009) studied the facial expressions of people with blindness, but they did not deeply investigate the degree to which each facial muscle is individually engaged in expressing different voluntary emotions.

Furthermore, little is known about how people with visual impairment and blindness display voluntary facial expressions via video conferencing. Studies that focused on facial expressions in online contexts tended to examine spontaneous facial expressions (Bian et al., 2019; Biel et al., 2012; Kim & Sutharson, 2022). Today, many video conferencing tools (e.g., Skype, Google Hangouts, Teams, and Zoom) have been advanced to improve the accessibility for users with visual impairment and blindness (Hersh et al., 2020; Leporini et al., 2021). For example, Google Hangouts software works well with assistive technology such as JAWS, NVDA, VoiceOver, and ChromeVox. Understanding facial expressions of emotions in the context of online communication can be beneficial to various areas, such as online learning and telehealth. For instance, emotions would affect students’ cognitive, motivational, behavioral, cognitive-behavioral, and socio-behavioral engagement, ultimately influencing students’ academic performance (Linnenbrink, 2006; Linnenbrink-Garcia & Pekrun, 2011; Pekrun & Linnenbrink-Garcia, 2012). However, many studies focused on sighted individuals over their peers with visual impairment and blindness (Bian et al., 2019; Biel et al., 2012; Kim & Park, 2017; Sun et al., 2017; Zhan et al., 2006) and negative emotions only, such as boredom, confusion, and frustration (Bosch et al., 2016; Grafsgaard et al., 2013; Vail et al., 2016; Whitehill et al., 2014). Many healthcare professionals have been using remote consultations via online platforms (e.g., video conferencing) (Greenhalgh et al., 2016; Mann et al., 2021). It is critical for healthcare professionals to understand and diagnose the emotional status of e-healthcare consumers (including those with visual impairment and blindness) accurately by reading facial expressions, develop empathy, and provide adequate accommodations and services online. Yet, little is known about how people with visual impairment and blindness use voluntary facial expressions to display emotions in cyberspace. This knowledge gap would lead to poor intervention designs and developments that aim to promote the emotional well-being among those with visual impairment and blindness. This study aims to advance knowledge of how people with visual impairment and blindness display voluntary facial expressions in video conferencing contexts and how their facial expressions are affected by individual factors such as visual acuity levels and emotion types.

Methods

Participants

A convenience sample of 28 participants with visual impairment and blindness were invited to participate in this study. Inclusion criteria included English-speaking, 18 years old or older, and severe visual impairment/blindness. Visual acuity between 20/200 and 20/400 is considered as severe visual impairment, while visual acuity of less than 20/400 is considered as blindness (World Health Organization, 2008). This study did not focus on user experience (e.g., usability and accessibility) of video conferencing, but focused on how those with visual impairment and blindness express emotions facially in cyberspace. Hence, even non-users of video conferencing tools could participate in this study. We recruited research participants in collaboration with community organizations such as community centers and a public library for the blind. Approval for this study was obtained from the Institutional Review Board (IRB). Characteristics of the participants are presented in Table 1.

Table 1.

Descriptions of the participants.

Participants	n = 28
Visual acuity
Between 20/200 and 20/400	9
Between 20/400 and 20/1200	4
Less than 20/1200, but has light perception	13
No light perception at all	2
Duration of visual impairment (years)	27.71 ± 23.92
Onset of visual impairment (years)^a
Early onset	0.4 ± 1.26
Late onset	47.89 ± 14.93
Age (years)	58.64 ± 17.93
Gender
Male	11
Female	17
Race/ethnicity
African American	10
European American	16
Hispanic American	1
Others	1
Occupation
Employed	9
Unemployed	19
Head of household
Living alone	7
With family, relatives, friends, or a combination of them	21
Education
High school or equivalent	6
Certificate or training program	1
Associate	6
Bachelor’s (BA or BS)	4
Master’s	10
Doctorate	1
Marital status
Married	14
Not married	6
Widow(er)	3
Divorced	5

Participants with early onset of vision loss had lost their sight before 11 years of age (Voss et al., 2004).

Procedures

Interviews were conducted via Zoom. Participants’ family members were allowed to assist with downloading the Zoom application on the participant’s computing devices (e.g., smartphones and computers) if necessary. A few participants (n = 4, 14%) obtained help from their family members with installing the Zoom app, while most participants (n = 24, 86%) had already been using the Zoom app. Once the interview begins, the participant’s helper should be absent from the interview. Figure 1 shows a screenshot of what the participants attended via Zoom.

Figure 1.

Participants with visual impairment and blindness participated in this study via Zoom.

The participants were instructed to pose facial expressions of the eight basic emotions, including anger, fear, disgust, happiness, surprise, neutrality, calmness, and sadness in a random order. To facilitate their voluntary expressions, we asked the participants to take a brief moment to recall their emotional experiences concerning the eight emotions and pose relevant facial expressions. It was not necessary for them to share such emotional experiences with us. While the participants expressed their emotions via Zoom, we took screenshots to capture their facial expressions. When they mistakenly looked at other places instead of the webcam due to their visual challenge, we instructed them to adjust their head position. A demographic survey was administered to obtain a deep understanding of the characteristics of the participants (e.g., age, visual acuity, race/ethnicity, education, and so on), which are summarized in Table 1. We read the survey questions out loud, clearly, and slowly to help the participants understand fully. We repeated questions when they requested.

Data analysis

The FACS helped to break down all visually discernible facial expressions into individual components of muscle movements, called AUs. The AUs cover various facial components (e.g., brows, eyelids, lips, head, nose, and cheeks). The coding was accomplished by FACS coders who were trained based on the instructions of Cohn et al. (2007) and competent with coding facial expressions using the FACS as shown in previous research (Kim, 2022; Kim & Sutharson, 2022). AUs were compared within each emotion and between emotions. Hence, a deep understanding could be obtained with regard to how facial expressions are engaged in expressing different emotions. In addition, AUs were compared by a variety of emotional categories, that is, neutral/non-neutral, valence, and arousal, that is, emotional dimensions (L. F. Barrett & Russell, 1999). Therefore, we could explore whether there were any differences in making facial expressions for each emotional dimension. Emotional valence refers to how pleasant or unpleasant an individuals’ feeling is, ranging from negative to positive, whereas emotional arousal refers to its intensity, ranging from low to high (L. F. Barrett & Russell, 1999; Lang et al., 1997; Russell, 2003). Statistical analyses were performed using the IBM SPSS Statistics for Macintosh, version 24 (IBM Corp., 2016). The interrater reliability (Leclerc & Dassa, 2009) was assessed using a subset (10%) of the total data. Another coder helped to assess the interrater reliability using Cohen’s kappa statistic, which resulted in substantial agreement among the raters as the interrater reliability was found to be k = 0.66 (95% confidence interval [CI] = 0.32–1.00), p < .05.

Results

AUs comparison within emotion

As shown in Figure 2, particular facial muscle movements (i.e., AUs) were extensively engaged in expressing each of the following emotions: anger, neutrality, fear, disgust, happiness, calmness, and surprise. In addition, one sample chi-square testing statistically confirmed that there were particular AUs significantly engaged in expressing each emotion: anger, χ²(25) = 129.99, p < .001; neutrality, χ²(16) = 79.00, p < .001; fear, χ²(21) = 87.14, p < .001; disgust, χ²(34) = 72.96, p < .001; happiness, χ²(18) = 182.75, p < .001; calmness, χ²(17) = 85.15, p < .001; and surprise, χ²(22) = 113.51, p < .001. However, there were no particular AUs greatly engaged in expressing sad, that is, “sad” was expressed facially via a wide range of AUs associated with the head, nose, brows, eyelids, lips, and cheeks.

Figure 2.

Action Units (AU) contributing to each emotion expression.

AUs comparison between emotions

We conducted chi-square testing to compare two emotions at a time in terms of AUs engaged in expressing the emotions. As shown in Table 2, the majority of the comparisons were found to be significantly different. It suggests that the participants with visual impairment and blindness tended to use different AUs to express different emotions. Yet, a few cases resulted in non-significance (i.e., Anger vs Disgust, Neutrality vs Calmness, and Fear vs Surprise), which suggests that similar AUs are engaged in expressing those particular emotions.

Table 2.

Significantly different engagements of Action Units across emotions.

	Anger	Neutrality	Fear	Disgust	Happiness	Sadness	Calmness	Surprise
Anger	–	N.S.	χ²(35) = 105.43, p < .001, v = .83	N.S.	χ²(34) = 128.71, p < .001, v = .89	χ²(36) = 66.57, p = .001, v = .72	χ²(31) = 65.10, p < .001, v = .74	χ²(36) = 129.44, p < .001, v = .87
Neutrality	–	–	χ²(28) = 67.67, p < .001, v = .77	χ²(39) = 68.27, p = .003, v = .76	χ²(27) = 92.99, p < .001, v = .85	χ²(27) = 47.85, p = .008, v = .73	N.S.	χ²(29) = 84.60, p < .001, v = .80
Fear	–	–	–	χ²(39) = 83.95, p < .001, v = .74	χ²(28) = 81.32, p < .001, v = .71	χ²(34) = 98.84, p < .001, v = .89	χ²(26) = 65.04, p < .001, v = .76	N.S.
Disgust	–	–	–	–	χ²(38) = 91.59, p < .001, v = .74	χ²(40) = 64.17, p = .009, v = .71	χ²(39) = 62.42, p = .01, v = .73	χ²(39) = 96.74, p < .001, v = .75
Happiness	–	–	–	–	–	χ²(33) = 119.03, p < .001, v = .93	χ²(26) = 89.05, p < .001, v = .84	χ²(27) = 74.64, p < .001, v = .64
Sadness	–	–	–	–	–	–	χ²(30) = 50.41, p = .011, v = .75	χ²(35) = 114.93, p < .001, v = .90
Calmness	–	–	–	–	–	–	–	χ²(29) = 84.69, p < .001, v = .80

N.S.: nonsignificant.

AUs comparison by neutral and non-neutral emotions

Even when the participants were instructed to express neutrality, 13 participants (46%) still used various AUs associated with brows, eyelids, lips, and head. On the contrary, when they were instructed to express non-neutral emotions (e.g., disgust, sadness, and calmness), their facial muscles were not engaged such that their emotion appeared to be neutral. For example, a neutral facial expression was used by two participants (7%) in expressing disgust, by five participants (18%) in expressing sadness, and by 15 participants (54%) in expressing calmness. As shown in Table 3, the participants used the AU # 0 (i.e., neutral facial expression) to display neutral emotion and others such as anger, disgust, sadness, and calmness. The AU # 0 was the most frequently engaged when expressing calmness (53.57%) and neutrality (53.57%).

Table 3.

Combination of Action Units engaged in expressing emotions.

Emotion	Action Units	Percentage of participants who used the same combination of Action Units
Anger	AU4 + AU15 + AU44	10.71
	AU4 + AU14	7.14
	AU0	7.14
Fear	AU1 + AU5 + AU25	7.14
Disgust	AU4	7.14
	AU0	7.14
Happiness	AU6 + AU12 + AU25	25.00
Sadness	AU0	17.86
	AU17 + 41 + 54 + 64	7.14
	AU15	7.14
Surprise	AU1 + 5 + 25 + 27	10.71
	AU25 + 45	7.14
Neutrality	AU0	53.57
	AU56	7.14
Calmness	AU0	53.57
	AU1 + AU13	7.14

AUs comparison by valence and arousal of emotions

According to the Circumplex Model of Affect (Russell, 1980), emotional expressions/perceptions are theoretically classified by considering two important dimensions: valence and arousal. Valence describes attractiveness (positive valence) or aversiveness (negative valence), while arousal refers to the perceived intensity of emotion from calming (low arousal) to exciting or agitating (high arousal). Thus, emotional expressions made by the participants were studied to examine the degree to which their facial expressions differ depending on the emotional dimensions. Chi-square testing was conducted to compare the AUs by valence, arousal, and a combination of both. First, a significant difference was found for valence (positive, negative, and neutral), χ²(98) = 258.43, p < .001, v = .49. More specifically, significantly different AUs were engaged in expressing between positive and negative emotions, χ²(48) = 151.81, p < .001, v = .60; between positive and neutral emotions, χ²(31) = 61.89, p < .001, v = .61; and between negative and neutral emotions, χ²(49) = 121.57, p < .001, v = .54.

Second, chi-square testing found a significant difference for arousal (high, low, and neutral), χ²(98) = 287.78, p < .001, v = .52. More specifically, significantly different AUs were engaged in expressing between high and low arousal emotions, χ²(48) = 198.12, p < .001, v = .63 and between high and neutral emotions, χ²(46) = 162.24, p < .001, v = .60.

Third, chi-square testing found a significant difference for a combination of valence and arousal levels, χ²(144) = 507.63, p < .001, v = .58. More specifically, significantly different AUs were engaged in expressing between high-arousal negative emotions and low-arousal negative emotions, χ²(33) = 119.03, p < .001, v = 0.93; between high-arousal negative emotions and low-arousal positive emotions, χ²(45) = 129.33, p < .001, v = 0.60; and between low-arousal positive emotions and low-arousal negative emotions, χ²(30) = 50.41, p < .01, v = .75.

Discussion

Findings in this study are discussed by referring to the earlier research findings although there are only a few published articles that investigated the facial expressions of emotions in people with visual impairment and blindness.

AUs within emotion

Individual differences were found in AUs that were utilized to express the eight emotions. For example, there were a particular set of AUs significantly engaged in expressing each emotion, except sadness. The participants used a wide range of facial components (i.e., brows, eyelids, lips, head, nose, and cheeks) in expressing sadness as compared with other emotions. Thus, it would be difficult to suggest that a particular set of facial components is primarily engaged in expressing sadness among those with visual impairment and blindness. The research finding is consistent with the study of Galati et al. (1997), in that their study also observed that participants with congenital blindness used a variety of facial components, including brows, eyelids, lips, and head to express sadness. A later study by Galati et al. (2003) observed children with congenital blindness in their classrooms during normal school hours after informing those children that their teacher would leave the school. The situation was expected to contribute to inducing those students to feel sad. Their study confirmed again that children with congenital blindness used a range of different facial expressions, for example, AU 1, 2, 4, 7, 44, 15, 26, 53, 54, 56, 63, and 64. It infers that sighted people (e.g., teachers and health professionals) would probably have difficulty in recognizing whether people with visual impairment and blindness (e.g., students and patients) suffer from such negative emotion by reading their facial expressions only, unless they show an obvious sign of sadness such as crying. It also infers that an automated system for the classification of facial expressions and the recognition of relevant emotions would be less reliable for the emotion of sad in people with visual impairment and blindness.

AUs between emotions

AUs were found to be differently engaged in expressing different emotions except in a few cases. For example, the AUs for anger were not significantly different from those for neutrality. Such non-significance was also found between anger and disgust; between neutrality and calmness; and between fear and surprise. Thus, it would be challenging for sighted people to recognize and classify those emotions by relying on facial expressions only. A similar case was also found in the literature. For instance, Russell and Fehr (1987) argued that the same facial expression could be seen as expressing different types and degrees of emotions, for example, a neutral face could seem sad when presented alongside a happier face. According to Carroll and Russell (1996), a facial expression can provide information relevant to an emotion; however, it may not signal a specific emotion. Hence, they argued that specific situational information could contribute to determining emotions. In this study, we did not ask the participants to report what situational information (i.e., their personal emotional experiences and memories) they recalled when making facial expressions. Their memories might have affected them differently while making facial expressions. Further research is needed to examine the relationship between situational information and facial expressions of those with visual impairment and blindness. Additional emotional cues (e.g., vocal expressions) may be helpful in recognizing emotions more accurately. L. F. Barrett (2006) suggested that facial muscles convey basic affective information, but more specific emotions are inferred through additional accessible conceptual contexts such as words. For example, Kessous et al. (2010) used a set of facial, gesture, and acoustic data to extract features relevant to a particular emotion. They reported that the multimodal approach increased the recognition rate by more than 10% as compared with the unimodal approach.

AUs between neutral and non-neutral emotions

It was observed that the participants expressed a neutral emotion with non-neutral facial expressions, while they also expressed non-neutral emotions with a neutral facial expression. It was interesting that over 50% of the participants repeatedly used the single AU # 0 (i.e., neutral facial expression) in expressing neutral and other non-neutral emotions such as anger, disgust, calmness, and sadness. Although the neutral facial expression is typically considered as one that does not involve significant facial muscle movements, the participants in this study used it for expressing various emotions. From a theoretical point of view, it implies that the participants’ neutral emotion is not located at the center of the Circumplex Model of Affect (i.e., at the intersection between the positive/negative dimension and the high/low arousal dimension) (Russell, 1980). Yet, Katsikitis (1997) and Schlosberg (1952) argued that a neutral expression is located at the center of the Circumplex Model of Affect. On the contrary, there are other researchers (Galati et al., 1997; Russell & Bullock, 1985; Shah & Lewis, 2003) who shared the same view with the present study as they observed that a neutral expression was often found to be located at the periphery in the Circumplex Model of Affect. Galati et al. (1997) particularly examined people who were totally blind since birth and observed that the participants tended to categorize a neutral emotional stimulus as negative emotions, which is not located at the center of the Circumplex Model of Affect. However, it should be noted that the experimental condition between Galati et al. (1997) and the present study is not identical because the present study included participants who had various degrees of vision loss (ranging from visual impairment to blindness with no light perception) and different onset times of vision loss (ranging from early to late onset). Our future research will, thus, compare their neutral emotion expressions under the same vision condition.

AUs between valence and arousal emotions

The participants used a different set of AUs in expressing emotions depending on valence factors (positive, negative, and neutral). It suggests that the Circumplex Model of Affect (Russell, 1980) associated with valence factors are applicable to everyone regardless of vision status, including normal vision, visual impairment, and blindness. AUs were also compared by arousal factors (high, low, and neutral). Significant differences were found between high and low arousal emotions as well as high and neutral arousal emotions. However, there was no significant difference between low and neutral arousal emotions. It suggests that facial expressions of people with visual impairment and blindness would appear to be similar between “when they feel sad or calm (i.e., low arousal)” and “when they feel neutral (i.e., neutral arousal).”

We further examined the multidimensions of emotions via a combination of arousal and valence factors. When different factors were exclusively combined (e.g., between high-arousal positive and low-arousal negative emotions; between high-arousal negative and low-arousal positive emotions), a significant difference was found in the AUs. On the contrary, the combinations that were not exclusively different did not show a significant difference in the AUs, except for the combination of low-arousal positive and low-arousal negative emotions (see Figure 3). It suggests that it would be challenging for a sighted person to distinguish a visually impaired person’s facial expressions with the non-exclusive combinations of arousal and valence factors, for example, between High + Positive (happiness) and High + Negative (anger, fear, disgust, and surprise); between Low + Negative (sadness) and High + Negative (anger, fear, disgust, and surprise); between High + Positive (happiness) and Low + Positive (calmness).

Figure 3.

Non-significantly different facial expressions of emotions (that are categorized by non-exclusive combinations of valence and arousal factors) mapped onto Russell’s Circumplex Model of Affect (Russell, 1980).

The multidimensional analysis suggests that a single modality-based approach would not be effective in accurately recognizing the emotions of people with visual impairment and blindness. As addressed above, a variety of emotional cues (e.g., facial, gesture, and acoustic expressions of emotions) should be incorporated to accurately classify the multifarious emotions of people with visual impairment and blindness.

Limitations

A few limitations might have affected this study. The participants may already have perceived certain strong emotions (e.g., bad mood) due to personal reasons before participating in this study, which could affect their facial expressions while participating in this study. The participants may be those who typically do not (or are less likely to) show significant facial expressions regardless of the type and intensity of emotional stimulus, which is considered as the flat affect (Duschinsky & Wilson, 2015)—that is, total or near absence of appropriate emotional responses (e.g., facial expressions) to situations or events. They may have felt uncomfortable with showing their facial expressions to other persons although it was for research purposes. This study was conducted amid the COVID-19 pandemic such that the participants might have emotionally been affected by social distancing and stay-at-home practices. A future study with a larger sample size will be designed to address those limitations and compare the research results before and after the pandemic.

Conclusion

This study addresses the knowledge gap, that is, little is known about voluntary facial expressions of emotions in people with visual impairment and blindness while interacting with others in cyberspace. It was found that the participants relied on a particular set of facial muscle movements (AUs) in expressing anger, neutrality, fear, disgust, happiness, calmness, and surprise. Yet, this study found variations in that some emotions were expressed with statistically non-significantly different AUs; thus, facial expressions of those emotions are likely to appear similar. Neutral emotions were often displayed with non-neutral facial expressions and vice versa. When emotions were exclusively categorized by arousal (high/low) and valence (positive/negative) factors, significantly different AUs were engaged in expressing those emotions. On the contrary, it suggests that emotions with a non-exclusive combination of arousal and valence factors would appear to be similar.

As there have been merely a handful of published reports, the research findings of this study are expected to be widely beneficial to many researchers and professionals in the area of human emotion-sensing technologies and facial recognition systems. A future study will expand the scope of research to obtain a deeper understanding; for example, it will compare AUs of neutral emotion expressions under the same vision condition associated with vision loss (ranging from residual vision to total blindness) and onset time (ranging from early to late onset). A future study will also examine the relationship between facial expressions and other emotional cues (e.g., body gestures and acoustic expressions of emotions) that are presented in cyberspace. This study included a combination of users and non-users of Zoom, such that our future research will separate the two groups and examine their facial expressions online and offline.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This material is based on work supported by the National Science Foundation under Grant No. 1831969.

ORCID iD

Hyung Nam Kim

References

Adolphs

Damasio

Tranel

(2002). Neural systems for recognition of emotional prosody: A 3-D lesion study. Emotion, 2(1), 23.

Aviezer

Trope

Todorov

(2012). Body cues, not facial expressions, discriminate between intense positive and negative emotions. Science, 338(6111), 1225–1229.

Barrett

L. F.

(2006). Solving the emotion paradox: Categorization and the experience of emotion. Personality and Social Psychology Review, 10(1), 20–46.

Barrett

L. F.

Adolphs

Marsella

Martinez

A. M.

Pollak

S. D.

(2019). Emotional expressions reconsidered: Challenges to inferring emotion from human facial movements. Psychological Science in the Public Interest, 20(1), 1–68.

Barrett

L. F.

Russell

J. A.

(1999). The structure of current affect: Controversies and emerging consensus. Current Directions in Psychological Science, 8(1), 10–14.

Barrett

(2016). A new psychology of human well-being: An exploration of the influence of ego-soul dynamics on mental and physical health. Lulu.com.

Behere

R. V.

(2015). Facial emotion recognition deficits: The new face of schizophrenia. Indian Journal of Psychiatry, 57(3), 229.

Benitez-Quiroz

Srinivasan

Martinez

A. M.

(2016). Emotionet: An accurate, real-time algorithm for the automatic annotation of a million facial expressions in the wild. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. https://cbcsl.ece.ohio-state.edu/cvpr16.pdf

Bian

Zhang

Yang

(2019). Spontaneous facial expression database for academic emotion inference in online learning. IET Computer Vision, 13(3), 329–337.

10.

Biel

J.-I.

Teijeiro-Mosquera

Gatica-Perez

(2012). Facetube: Predicting personality from facial expressions of emotion in online conversational video. Proceedings of the 14Th ACM International Conference on Multimodal Interaction, Santa Monica, California. https://www.idiap.ch/~gatica/publications/BielTeijeiroGatica-icmi12.pdf

11.

Bosch

D’Mello

S. K.

Baker

R. S.

Ocumpaugh

Shute

Ventura

Wang

Zhao

(2016). Detecting student emotions in computer-enabled classrooms. IJCAI’16: Proceedings of the Twenty-fifth International Joint Conference on Artificial Intelligence, New York. https://www.ijcai.org/Proceedings/16/Papers/615.pdf

12.

Buck

(1980). Nonverbal behavior and the theory of emotion: The facial feedback hypothesis. Journal of Personality and Social Psychology, 38(5), 811.

13.

Cabanac

(2002). What is emotion? Behavioural Processes, 60(2), 69–83.

14.

Cannon

W. B.

(1927). The James-Lange theory of emotions: A critical examination and an alternative theory. The American Journal of Psychology, 39(1–4), 106–124.

15.

Carroll

J. M.

Russell

J. A.

(1996). Do facial expressions signal specific emotions? Judging emotion from the face in context. Journal of Personality and Social Psychology, 70(2), 205.

16.

Cohn

J. F.

Ambadar

Ekman

(2007). Observer-based measurement of facial expression with the Facial Action Coding System. The Handbook of Emotion Elicitation and Assessment, 1(3), 203–221.

17.

Cole

P. M.

Jenkins

P. A.

Shott

C. T.

(1989). Spontaneous expressive control in blind and sighted children. Child Development, 60, 683–688.

18.

Cordaro

D. T.

Sun

Keltner

Kamble

Huddar

McNeil

(2018). Universals and cultural variations in 22 emotional expressions across five cultures. Emotion, 18(1), 75.

19.

Dimberg

Andréasson

Thunberg

(2011). Emotional empathy and facial reactions to facial expressions. Journal of Psychophysiology, 25(1), 26.

20.

Duschinsky

Wilson

(2015). Flat affect, joyful politics and enthralled attachments: Engaging with the work of Lauren Berlant. International Journal of Politics, Culture, and Society, 28(3), 179–190.

21.

Ekman

(1993). Facial expression and emotion. American Psychologist, 48(4), 384–392.

22.

Ekman

Friesen

(1978). Facial action coding system: A technique for the measurement of facial movement. Consulting Psychologists Press.

23.

Ekman

Friesen

W. V.

(1976). Measuring facial movement. Environmental Psychology and Nonverbal Behavior, 1(1), 56–75.

24.

Fiske

S. T.

Taylor

S. E.

(2013). Social cognition: From brains to culture. SAGE.

25.

Galati

Scherer

K. R.

Ricci-Bitti

P. E.

(1997). Voluntary facial expression of emotion: Comparing congenitally blind with normally sighted encoders. Journal of Personality and Social Psychology, 73(6), 1363–1379.

26.

Galati

Sini

Schmidt

Tinti

(2003). Spontaneous facial expressions in congenitally blind and sighted children aged 8–11. Journal of Visual Impairment & Blindness, 97(7), 418–428.

27.

Grafsgaard

J. F.

Wiggins

J. B.

Boyer

K. E.

Wiebe

E. N.

Lester

J. C.

(2013). Automatically recognizing facial indicators of frustration: A learning-centric analysis. 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction, Geneva, Switzerland. https://www.intellimedia.ncsu.edu/wp-content/uploads/grafsgaard-edm-2013.pdf

28.

Greenhalgh

Vijayaraghavan

Wherton

Shaw

Byrne

Campbell-Richards

Bhattacharya

Hanson

Ramoutar

Gutteridge

(2016). Virtual online consultations: Advantages and limitations (VOCAL) study. BMJ Open, 6(1), e009388.

29.

Hersh

Leporini

Buzzi

(2020). Accessibility evaluation of video conferencing tools to support disabled people in distance teaching, meetings and other activities. ICCHP.

30.

IBM Corp. (2016). IBM SPSS statistics for Macintosh (Version 24.0).

31.

Iverson

J. M.

Goldin-Meadow

(1998). Why people gesture when they speak. Nature, 396(6708), 228–228.

32.

Kang

Derva

Kwon

D.-Y.

Wallraven

(2019). Voluntary and spontaneous facial mimicry toward other’s emotional expression in patients with Parkinson’s disease. PLOS ONE, 14(4), e0214957. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6459535/pdf/pone.0214957.pdf

33.

Katsikitis

(1997). The classification of facial expressions of emotion: A multidimensional-scaling approach. Perception, 26(5), 613–626.

34.

Keltner

Sauter

Tracy

Cowen

(2019). Emotional expression: Advances in basic emotion theory. Journal of Nonverbal Behavior, 43(2), 133–160.

35.

Kessous

Castellano

Caridakis

(2010). Multimodal emotion recognition in speech-based interaction using facial expression, body gesture and acoustic analysis. Journal on Multimodal User Interfaces, 3(1), 33–48.

36.

Kim

H. N.

(2022). The frequency of facial action units engaged in expressing emotions in people with visual disabilities via cloud-based video communication. Theoretical Issues in Ergonomics Science. Advance online publication. https://doi.org/10.1080/1463922X.2022.2081374

37.

Kim

H. N.

Sutharson

(2022). Individual differences in spontaneous facial expressions in people with visual impairment and blindness. British Journal of Visual Impairment. Advance online publication. https://doi.org/10.1177/02646196211070927

38.

Kim

Park

(2017). Does facial expression matter even online? An empirical analysis of facial expression of emotion and crowdfunding success. https://aisel.aisnet.org/icis2017/EBusiness/Presentations/1/

39.

Kohler

C. G.

Turner

Stolar

N. M.

Bilker

W. B.

Brensinger

C. M.

Gur

R. E.

Gur

R. C.

(2004). Differences in facial expressions of four universal emotions. Psychiatry Research, 128(3), 235–244.

40.

Kunz

Faltermeier

Lautenbacher

(2012). Impact of visual learning on facial expressions of physical distress: A study on voluntary and evoked expressions of pain in congenitally blind and sighted individuals. Biological Psychology, 89(2), 467–476.

41.

Lang

P. J.

Bradley

M. M.

Cuthbert

B. N.

(1997). Motivated attention: Affect, activation, and action. Attention and Orienting: Sensory and Motivational Processes, 97, 135.

42.

Lazarus

R. S.

Averill

J. R.

Opton

E. M

Jr . (1970). Towards a cognitive theory of emotion. In Arnold

(Ed.), Feelings and emotions (pp. 207–232). Elsevier.

43.

Leclerc

B.-S.

Dassa

(2009). Interrater reliability in content analysis of healthcare service quality using Montreal’s conceptual framework. The Canadian Journal of Program Evaluation, 24(2), 81–102.

44.

Leporini

Buzzi

Hersh

(2021). Distance meetings during the covid-19 pandemic: Are video conferencing tools accessible for blind people? In Proceedings of the 18th international web for all conference. https://dl.acm.org/doi/10.1145/3430263.3452433

45.

Linnenbrink

E. A.

(2006). Emotion research in education: Theoretical and methodological perspectives on the integration of affect, motivation, and cognition. Educational Psychology Review, 18(4), 307–314.

46.

Linnenbrink-Garcia

Pekrun

(2011). Students’ emotions and academic engagement: Introduction to the special issue. Contemporary Educational Psychology, 36(1), 1–3.

47.

Mann

Turner

Salisbury

(2021). The impact of remote consultations on personalised care. https://arc-w.nihr.ac.uk/Wordpress/wp-content/uploads/2021/08/Remote-consultation-briefing-website-final.pdf

48.

Matsumoto

Keltner

Shiota

M. N.

O’Sullivan

Frank

(2008). Facial expressions of emotion. In Lewis

Haviland-Jones

J. M.

Barrett

L. F.

(Eds.), Handbook of emotions (pp. 211–234). The Guilford Press.

49.

Matsumoto

Willingham

(2009). Spontaneous facial expressions of emotion of congenitally and noncongenitally blind individuals. Journal of Personality and Social Psychology, 96(1), 1–10.

50.

Meltzoff

A. N.

Moore

M. K.

(1983). Newborn infants imitate adult facial gestures. Child Development, 54, 702–709.

51.

Mesquita

Frijda

N. H.

(1992). Cultural variations in emotions: A review. Psychological Bulletin, 112(2), 179.

52.

Mujica-Parodi

L. R.

Strey

H. H.

Frederick

Savoy

Cox

Botanov

Tolkunov

Rubin

Weber

(2009). Chemosensory cues to conspecific emotional stress activate amygdala in humans. PLOS ONE, 4(7), e6415. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2713432/pdf/pone.0006415.pdf

53.

Myers

D. G.

(2004). Theories of emotion. Psychology (7th ed.). Worth Publishers.

54.

Pekrun

Linnenbrink-Garcia

(2012). Academic emotions and student engagement. In Christenson

Reschly

Wylie

(Eds.), Handbook of research on student engagement (pp. 259–282). Springer.

55.

Peleg

Katzir

Peleg

Kamara

Brodsky

Hel-Or

Keren

Nevo

(2009). Facial expressions in various emotional states in congenitally blind and sighted subjects. Israel Journal of Ecology & Evolution, 55(1), 11–30.

56.

Pell

M. D.

Monetta

Paulmann

Kotz

S. A.

(2009). Recognizing emotions in a foreign language. Journal of Nonverbal Behavior, 33(2), 107–120.

57.

Riggio

R. E.

Riggio

H. R.

(2012). Face and body in motion: Nonverbal communication. In Cash

T. F.

(Ed.), Encyclopedia of body image and human appearance (pp. 425–430). Elsevier.

58.

Russell

J. A.

(1980). A circumplex model of affect. Journal of Personality and Social Psychology, 39(6), 1161–1178.

59.

Russell

J. A.

(2003). Core affect and the psychological construction of emotion. Psychological Review, 110(1), 145.

60.

Russell

J. A.

Bullock

(1985). Multidimensional scaling of emotional facial expressions: Similarity from preschoolers to adults. Journal of Personality and Social Psychology, 48(5), 1290–1298.

61.

Russell

J. A.

Fehr

(1987). Relativity in the perception of emotion in facial expressions. Journal of Experimental Psychology: General, 116(3), 223.

62.

Sauter

D. A.

Eisner

Ekman

Scott

S. K.

(2010). Cross-cultural recognition of basic emotions through nonverbal emotional vocalizations. Proceedings of the National Academy of Sciences, 107(6), 2408–2412.

63.

Schlosberg

(1952). The description of facial expressions in terms of two dimensions. Journal of Experimental Psychology, 44(4), 229.

64.

Shah

Lewis

(2003). Locating the neutral expression in the facial-emotion space. Visual Cognition, 10(5), 549–566.

65.

Söderkvist

Ohlén

Dimberg

(2018). How the experience of emotion is modulated by facial feedback. Journal of Nonverbal Behavior, 42(1), 129–151.

66.

Song

(2021). Facial expression emotion recognition model integrating philosophy and machine learning theory. Frontiers in Psychology, 12, 759485.

67.

Sun

Y.-J.

Huang

Y.-M.

(2017). Using facial expression to detect emotion in e-learning system: A deep learning method. In Huang

T. C.

Lau

Huang

Y. M.

Spaniol

Yuen

C. H.

(Eds.), International symposium on emerging technologies for education (pp. 446–454). Springer.

68.

Thompson

(1941). Development of facial expression of emotion in blind and seeing children. Archives of Psychology, 264(37), 1–47.

69.

Tian

Y.-L.

Kanade

Colin

J. F.

(2002). Recognizing action units for facial expression analysis. In Multimodal interface for human-machine communication (pp. 32–66), World Scientific. https://www.ri.cmu.edu/pub_files/pub2/tian_ying_li_1999_2/tian_ying_li_1999_2.pdf

70.

Vail

A. K.

Grafsgaard

J. F.

Boyer

K. E.

Wiebe

E. N.

Lester

J. C.

(2016). Predicting learning from student affective response to tutor questions. In Micarelli

Stamper

Panourgia

(Eds.), ITS 2016: Proceedings of the 13th international conference on intelligent tutoring systems (pp. 154–164). Springer.

71.

Valente

Theurel

Gentaz

(2018). The role of visual experience in the production of emotional facial expressions by blind people: A review. Psychonomic Bulletin & Review, 25(2), 483–497.

72.

Voss

Gougoux

Guillemot

(2004). Early-and late-onset blind individuals show supra-normal auditory abilities in far-space. Current Biology, 14(19), 1734–1738.

73.

Whitehill

Serpell

Lin

Y.-C.

Foster

Movellan

J. R.

(2014). The faces of engagement: Automatic recognition of student engagement from facial expressions. IEEE Transactions on Affective Computing, 5(1), 86–98.

74.

World Health Organization. (2008). Change the definition of blindness. https://www.who.int/health-topics/blindness-and-vision-loss

75.

Zhan

Ogunbona

P. O.

Safaei

(2006). Facial expression recognition for multiplayer online games. https://ro.uow.edu.au/cgi/viewcontent.cgi?article=2355&context=infopapers