Is ☺ Smiling? Cross-Cultural Study on Recognition of Emoticon’s Emotion

Abstract

Emoticons are getting more popular as the new communication channel to express feelings in online communication. Although familiarity to emoticons depends on cultures, how exposure matters in emotion recognition from emoticon is still open. To address this issue, we conducted a cross-cultural experimental study among Cameroon and Tanzania (hunter-gatherers, swidden farmers, pastoralists, and city dwellers) wherein people rarely experience emoticons and Japan wherein emoticons are popular. Emotional emoticons (e.g., ☺) as well as pictures of real faces were presented on a tablet device. The stimuli expressed a sad, neutral, or happy feeling. The participants rated the emotion of stimulus on a Sad–Happy Scale. We found that the emotion rating for the real faces was slightly different but similar among three cultural groups, which supported the “dialect” view of emotion recognition. Contrarily, while Japanese people were also sensitive to the emotion of emoticons, Cameroonian and Tanzanian people hardly read emotion from emoticons. These results suggested that the exposure to emoticons would shape the sensitivity to emotion recognition of emoticons, that is, ☺ does not necessarily look smiling to everyone.

Keywords

emotion perception social cognition

Facial expression is a primal communication channel to convey emotion. Cultural universality versus cultural dependency of emotion and facial expression is one of the long-lasting debates (Brown, 1991; Russell, 1994). Darwin hypothesized that facial expression had been evolutionally formed (Darwin, 1872). Similar facial expressions were shared across cultures, even across people who have not met each other, which supports Darwin’s evolutional view of facial expression (Eibl-Eibesfeldt, 2007). While the cultural universality of facial expression is proposed (Ekman & Friesen, 1971; Ekman, Sorenson, & Friesen, 1969; Susskind et al., 2008), recent studies have challenged this view by revealing cultural differences of facial expression and emotion recognition (Elfenbein & Ambady, 2002a, 2002b; Gendron, Roberson, van der Vyver, & Barrett, 2014; Jack, Blais, Scheepers, Schyns, & Caldara, 2009; Russell, 1994). However, the cultural differences in these studies were slight and hence it might be regarded as a “dialect” (Elfenbein, Beaupré, Lévesque, & Hess, 2007). Taken together, we can convey emotion via facial expression in the cross-culture situation, even though there may be subtle variation.

Most of previous studies on the emotion and facial expression have addressed the communication via real faces. Recent development of information technologies, however, have brought us a new nonverbal channel to convey emotion—emoticon. On a text-based communication, emoticons serve as a substitute for facial expression and help us to understand the magnitudes and directions of emotion (Lo, 2008; Tossell et al., 2012). There would be no doubt that emoticons are widely used among those people who have Internet access and engage in online communication (e.g., texting, Twitter, Facebook, etc.), and hence, emoticons are playing an important role in communication. This situation causes us the need to tackle a new issue on emotion: cultural universality versus cultural dependency of emoticons.

Emoticons may be culturally universal. Since the morphology of emotional emoticon reflects emotional facial expression of real faces (Lo, 2008), people might recognize emotion of emoticons in the same way in every culture, even without frequent exposure to emoticons. Since the preattentive behavioral and neural responses to emoticons were similar to those to real faces (Churches, Nicholls, Thiessen, Kohler, & Keage, 2014), emoticons might serve as a substitute for facial expression.

Emotions may be culturally dependent. Emotion recognition accompanies the in-group advantage; we can judge facial expressions of people belonging to the same culture more accurately rather than those who belonging to the other cultures (Elfenbein & Ambady, 2002a, 2003); These findings imply that experience shapes emotion recognition. Emoticons are a relatively new communication channel, and there are several cultural groups where the emoticons are still not widely used. Thus, people who experience emoticons less frequently may hardly recognize the emotion of emoticons.

Some studies examined and revealed the cultural dependency of emotion recognition using emoticons or line-drawing stimuli. For example, Eastern people are more sensitive to the shape of eye when reading out the emotion of emoticons, while Western people are sensitive to the shape of mouth (Yuki, Maddux, & Masuda, 2007). In addition, sociological studies investigated the usage of emoticons on Twitter across cultures (Park, Baek, & Cha, 2014). People within the individualistic cultures use horizontal and mouth-oriented emoticons, whereas people within the collective cultures use vertical and eye-oriented emoticons. These results imply that there are “dialects” in emoticon usage.

Thus, configuration, usage, and the style of recognition of emoticons would more or less vary across cultures. However, these studies focused on people who are expected to be familiar with emoticons. The experimental study (Yuki et al., 2007) compared Western (North America) and Eastern (Japan) people where the online communication is widely spread. The sociological study (Park et al., 2014) investigated the broader cultures but targeted Twitter users who must be familiar with emoticons. If experience shapes emotion recognition, how the emotion of emoticons is recognized by people who are not familiar with them? While this is an important piece to achieve universal transcultural communication, it has never been addressed so far.

In the present study, therefore, we investigated whether emotion of emoticons is recognized as expected way by people those who rarely experience emoticon. For this purpose, we asked Cameroonian, Tanzanian, and Japanese people to evaluate the emotion of emoticons. Although we did not directly measure the usage of emoticons, the number of Internet users per 100 people was 20.7, 5.4, and 94.3 in Cameroon, Tanzania, and Japan, respectively (World Bank data, 2015). It is worth noting that the Internet usage would even reduce in the regions far from city where we conducted experiment. Therefore, the Cameroonian and Tanzanian people would less frequently experience and use emoticons than Japanese people do. If the emotion of emoticons perceived universally, we would observe no or slight cultural dependency—dialect—if any, as in the case of real faces. On the contrary, if exposure and experience of emoticons shape perception of emoticon, Japanese people would be expected more sensitive to the emotion of emoticons than Cameroonian and Tanzanian people.

Method

Participants

The experiment was conducted in Cameroon, Tanzania, and Japan. In Cameroon, 30 participants, including Baka hunter-gatherers, Bakwele farmers of tropical forested area, and city dwellers of Yaoundé, capital, privately took part in and completed the experiment. In Tanzania, 37 participants, including Tongwe and the other swidden farmers in the woodland in western Tanzania, Sukuma pastoralists, and Ha and the other city dwellers of Kigoma, provincial capital, participated. In Japan, 24 university students in Tokyo city area participated. The Cameroonian and Tanzanian participants gave a verbal agreement to participate in the experiment, whereas the Japanese participants gave a written informed consent. The study was approved by the Ethics Committee of the University of Tokyo and run in accordance with the Declaration of Helsinki. Field survey in the Republic of Cameroon was conducted under the research permission by the Ministry of Scientific Research and Innovation, Yaoundé. Field survey in Tanzania was conducted under the research permission by the Tanzania Commission for Science and Technology (COSTECH).

Apparatus and Stimuli

We used a tablet device (Sony Xperia Z2, 1920 × 1200 pixel) to present the stimuli and to acquire participants’ response. The stimulus display consisted of a stimulus pictures (750 × 750 pixel) on the left side of the tablet screen, a slider control to acquire the participant’s rating of emotion, and a button proceeding to the next trial (Figure 1).

Figure 1.

Example of stimulus display on a tablet device.

For the real face stimuli, we selected three Asian, African, and Caucasian females from “Face Places” database (Righi, Peissig, & Tarr, 2012; see Appendix A for the list of stimuli). The persons had happy, neutral, and sad facial expressions. For the emoticon stimuli, we generated typical Western style, Japanese style, and Smiley emoticons (Figure 2). These emoticons also represented the happy, neutral, and sad emotions. Thus, 18 stimuli (3 emotions × [3 real faces + 3 emoticons]) were used in the experiment.

Figure 2.

Emotional emoticons used in the experiment.

Procedure

First, the experimenters gave a verbal instruction to the participants about the experimental task with showing the example stimulus display and demonstrating how to operate. After the participants understood the experimental task, they underwent the experiment by holding and operating the tablet by themselves. The participants rated the expressed emotion of faces on an analog scale from sad (indicated by X symbol) to happy (indicated by O symbol) using the slider control. All 18 stimuli were presented once. The participants conducted at least two practical trials with the pictures that were not used in the experiment. Then, they conducted 18 experimental trials.

We aimed at making the experimental procedure reproducible. It is worth noting here that our experimental setting and procedures were almost uniform among three cultural groups, except for some uncontrollable environments (e.g., lighting, temperature, location, etc.). We used the same apparatus (i.e., a tablet device) for all experiments. Furthermore, the tablet device enabled participants to engage in the experiment by themselves and to remove interactions with the experimenter, which minimized observer-expectancy effect.

Results

Figure 3 shows the rating score of emotion for the real faces and emoticons. The entire data apperas in Appendix B. A three-way mixed effect analysis of variance (ANOVA) of emotion of facial expression, face type (real vs. emoticon), and cultural group revealed the significant main effect of facial expression—F(2, 176) = 185.7, p < .001, $η_{p}^{2}$ = 0.68; as a general tendency, the happy faces are rated as happy, and the sad faces are rated as sad. On the contrary, the main effect of cultural group was not significant—F(2, 88) = 1.63, p = .20, $η_{p}^{2}$ = 0.04—which indicated that the baseline rating scores were comparable across three cultural groups.

Figure 3.

Mean rating scores of real faces and emoticons as a function of stimulus emotion, face type, and cultural group.

As for the cultural dependency, we observed the significant two-way interaction of cultural group × emotion—F(4, 176) = 8.72, p < .001, $η_{p}^{2}$ = 0.17—and the significant three-way interaction—F(20, 880) = 4.18, p < .001, $η_{p}^{2}$ = 0.09. As these interactions suggested that emotion recognition depended on the cultural groups and face types, we conducted two-way mixed effect ANOVAs of cultural group and emotion separately for each face type. For the real faces, we observed no interaction of cultural group × emotion—Asian: F(4, 176) = 2.21, p = .07, $η_{p}^{2}$ = 0.05; African: F(4, 176) = 0.44, p = .78, $η_{p}^{2}$ = 0.01; Caucasian: F(4, 176) = 0.34, p = .85, $η_{p}^{2}$ = 0.01. These results can be interpreted that the patterns of emotion recognition for real faces were similar across the Cameroonian, Tanzanian, and Japanese participants regardless of the race of face stimuli. Note that the Cameroonian and Tanzanian participants tended to confuse Asian neutral and sad faces, which would be perhaps due to difference of exposure to the out-group faces (Elfenbein & Ambady, 2002a).

On contrary to the real faces, we found the clear cultural differences in emotion recognition for the emoticons. We observed significant interactions of cultural group × emotion for all emoticon conditions—Western emoticon: F(4, 176) = 5.01, p < .001, $η_{p}^{2}$ = 0.10; Japanese emoticon: F(4, 176) = 18.1, p < .001, $η_{p}^{2}$ = 0.29; Smiley: F(4, 176) = 6.31, p < .001, $η_{p}^{2}$ = 0.13. With Japanese participants, the main effects of emotion was significant for all types of emoticons—Western emoticon: F(2, 46) = 67.1, p < .001, $η_{p}^{2}$ = 0.74; Japanese emoticon: F(2, 46) = 255.2, p < .001, $η_{p}^{2}$ = 0.92; Smiley: F(2, 46) = 186.9, p < .001, $η_{p}^{2}$ = 0.89. Thus, Japanese participants discriminated emoticons’ emotion in the similar way to the real faces. However, the main effects of emotion was absent in the Cameroonian participants—Western emoticon: F(2, 58) = 1.45, p = .24, $η_{p}^{2}$ = 0.05; Japanese emoticon: F(2, 58) = 1.84, p = .17, $η_{p}^{2}$ = 0.06; Smiley: F(2, 58) = 1.59, p = .21, $η_{p}^{2}$ = 0.05—as well as in the Tanzanian participants—Western emoticon: F(2, 72) = 0.02, p = .99, $η_{p}^{2}$ = 0.00; Japanese emoticon: F(2, 72) = 0.89, p = .41, $η_{p}^{2}$ = 0.02; Smiley: F(2, 72) = 3.09, p = .052, $η_{p}^{2}$ = 0.08—which suggested that emotion of the emoticons did not affect the Cameroonian and Tanzanian participants’ rating of emotion.

Discussion

The present study demonstrated that the emotion recognition of emoticon is not culturally universal but varies across cultures. Japanese people were sensitive to the emotion of emoticons to the same extent as the emotion of real faces. However, while Cameroonian and Tanzanian people also discriminate the emotion of real faces, they hardly read emotion of emoticons.

We setup the experimental stimuli, apparatus, and procedures as uniform as possible among three cultural groups. For example, we used the same tablet device and asked the participants to conduct the experiment by themselves. This manipulation minimized the possibility of the methodological confounds such as word translation or observer-expectancy effects, even though we might not able to completely rule out them. Furthermore, since the data from all cultural group demonstrated that the emotion of real faces was discriminated almost successfully, every participant seemed to understand the experimental task, how to manipulate the tablet device, and how to rate the emotion of faces on the analog scale. Thus, it is unlikely that the uncontrolled experimental setup or procedures resulted in the cultural dependency of emotion recognition from emoticons.

Before focusing on the cultural dependency of emoticon, we briefly address the out-group effect for the real faces. The patterns of emotion recognition for the real faces were generally similar but slightly different among three cultural groups. In particular, Cameroonian and Tanzanian people hardly discriminated the neutral and sad emotion of Asian faces, although the interaction was not statistically significant. Thus, our results would support the “dialect” interpretation of the emotion recognition and facial expression (Elfenbein et al., 2007).

Unlike the real faces, we found cultural dependency in the recognition patterns of emotion from the emoticons. In short, Japanese did read out the emotion from the emoticons but Cameroonian and Tanzanian people did not. One possible and simple account is that they had no idea about what the shape of emoticon indicates. It would be noteworthy that we found no difference between the city dwellers (n = 28) and other participants (n = 39) in Cameroonian and Tanzanian, while the city dwellers use communication tools such as Facebook in their daily life, and hence, they would be exposed to emoticons. As the sample was small and the familiarity to the emoticons were not investigated and controlled in the present study, we need further studies, for example, asking how familiar are they with the emoticons, to uncover the effects of experience.

This result may reflect the difference of the style of viewing emoticon—whether the emoticons are seen as signaling their internal feelings or as a symbolic expression of emotion. Emotional expressions do not necessarily signal the internal feeling states; sometimes they convey behavioral intention or action requests to others (Horstmann, 2003). This notion is evident when we consider a picture of smiling face and a word “smiling face.” Both emotion-related stimuli would be associated with happiness. However, while we read out the internal feeling from the picture of emotional face, we do not so when seeing the word stimulus.

Then, are emoticons signal or symbol? Emoticons may serve in the symbolic way in nature, since emoticons are just a pictorial representation of emotion and have no internal feelings in themselves. However, frequent experience using emoticons as an emotion expression in communication may reshape the emoticons from symbol to signal. This view would be consistent with the notion that the anthropomorphism is dynamics—specific knowledge and motivations shape anthropomorphism (Epley, Waytz, & Cacioppo, 2007). Accordingly, Japanese people are getting to suppose internal feelings for the emoticons and rated the emotion in the same way as real faces. On the contrary, Cameroonian and Tanzanian people, those who have less frequent exposure to the emoticons may regard the emoticons as a symbolic representation of emotion—like the word “smiling face.”

The signal versus symbol account is not conclusive, and hence, further investigation is needed to address the theoretical framework for our findings. In particular, we need to investigate the explicit knowledge about emoticons as well as to verify whether the levels of anthropomorphism for emoticons vary across cultures. Furthermore, even if the signal versus symbol account is correct, it is still unclear whether only the frequency of exposure to emoticons matters or other cultural factors, such as cultural practices, beliefs, and values, also influence the recognition of emotion from emoticons.

We showed that the style of emotion recognition for emoticons is not universal but varies across cultures. These days, as opportunities to use online communication are increasing, emoticons are widely used and play the more important role. It is worth noting that people use the online communication more frequently when communicating with people who are far away (Baym, Zhang, & Lin, 2004). To prevent miscommunication in the online transcultural communication, people would need to learn that emotion of emoticons may be recognized in a different way from what they expect.

Footnotes

Appendix

Appendix B

The Means and Standard Deviations of Emotion Rating.

Culture	Face type		M			SD
Culture	Face type		Happy	Neutral	Sad	Happy	Neutral	Sad
Cameroonian (N = 30)	Real face	Asian	78.80	33.13	46.30	36.30	43.56	44.73
		African	95.13	55.33	20.80	12.77	37.07	37.99
		Caucasian	85.70	56.80	38.77	34.24	42.92	39.46
	Emoticon	Western	45.23	35.27	31.67	48.91	43.14	41.75
		Eastern	28.87	33.73	42.20	40.63	41.92	42.91
		Smiley	39.60	37.13	25.17	48.82	42.27	37.77
Tanzanian (N = 37)	Real face	Asian	90.84	33.43	35.81	18.39	37.61	36.37
		African	84.97	50.46	20.14	29.07	33.76	33.53
		Caucasian	86.62	48.30	36.00	29.93	34.47	35.78
	Emoticon	Western	44.59	45.51	43.81	36.00	37.90	37.20
		Eastern	58.92	49.16	50.19	36.81	37.88	39.87
		Smiley	58.81	56.97	41.24	40.87	37.14	39.92
Japanese (N = 24)	Real face	Asian	75.33	40.63	25.67	13.86	8.85	15.51
		African	83.63	44.54	20.92	12.61	11.03	12.33
		Caucasian	82.54	49.79	39.75	13.98	14.42	12.41
	Emoticon	Western	75.46	42.79	27.04	15.63	12.50	14.29
		Eastern	85.50	42.08	10.42	10.13	11.53	9.84
		Smiley	84.92	47.63	21.00	11.86	5.66	12.76

Acknowledgements

The authors thank Dr. Michio Nakamura and Dr. Katsumi Watanabe for their helpful comments.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was supported by Japan Society for the Promotion of Science (JSPS) KAKENHI (25700013, 16H01664, and 17H06342 to K.T.; 26870297 to T.O.; and 26284138 to M.S.).

References

Baym

N. K.

Zhang

Y. B.

Lin

M.-C.

(2004). Social interactions across media. New Media & Society, 6, 299-318. doi:10.1177/1461444804041438

Brown

(1991). Human universals. New York, NY: McGraw-Hill.

Churches

Nicholls

Thiessen

Kohler

Keage

(2014). Emoticons in mind: An event-related potential study. Social Neuroscience, 9, 196-202. doi:10.1080/17470919.2013.873737

Darwin

(1872). The expression of the emotions in man and animals. London, England: John Murray.

DataBankWorld Development Indicators. (2015). Individuals using the Internet (% of population). Retrieved from http://databank.worldbank.org/data/reports.aspx?source=2&series=IT.NET.USER.ZS&country=

Eibl-Eibesfeldt

(2007). Human ethology. New Brunswick, NJ: Transaction Publishers.

Ekman

Friesen

W. V.

(1971). Constants across cultures in the face and emotion. Journal of Personality and Social Psychology, 17, 124-129.

Ekman

Sorenson

E. R.

Friesen

W. V.

(1969). Pan-cultural elements in facial displays of emotion. Science, 164, 86-88.

Elfenbein

H. A.

Ambady

(2002a). Is there an in-group advantage in emotion recognition? Psychological Bulletin, 128, 243-249. doi:10.1037//0033-2909.128.2.243

10.

Elfenbein

H. A.

Ambady

(2002b). On the universality and cultural specificity of emotion recognition: A meta-analysis. Psychological Bulletin, 128, 203-235. doi:10.1037//0033-2909.128.2.203

11.

Elfenbein

H. A.

Ambady

(2003). When familiarity breeds accuracy: Cultural exposure and facial emotion recognition. Journal of Personality and Social Psychology, 85, 276-290. doi:10.1037/0022-3514.85.2.276

12.

Elfenbein

H. A.

Beaupré

Lévesque

Hess

(2007). Toward a dialect theory: Cultural differences in the expression and recognition of posed facial expressions. Emotion, 7, 131-146. doi:10.1037/1528-3542.7.1.131

13.

Epley

Waytz

Cacioppo

J. T.

(2007). On seeing human: A three-factor theory of anthropomorphism. Psychological Review, 114, 864-886. doi:10.1037/0033-295X.114.4.864

14.

Gendron

Roberson

van der Vyver

J. M.

Barrett

L. F.

(2014). Perceptions of emotion from facial expressions are not culturally universal: Evidence from a remote culture. Emotion, 14, 251-262. doi:10.1037/a0036052

15.

Horstmann

(2003). What do facial expressions convey: Feeling states, behavioral intentions, or action requests? Emotion, 3, 150-166. doi:10.1037/1528-3542.3.2.150

16.

Jack

R. E.

Blais

Scheepers

Schyns

P. G.

Caldara

(2009). Cultural confusions show that facial expressions are not universal. Current Biology, 19, 1543-1548. doi:10.1016/j.cub.2009.07.051

17.

S.-K.

(2008). The nonverbal communication functions of emoticons in computer-mediated communication. CyberPsychology & Behavior: The Impact of the Internet, Multimedia and Virtual Reality on Behavior and Society, 11, 595-597. doi:10.1089/cpb.2007.0132

18.

Park

Baek

Y. M.

Cha

(2014). Cross-cultural comparison of nonverbal cues in emoticons on Twitter: Evidence from big data analysis. Journal of Communication, 64, 333-354. doi:10.1111/jcom.12086

19.

Righi

Peissig

J. J.

Tarr

M. J.

(2012). Recognizing disguised faces. Visual Cognition, 20, 143-169. doi:10.1080/13506285.2012.654624

20.

Russell

J. A.

(1994). Is there universal recognition of emotion from facial expression? A review of the cross-cultural studies. Psychological Bulletin, 115, 102-141.

21.

Susskind

J. M.

Lee

D. H.

Cusi

Feiman

Grabski

Anderson

A. K.

(2008). Expressing fear enhances sensory acquisition. Nature Neuroscience, 11, 843-850. doi:10.1038/nn.2138

22.

Tossell

C. C.

Kortum

Shepard

Barg-Walkow

L. H.

Rahmati

Zhong

(2012). A longitudinal study of emoticon use in text messaging from smartphones. Computers in Human Behavior, 28, 659-663. doi:10.1016/j.chb.2011.11.012

23.

Yuki

Maddux

W. W.

Masuda

(2007). Are the windows to the soul the same in the East and West? Cultural differences in using the eyes and mouth as cues to recognize emotions in Japan and the United States. Journal of Experimental Social Psychology, 43, 303-311. doi:10.1016/j.jesp.2006.02.004