Culture Reveals a Flexible System for Face Processing

Face Recognition

Cultural diversity has been documented across a variety of perceptual tasks and paradigms (Nisbett & Miyamoto, 2005). People from European and American cultures focus on salient objects and use analytical rules for categorizing and organizing the environment. By contrast, people from China, Korea, and Japan—all East Asian cultures—focus more globally on contextual relationships and similarities among objects when organizing the environment. Our work has contributed to this body of evidence by demonstrating that social experience and environmental factors also shape face processing.

Since the seminal work of Yarbus (1967), eye-movement studies have persistently revealed a systematic local triangular sequence of fixations over the eyes and the mouth, predominantly to the eyes, suggesting that the presence of a face triggers a universal, biologically determined information-extraction pattern. Recently, it has been demonstrated that this triangular fixation pattern is an artifact caused by averaging eye-movement patterns across individuals (Arizpe, Walsh, Yovel, & Baker, 2016; Mehoudar, Arizpe, Baker, & Yovel, 2014). Importantly, the recent observation that similar fixation strategies are deployed in laboratory and ecological settings (Peterson, Lin, Zaun, & Kanwisher, 2016) emphasizes the reliability and robustness of observers’ eye-movement strategies.

Note, however, that the aforementioned research involved observations made with adults from Western cultures only. My colleagues and I controlled for culture and used an old/new face-recognition task requiring the memorization and subsequent recognition of previously unfamiliar faces. Importantly, in our paradigm, we used different images of Western and Eastern faces between the encoding and recognition stages to prevent participants from using simple image-matching strategies. Moreover, we randomized the location at which the stimuli appeared on the screen to avoid anticipatory strategies on initial fixations. This task more closely parallels ecological constraints and requires more than the two fixations typical of face-matching tasks using identical stimuli between encoding and recognition, with a design allowing anticipatory strategies (Hsiao & Cottrell, 2009; Or, Peterson, & Eckstein, 2015). Using our experimental design, we found that observers required 1.5 seconds and five saccades to achieve accurate face recognition, and the results also replicated the well-established triangular fixation pattern (see Fig. 1a, left). However, in stark contrast to adults from Western societies, individuals from Eastern cultural backgrounds predominantly adopted a global strategy, fixating more at the center of the face (see Fig. 1a, right, and 1b; Blais, Jack, Scheepers, Fiset, & Caldara, 2008). This culturally determined perceptual difference has been corroborated by other research groups (e.g., Chuk, Chan, & Hsiao, 2014; Kita et al., 2010; Watanabe, Matsuda, Nishioka, & Namatame, 2011), persists with inverted faces (Rodger, Kelly, Blais, & Caldara, 2010), and generalizes to visually homogenous objects (Kelly, Miellet, & Caldara, 2010). Interestingly, Malaysian observers living in a multicultural society showed a combined strategy (Tan, Stephen, Whitehead, & Sheppard, 2012), whereas British-born Chinese adults mostly deployed an Eastern eye-movement strategy (Kelly, Jack, et al., 2011). Before attending school, these adults were immersed in an Eastern culture, which suggests that early cultural exposure played a critical role in shaping their visual mechanisms dedicated to face processing. This hypothesis was confirmed by a series of studies reporting a similar cultural contrast in the fixation patterns of Eastern (Liu et al., 2011) and Western infants (Wheeler et al., 2011), as well as 7- to 8-year-old children (Kelly, Liu, et al., 2011).

OPEN IN VIEWER

Fig. 1.

Fixation patterns reflecting cultural differences in face recognition. Panel (a) shows z-scored fixation maps of Western Caucasian and East Asian observers, calculated independently for each face race. Panel (b) illustrates the fixation biases of Western Caucasian and East Asian observers, which have been highlighted by subtracting the Western Caucasian and East Asian z-scored fixation distribution maps during face encoding. Areas showing a significant fixation bias are delimited by white borders. Panel (c) reconstructs the information sampled by the eye movements using a retina filter from gaze-contingent data. Note the differences in spatial frequencies used: Westerners used local high-spatial-frequency information sampling of the eyes and the mouth, whereas Easterners used global low-spatial-frequency information from those features. Panel (b) is adapted from Figure 2 of “Culture Shapes How We Look at Faces,” by C. Blais, R. E. Jack, C. Scheepers, D. Fiset, and R. Caldara, 2008, PLoS ONE, 3(8), e3022. Copyright 2008 by the authors. Panel (c) was adapted from Figure 5 of “Mapping Face Recognition Information Use Across Cultures,” by S. Miellet, L. Vizioli, L. He, X. Zhou, and R. Caldara, 2013, Frontiers in Psychology, 4, Article 34. Copyright 2013 by the authors.

However, eye movements in natural viewing conditions do not provide unequivocal evidence on the visual information used by observers, given that diagnostic information can be extracted without focal fixations. To address this issue systematically, my colleagues and I used a series of gaze-contingent paradigms in which we restricted the available information by continuously updating the stimulus display as a function of the observers’ current gaze position. Our data showed that Westerners used local high-spatial-frequency information sampling, covering all the features critical for effective face recognition. In contrast, Easterners achieved a similar result by using global low-spatial-frequency information from those facial features (see Fig. 1c; Caldara, Zhou, & Miellet, 2010; Miellet, He, Zhou, Lao, & Caldara, 2012; Miellet, Vizioli, He, Zhou, & Caldara, 2013). This observation was recently corroborated by a psychophysical study showing that during face recognition, Eastern observers showed a greater tuning toward coarse facial features than did Western observers, despite having comparable visual sensitivity in distinguishing between increasingly finer increments of light versus dark (Tardif et al., 2016).

These differences across cultures accord with findings from a series of studies reporting reliable differences between individuals in their average fixation patterns (Arizpe et al., 2016; Kanan, Bseiso, Ray, Hsiao, & Cottrell, 2015; Mehoudar et al., 2014) and strategies, which alternated from local to global during the recognition of the very same identity, even within the same Western observers (Miellet, Caldara, & Schyns, 2011). The location of the first fixation, and more specifically its distance from the eyes, seems to critically determine the exhibited perceptual strategy (Arizpe, Kravitz, Yovel, & Baker, 2012; Miellet et al., 2011).

Altogether, our data show that the face system flexibly engages in culturally modulated eye-movement strategies by relying on distinct facial information. Given that both Western and Eastern observers achieve face recognition with comparable efficiency despite sampling different information, these observations challenge the widely accepted view of a mandatory, universal process for face recognition, whether configural or holistic.

Facial Expressions of Emotion

Central to the regulation of human social interactions is the mutual understanding of emotion. This is primarily achieved by the communication of a set of potent signals: facial expressions of emotion. With biological and evolutionary origins, the basic facial expressions of emotions (i.e., happiness, surprise, fear, disgust, anger, and sadness) have long been considered universal. In the past years, however, my research group has challenged this widely held assumption by investigating whether the decoding of emotional signals is achieved universally across human beings.

In a first study, we monitored the eye movements of Western and Eastern observers while they were recognizing facial expressions of Western and Eastern faces (see Fig. 2a; Jack, Blais, Scheepers, Schyns, & Caldara, 2009). Whereas Westerners sampled all facial features to perform this task, Easterners persistently fixated the eye region more than the mouth and experienced significantly greater confusion in emotion decoding (e.g., between fear and surprise or anger and disgust). By using reverse correlation techniques, we then demonstrated that the information used to transmit and decode emotional signals differed between Western and Eastern adults (Jack, Caldara, & Schyns, 2012; Jack, Garrod, Yu, Caldara, & Schyns, 2012). Specifically, the mouth was more informative for transmitting emotional signals in Westerners, and the eye region was more informative for transmitting emotional signals in Easterners, with cultural accents (Elfenbein & Ambady, 2003) emerging at the later stages of the facial-expression dynamics (Jack, Garrod, & Schyns, 2014). This difference resonates with the use of emoticons across cultures, whereby Easterners code transitions of expressions from happy to sad with the eyes and Westerners do so with the mouth (see Fig. 2b; Yuki, Maddux, & Masuda, 2007). Finally, my collaborators and I recently reported that such cultural differences are already present in 7-month-old infants (see Fig. 2c; Geangu et al., 2016). Altogether, these findings demonstrate that from an early stage in life, culture shapes the development of perceptual strategies used to process facial expressions of emotion.

OPEN IN VIEWER

Fig. 2.

Fixation patterns and emoticons reflecting cultural differences in emotion-expression recognition and transmission, respectively. Panel (a) shows Western Caucasian and East Asian fixation distributions averaged across the six basic emotion expressions (happiness, surprise, fear, disgust, anger, and sadness) plus a neutral expression (Jack, Blais, Scheepers, Schyns, & Caldara, 2009). Note that Eastern observers did not fixate the mouth even for the happy expression. Panel (b) illustrates the distinctly different emoticons used by each culture to convey expressions of emotion (here, happy and sad, respectively), which accord with these observations. Panel (c) shows a similar pattern of fixation distributions observed in Western Caucasian and East Asian 7-month-old infants (Geangu et al., 2016). Panel (a) is adapted from Figure 1a of “Cultural Confusions Show That Facial Expressions Are Not Universal,” by R. E. Jack, C. Blais, C. Scheepers, P. G. Schyns, and R. Caldara, 2009, Current Biology, 19, p. 1544. Copyright 2009 by Elsevier. Panel (c) is adapted from “Culture Shapes 7-month-olds’ Perceptual Strategies in Discriminating Facial Expressions of Emotion,” by E. Geangu, H. Ichikawa, J. Lao, S. Kanazawa, M. K. Yamaguchi, R. Caldara, and C. Turati, 2016, Current Biology, 26, p. R663. Copyright 2016 by Elsevier.

Conclusions and Future Directions

We and others have shown that from an early stage in life, culture shapes the eye-movement strategies deployed by humans to recognize identities and facial expressions of emotion. The face system relies on culture-specific strategies, such that observers have idiosyncratic preferences (Arizpe et al., 2016; Kanan, Bseiso, Ray, Hsiao, & Cottrell, 2015; Mehoudar et al., 2014), which can flexibly shift from one strategy to another as a function of task constraints and the landing of the first fixation location (Miellet et al., 2011). These observations do not only emphasize the need to acknowledge and understand individual differences in visual (neuro)science; importantly, they also call into question the existence of universal mechanisms for face processing. Interestingly, these fine-grained perceptual cultural contrasts resonate with and find confirmation in cross-cultural macrosocietal observations. The results for face recognition offer an empirical explanation for culturally stereotypical facial-identity descriptions offered for other-race faces. Westerners describe Easterners as having “slanted eyes,” whereas Easterners describe Westerners as having “big noses”—referring to precisely the facial features that are sampled comparatively more within the respective cultures. The facial features sampled more during the recognition of expressions of emotion are also mirrored in culture-dependent differences between emoticons (see Fig. 2b).

Despite the solidity of the observations outlined here, the cultural roots and the neural bases determining these perceptual differences are still unclear. The social norm of not directly gazing at the eyes typical of Eastern cultures cannot straightforwardly account for the results in face recognition, since the central fixation bias persists for visually homogenous non-face objects (Kelly et al., 2010). It is worth noting that this central fixation bias is not present anymore during the decoding of facial expressions, because this task requires direct fixations toward the eye region for the accurate categorization of the conveyed emotions. In addition, Easterners are culturally encouraged to suppress their emotions. Given that the muscles of the mouth are easier to control than those of the eyes, Easterners might focus more on the eyes to decode more subtle expressions from this face region. Future studies are necessary to clarify whether these two social norms learned early in life are directly responsible for shaping the perceptual strategies observed in adults.

Regardless of those potential explanations, further studies are also necessary to test additional cultures and to relate observed eye-movement differences to those obtained in other visual tasks (e.g., Miellet, Zhou, He, Rodger, & Caldara, 2010), as well as to test the fine-grained contribution of attention in our observations with more appropriate task designs (e.g., change-blindness paradigms, binocular rivalry). It also remains to be clarified whether these cultural perceptual strategies can be abolished through holistic or analytical cognitive priming and whether the organization of societies (i.e., collectivistic vs. individualistic) is responsible for these perceptual biases. It is also necessary to understand whether such preferential cultural fixation biases can be accounted for by differences in diagnostic facial information across races. This would require the use of large 3D face databases in order to clarify whether eye movements deployed for processing same- or other-race faces are (or are not) at the core of the other-race effect (e.g., Caldara & Abdi, 2006), a well-known impairment in face recognition that we have observed exclusively at the behavioral level in our eye-movement studies. Further, it is well established that language shapes cognition, and it remains to be determined whether and how language could also shape such cultural perceptual strategies. Finally, the most crucial question that remains to be addressed is whether the differences reported by my lab so far are genuinely related to culture or related to genes, a factor that has been confounded in the large majority of cross-cultural studies. This limitation can be addressed only by testing adoptees raised in another culture.

Importantly, regardless of these open questions, the marked behavioral diversity across human beings outlined here forces us to reconsider the very nature of perception and the determinants that shape the way we see the world. In the words of Anaïs Nin, “We do not see things as they are, we see them as we are” (1961, p. 124).