The Role of Gaze in the Processing of Emotional Facial Expressions

Abstract

Gaze plays a fundamental role in the processing of facial expressions from birth. Gaze direction is a crucial part of the social signal encoded in and decoded from faces. The ability to discriminate gaze direction, already evident early in life, is essential for the development of more complex socially relevant tasks, such as joint and shared attention. At the same time, facial expressions play a fundamental role in the encoding of gaze direction and, when combined, expression and gaze communicate behavioural motivation to approach or avoid. However, the investigation of how gaze direction and emotional expression interact during the processing of a face has been relatively neglected, and is the key question of this review.

Keywords

emotion perception gaze direction infants newborns shared signal hypothesis

Recent studies have found that gaze direction influences the perceived emotion conveyed by neutral faces and have demonstrated the existence of specific combinations of facial expressions and gaze direction (Adams & Kleck, 2003, 2005). The suggestion is that emotional expressions and gaze direction interact and jointly contribute to behavioural motivations to approach or avoid. Adams and Kleck’s “shared signal” hypothesis postulates that when gaze direction is combined with the intent communicated by a specific expression, it will enhance the perception of that emotion. Specifically, joyful and angry expressions are categorized as “approach-oriented” emotions, and so are usually accompanied with direct gaze. Fearful and sad expressions are categorized as “avoidance-oriented” emotions and so are more intense when accompanied by averted gaze.

Evidence suggests that the “shared signal” hypothesis may have validity from infancy, gradually extending from approach-oriented emotions to avoidance-oriented emotions over the course of development. In addition, recent findings with newborns suggest that such particular combinations of expression and gaze might exist already at birth. In this review, we examine evidence that tested this hypothesis, and, more generally, that explored the interaction between emotional expression and gaze direction.

Behavioural Evidence

Infant Studies

Newborns can discriminate among gaze directions and prefer a neutral facial expression accompanied by direct gaze to one accompanied by averted gaze (Farroni, Csibra, Simion, & Johnson, 2002). A more recent study extended this finding to faces displaying happy and fearful emotional expressions (Rigato, Menon, Johnson, & Farroni, 2011). As suggested by the “shared signal hypothesis,” direct gaze triggered a visual preference only when presented within a happy expression. However, newborns did not show a preference for a fearful face displaying averted gaze over a nonfearful face with averted gaze. Consequently, it is reasonable to conclude that newborns are sensitive to at least some congruent eye gaze–expression combinations.

Evidence suggests a developmental shift with a gradual extension from approach-oriented emotions to avoidance-oriented emotions. Newborns are sensitive to gaze differences only for approach emotions, but adults are sensitive to gaze differences for both approach and avoidance emotions. Supportive of this gradual shift is a study with 4-month-old infants who were sensitive to both, but showed a greater sensitivity for the congruent emotion–gaze combination in the approach-oriented emotions (i.e., happy and angry with direct gaze) (Rigato, Menon, Farroni, & Johnson, 2011).

Altogether, the studies discussed earlier suggest that within the first few days after birth newborns are sensitive to characteristics of faces likely to maximize their chances of interacting with other people. Several characteristics of newborns’ visual preferences indicate that they may have been selected not only for detecting faces, but also for detecting communicative partners (Farroni et al., 2005). Various pieces of evidence support this view: (a) newborns prefer upright to inverted face configurations (Johnson, Dziurawiec, Ellis, & Morton, 1991; Valenza, Simion, Macchi Cassia, & Umiltà, 1996); (b) newborns prefer faces with direct gaze to faces with averted gaze (Farroni et al., 2002), even when schematic faces are used as stimuli (Farroni, Massaccesi, Pividori, & Johnson, 2004); (c) newborns’ preferences are restricted to stimuli composed of darker elements on lighter background (Farroni et al., 2005), which is the same contrast relation that is used in identifying gaze direction in humans (Kobayashi & Kohshima, 1997); and (d) happy emotional expressions displaying a direct gaze are the most preferred face stimuli from birth (Farroni, Menon, Rigato, & Johnson, 2007; Rigato et al., 2011). Taken together, these aspects of newborns’ preferences imply that human babies at birth are most attracted to stimuli appropriate for social and positive interaction.

Additional, and nonetheless complementary, models and explanations can also be applied to these early preferences. A happy expression accompanied by a direct gaze is not only the most likely facial expression experienced in a typical emotional environment during the first few postnatal hours, but also the stimulus that enhances the chances of interacting with other people. Importantly, in the absence of verbal communication, social interactions via facial expression may be one of the most important ways for the newborn to communicate with a caregiver, and therefore they provide an advantage in terms of survival and adaptability. Recent application of theories of embodied cognition to face recognition suggests that the newborns’ interest in eye contact could reflect an evolutionary-based mechanism for triggering embodied simulation in their caretakers (Niedenthal, Mermillod, Maringer, & Hess, 2010). In this respect, smiling is one of the most efficient ways to obtain eye contact with the adult (Blass & Camp, 2001).

Adult Studies

The interaction between expression and gaze has been studied with the Garner interference paradigm, judgment tasks, or gaze shifting tasks. The logic of the Garner paradigm is that if two stimulus dimensions are processed in an integral manner, it will be impossible to attend to one dimension and ignore the other (Graham & LaBar, 2007). For example, in one study participants were asked to make rapid classifications either of expression (happy or angry) or gaze direction (direct or averted) (Ganel, Goshen-Gottstein, & Goodale, 2005). Overall, expression judgments were slower than gaze judgments, and gaze judgments in the direct gaze condition were faster than the averted gaze condition. Further, bidirectional interference between expression and gaze was found, determining that the processing of each of the two dimensions is dependent on the other.

A similar study found an asymmetrical pattern of Garner interference, where expression interfered with gaze judgment but gaze did not interfere with expression judgment (Graham & LaBar, 2007). However, when less intense and less identifiable facial expressions were used, a symmetrical pattern of Garner interference was restored. On the basis of these findings, it is plausible to conclude that when it is hard to discriminate, the emotional expression is processed slowly, and gaze interferes with emotion judgments. Therefore, even though it is clear that judgments on these two dimensions interfere to some extent, such interference is stimulus-bound.

Recent research has also focused on the effect of facial expression on the automatic orienting cued by gaze direction. These studies employed Posner’s classic spatial-cueing paradigm (Posner, 1980), that is, in a target detection task, location cues can facilitate reaction times depending on whether or not the cue is a valid indicator of the target’s location. In the case of facial expressions, emotional faces that orient their gaze are used as the central cue to facilitate reaction times depending on whether or not the direction of the gaze is towards the target’s location or towards the opposite location.

The studies that reported an interaction between the facial expression and the congruency of the gaze direction with the target location revealed faster attention shifting in the presence of a fearful expression (Fichtenholtz, Hopfinger, Graham, Detwiler, & LaBar, 2007; Tipples, 2006) or of a happy one (Hori et al., 2005) relative to neutral faces. However, other studies did not replicate this effect (Bayliss, Frischen, Fenske, & Tipper, 2007; Hietanen & Leppänen, 2003), or found the effect only with special populations (Mathews, Fox, Yiend, & Calder, 2003). These diverse outcomes might have been due to method factors, such as the number of face identities used or the sequential versus the simultaneous modification of the dimensions of gaze direction and facial expression of the central cue. However, the diverse outcomes might also reflect substantive factors. A recent study suggests that the search target has an effect on such tasks (Kuhn & Tipples, 2011). In fact, participants were more likely to follow the gaze on a fearful face compared to a happy face when searching for a threatening target. This stronger cueing effect for fearful faces disappeared when searching for a pleasant target. Thus, it can be concluded that facial expression modulates and facilitates attention orienting by eye gaze cues only under specific conditions.

Neural Bases of Expression and Gaze Processing

Neuroimaging Studies

Two brain regions have been found to be particularly involved in the processing of gaze direction and facial expression: the amygdala and the superior temporal sulcus (STS). For example, a functional magnetic resonance imaging (fMRI) study investigated the role of the amygdala in processing threat-related ambiguity by presenting adults with angry and fearful facial displays with direct and averted gaze (Adams, Gordon, Baird, Ambady, & Kleck, 2003). Based on their previous results (Adams & Kleck, 2003), which suggested that expression and gaze are associated on the basis of their communicated intent, the authors reasoned that angry faces with averted gaze and fearful faces with direct gaze corresponded to ambiguous stimuli, and therefore they would have enhanced activity in the amygdala. The findings were consistent with such predictions and demonstrated an important interaction of expression and gaze on amygdala functioning.

However, more recent findings suggest that in emotion perception the role of the amygdala is secondary and possibly restricted to that of seeking out the eyes, that is, driving saccades towards the eye region (see Atkinson & Smithson, 2013). Accordingly, the role of the amygdala might be that of more general and abstract information processing, including processing of salient and biological significant stimuli via its broad connectivity with the cortex and other subcortical structures (Pessoa & Adolphs, 2010). In the case of Adams et al.’s (2003) study, therefore, their ambiguous stimuli might have represented salient and biological significant stimuli in order to enhance activity in the amygdala.

The STS has been found to contain dissociable representations for the perception of facial expression and gaze. For instance, an fMRI study found a greater response for emotional than for neutral faces with direct gaze in the bilateral STS, the right inferior frontal gyrus, and the bilateral occipital lobe (Engell & Haxby, 2007). A stronger activation to neutral faces with averted gaze than with direct gaze was observed in the right STS. Finally, a greater activation to emotional than to neutral faces with averted gaze was observed within the inferior occipital gyrus which extended bilaterally into the lateral fusiform gyrus.

Similarly, dissociable roles for processing emotional expression and gaze have been observed in the somatosensory and superior temporal cortices (posterior STS) respectively (Pourtois et al., 2004). Transcranial magnetic stimulation (TMS) was used to deliver single pulses over these two regions of interest to interfere selectively with the emotional and gaze judgment task. TMS over somatosensory areas interfered with emotion expression recognition, especially with the expression of fear, whereas TMS over superior temporal cortex interfered with the perception of gaze. Altogether, such findings suggest some crucial anatomical–functional segregation between these neural systems for the processing of facial expression and gaze direction. Nevertheless, it is unclear whether co-occurring dimensions of face stimuli, such as expression and direction of gaze, are processed jointly or independently by anatomically and functionally segregated neural structures.

Electrophysiological Studies

The behavioural results reported earlier correspond well with electrophysiological findings. In one study, event-related potentials (ERPs) were recorded from adults and 4-month-old infants while they watched pictures of faces that varied in emotional expression (happy and fearful) and in gaze direction (direct and averted) (Rigato, Farroni, & Johnson, 2010). In adults, the results showed an expression–gaze interaction over a posterior channel location reflected as a greater response to fearful expressions with averted gaze (avoidance-oriented emotion), and to happy faces with direct gaze (approach-oriented emotion), in line with the combinations suggested by the “shared signal” hypothesis.

Interestingly, the effects of facial expression on visual evoked potentials were observed at shorter latencies than those of gaze direction, and the interactions between the processing of emotional expressions and gaze direction appeared at longer latencies. This pattern of results could indicate an initial phase of independent processing of the two dimensions prior to the stage at which the information is integrated. Nonetheless, the main effect of gaze occurred at the same latency as its interaction with facial expression. This means that it is not possible to conclude definitively that gaze processing is independent from facial expression processing; on the contrary, it suggests that gaze is encoded in facial expressions.

In infants, an enhanced ERP response to a happy expression at the frontocentral negative component (Nc) was found, and this was due to the direct gaze condition only. This effect, according to the “shared signal” hypothesis, might reflect an early sensitivity to the approach-oriented emotions; however, it may also represent familiarity with such a stimulus (de Haan & Nelson, 1997) or allocation of attention processes (Courchesne, Ganz, & Norcia, 1981; Nelson, 1994). In another study, infants of 7 months of age showed enhanced responses of the Nc to angry faces with direct gaze, but not with averted gaze, possibly driven by the novelty of the stimulus that facilitates the allocation of attentional resources (Hoehl & Striano, 2008).

Accordingly, other ERP studies have confirmed that from an early age infants are sensitive to socially relevant stimuli and allocate greater attention to them. In one study, infants who saw angry faces with direct gaze had a larger positive slow wave (PSW) amplitude, suggesting an enhanced neural processing of these emotional displays (Striano, Kopp, Grossmann, & Reid, 2006). Increased PSW amplitude may be involved in maintaining information over a period of time, and it has been found to reflect the novelty of the stimulus (de Haan & Nelson, 1999; Nelson, 1997) and allocation of attention (Reid, Striano, Kaufman, & Johnson, 2004).

Therefore already in young infants a direct gaze not only enhances the processing of a happy expression, which is generally the most familiar to the infant, but it also enhances the processing of a less familiar angry face. This might suggest that the familiarity, and therefore the experience with the stimulus, plays a secondary role in the processing of the interaction of certain combinations of expression and gaze. However, the more general development of facial expression processing is affected by experience. For example, studies with infants and children of clinically depressed mothers (Dawson et al., 2003) revealed that an atypical emotional environment involving lower frequency of positive emotion and higher frequency of flat or negative emotion affects this process (de Haan, Belsky, Reid, Volein, & Johnson, 2004).

Conclusions

Gaze plays a fundamental role in the processing of facial expressions from birth. We have reported evidence from adults that also suggests an important role of gaze in facial expression perception and processing. The differences observed between adults and infants suggest the general conclusion that experience with a large set of emotional expressions in a great number of situations allows adults to discriminate between them, and detect the presence or absence of the congruence between the communication conveyed by the expression and the eye region. However, evidence suggests that there are specific and preferred combinations of expression and gaze already at birth, supporting the hypothesis of an early interaction between the emotional facial expressions and the gaze direction displayed in the face context.

Although these specific expression–gaze combinations are consistent with the “shared signal” hypothesis, alternative accounts can also explain them and it is often difficult to attribute the findings only to one of those. For example, it remains an open question whether newborns prefer to look at happy expressions when accompanied with direct gaze because both of these social signals share an approach motivation, or because they represent the most likely facial expression experienced in a typical emotional environment during the first few postnatal hours. In addition, such a stimulus also enhances the chances of interacting with other people and therefore represents an advantage in terms of survival and adaptability for the neonate.

Footnotes

Author note:

This work was supported by the European Research Council under the European Community’s Seventh Framework Programme (FP7/2007-2013) (ERC Grant Agreement no. 241242) and by the Universitá degli Studi di Padova.

References

Adams

R. B.

Jr. Gordon

H. L.

Baird

A. A.

Ambady

Kleck

R. E.

(2003). Effects of gaze on amygdala sensitivity to anger and fear faces. Science, 300, 1536.

Adams

R. B.

Kleck

R. E.

(2003). Perceived gaze direction and the processing of facial displays of emotion. Psychological Science, 14, 644–647.

Adams

R. B.

Kleck

R. E.

(2005). Effects of direct and averted gaze on the perception of facially communicated emotion. Emotion, 5, 3–11.

Atkinson

A. P.

Smithson

H. E.

(2013). Distinct contributions to facial emotion perception of foveated versus nonfoveated facial features. Emotion Review, 5, 30–35.

Bayliss

A. P.

Frischen

Fenske

M. J.

Tipper

S. P.

(2007). Affective evaluations of objects are influenced by observed gaze direction and emotional expression. Cognition, 104, 644–653.

Blass

E. M.

Camp

C.A.

(2001). The ontogeny of face recognition: Eye contact and sweet taste induce face preference in 9- and 12-week-old human infants. Developmental Psychology, 37, 762–774.

Courchesne

Ganz

Norcia

A. M.

(1981). Event-related potentials to human faces in infants. Child Development, 52, 804–811.

Dawson

Ashman

S. B.

Panagiotides

Hessl

Self

Yamada

Embry

(2003). Preschool outcomes of children of depressed mothers: Role of maternal behavior, contextual risk, and children’s brain activity. Child Development, 74, 1158–1175.

de Haan

Belsky

Reid

Volein

Johnson

M. H.

(2004). Maternal personality and infants’ neural and visual responsivity to facial expressions of emotion. Journal of Child Psychology and Psychiatry, 45, 1209–1218.

10.

de Haan

Nelson

C. A.

(1997). Recognition of the mother’s face by 6-month-old infants: A neurobehavioral study. Child Development, 68, 187–210.

11.

de Haan

Nelson

C. A.

(1999). Brain activity differentiates face and object processing in 6-month-old infants. Developmental Psychology, 35, 1113–1121.

12.

Engell

A. D.

Haxby

J. V.

(2007). Facial expression and gaze-direction in human superior temporal sulcus. Neuropsychologia, 45, 3234–3241.

13.

Farroni

Csibra

Simion

Johnson

M. H.

(2002). Eye contact detection in humans from birth. Proceedings of the National Academy of Sciences, USA, 99, 9602–9605.

14.

Farroni

Johnson

M. H

Menon

Zulian

Faraguna

Csibra

(2005). Newborns’ preference for face-relevant stimuli: Effects of contrast polarity. Proceedings of the National Academy of Sciences, USA, 102, 17245.

15.

Farroni

Massaccesi

Pividori

Johnson

M. H.

(2004). Gaze following in newborns. Infancy, 5, 39–60.

16.

Farroni

Menon

Rigato

Johnson

M. H.

(2007). The perception of facial expressions in newborns. European Journal of Developmental Psychology, 4, 2–13.

17.

Fichtenholtz

H. M.

Hopfinger

J. B.

Graham

Detwiler

J. M.

LaBar

K. S.

(2007). Happy and fearful emotion in cues and targets modulate event-related potential indices of gaze-directed attentional orienting. Social Cognitive and Affective Neuroscience, 2, 323–333.

18.

Ganel

Goshen-Gottstein

Goodale

M. A.

(2005). Interactions between the processing of gaze direction and facial expression. Vision Research, 45, 1191–1200.

19.

Graham

LaBar

K. S.

(2007). Garner interference reveals dependencies between emotional expression and gaze in face perception. Emotion, 7, 296–313.

20.

Hietanen

J. K.

Leppänen

J. M.

(2003). Does facial expression affect attention orienting by gaze direction cues? Journal of Experimental Psychology: Human Perception and Performance, 29, 1228–1243.

21.

Hoehl

Striano

(2008). Neural processing of eye gaze and threat-related emotional facial expressions in infancy. Child Development, 79, 1752–1760.

22.

Hori

Tazumi

Umeno

Kamachi

Kobayashi

Ono

Nishijo

(2005). Effects of facial expression on shared attention mechanisms. Physiology & Behavior, 84, 397–405.

23.

Johnson

M. H.

Dziurawiec

Ellis

H. D.

Morton

(1991). Newborns preferential tracking of face-like stimuli and its subsequent decline. Cognition, 40, 1–19.

24.

Kobayashi

Kohshima

(1997). Unique morphology of the human eye. Nature, 387, 767–768.

25.

Kuhn

Tipples

(2011). Increased gaze following for fearful faces. It depends on what you’re looking for! Psychonomic Bulletin & Review, 18, 89–95.

26.

Mathews

Fox

Yiend

Calder

(2003). The face of fear: Effects of eye gaze and emotion on visual attention. Visual Cognition, 10, 823–835.

27.

Nelson

C. A.

(1994). Neural correlates of recognition memory in the first postnatal year of life. In Dawson

Fischer

(Eds.), Human behavior and the developing brain (pp. 269–313). New York, NY: Guilford Press.

28.

Nelson

C. A.

(1997). Electrophysiological correlates of memory development in the first year of life. In Reese

H. W.

Franzen

M. D.

(Eds.), Biological and neuropsychological mechanisms. Life-span developmental psychology (pp. 95–131). Mahway, NJ: Erlbaum.

29.

Niedenthal

P. M.

Mermillod

Maringer

Hess

(2010). The Simulation of Smiles (SIMS) model: Embodied simulation and the meaning of facial expression. Behavioral and Brain Sciences, 33, 417–480.

30.

Pessoa

Adolphs

(2010). Emotion processing and the amygdala: From a “low road” to “many roads” of evaluating biological significance. Nature Reviews Neuroscience, 11, 773–783.

31.

Posner

M. I.

(1980). Orienting of attention. Quarterly Journal of Experimental Psychology, 32, 2–25.

32.

Pourtois

Sander

Andres

Grandjean

Reveret

Olivier

Vuilleumier

(2004). Dissociable roles of the human somatosensory and superior temporal cortices for processing social face signals. European Journal of Neuroscience, 20, 3507–3515.

33.

Reid

V. M.

Striano

Kaufman

Johnson

M. H.

(2004). Eye gaze cueing facilitates neural processing of objects in 4-month-old infants. Neuroreport, 15, 2553–2555.

34.

Rigato

Farroni

Johnson

M. H.

(2010). The shared signal hypothesis and neural responses to expressions and gaze in infants and adults. Social Cognitive and Affective Neuroscience, 5, 88–97.

35.

Rigato

Menon

Farroni

Johnson

M. H.

(2011). The shared signal hypothesis: Effects of emotion–gaze congruency in infant and adult visual preferences. British Journal of Developmental Psychology. doi: 10.1111/j.2044–835X.2011.02069.x

36.

Rigato

Menon

Johnson

M. H.

Farroni

(2011). The interaction between gaze direction and facial expressions in newborns. European Journal of Developmental Psychology, 8, 624–636.

37.

Striano

Kopp

Grossmann

Reid

V. M.

(2006). Eye contact influences neural processing of emotional expressions in 4-month-old infants. Social Cognitive and Affective Neuroscience, 1, 87–94.

38.

Tipples

(2006). Fear and fearfulness potentiate automatic orienting to eye gaze. Cognition & Emotion, 20, 309–320.

39.

Valenza

Simion

Macchi Cassia

Umiltà

(1996). Face preference at birth. Journal of Experimental Psychology: Human Perception and Performance, 22, 892–903.