The Role of Gaze in the Processing of Emotional Facial Expressions

Infant Studies

Newborns can discriminate among gaze directions and prefer a neutral facial expression accompanied by direct gaze to one accompanied by averted gaze (Farroni, Csibra, Simion, & Johnson, 2002). A more recent study extended this finding to faces displaying happy and fearful emotional expressions (Rigato, Menon, Johnson, & Farroni, 2011). As suggested by the “shared signal hypothesis,” direct gaze triggered a visual preference only when presented within a happy expression. However, newborns did not show a preference for a fearful face displaying averted gaze over a nonfearful face with averted gaze. Consequently, it is reasonable to conclude that newborns are sensitive to at least some congruent eye gaze–expression combinations.

Evidence suggests a developmental shift with a gradual extension from approach-oriented emotions to avoidance-oriented emotions. Newborns are sensitive to gaze differences only for approach emotions, but adults are sensitive to gaze differences for both approach and avoidance emotions. Supportive of this gradual shift is a study with 4-month-old infants who were sensitive to both, but showed a greater sensitivity for the congruent emotion–gaze combination in the approach-oriented emotions (i.e., happy and angry with direct gaze) (Rigato, Menon, Farroni, & Johnson, 2011).

Altogether, the studies discussed earlier suggest that within the first few days after birth newborns are sensitive to characteristics of faces likely to maximize their chances of interacting with other people. Several characteristics of newborns’ visual preferences indicate that they may have been selected not only for detecting faces, but also for detecting communicative partners (Farroni et al., 2005). Various pieces of evidence support this view: (a) newborns prefer upright to inverted face configurations (Johnson, Dziurawiec, Ellis, & Morton, 1991; Valenza, Simion, Macchi Cassia, & Umiltà, 1996); (b) newborns prefer faces with direct gaze to faces with averted gaze (Farroni et al., 2002), even when schematic faces are used as stimuli (Farroni, Massaccesi, Pividori, & Johnson, 2004); (c) newborns’ preferences are restricted to stimuli composed of darker elements on lighter background (Farroni et al., 2005), which is the same contrast relation that is used in identifying gaze direction in humans (Kobayashi & Kohshima, 1997); and (d) happy emotional expressions displaying a direct gaze are the most preferred face stimuli from birth (Farroni, Menon, Rigato, & Johnson, 2007; Rigato et al., 2011). Taken together, these aspects of newborns’ preferences imply that human babies at birth are most attracted to stimuli appropriate for social and positive interaction.

Additional, and nonetheless complementary, models and explanations can also be applied to these early preferences. A happy expression accompanied by a direct gaze is not only the most likely facial expression experienced in a typical emotional environment during the first few postnatal hours, but also the stimulus that enhances the chances of interacting with other people. Importantly, in the absence of verbal communication, social interactions via facial expression may be one of the most important ways for the newborn to communicate with a caregiver, and therefore they provide an advantage in terms of survival and adaptability. Recent application of theories of embodied cognition to face recognition suggests that the newborns’ interest in eye contact could reflect an evolutionary-based mechanism for triggering embodied simulation in their caretakers (Niedenthal, Mermillod, Maringer, & Hess, 2010). In this respect, smiling is one of the most efficient ways to obtain eye contact with the adult (Blass & Camp, 2001).

Adult Studies

The interaction between expression and gaze has been studied with the Garner interference paradigm, judgment tasks, or gaze shifting tasks. The logic of the Garner paradigm is that if two stimulus dimensions are processed in an integral manner, it will be impossible to attend to one dimension and ignore the other (Graham & LaBar, 2007). For example, in one study participants were asked to make rapid classifications either of expression (happy or angry) or gaze direction (direct or averted) (Ganel, Goshen-Gottstein, & Goodale, 2005). Overall, expression judgments were slower than gaze judgments, and gaze judgments in the direct gaze condition were faster than the averted gaze condition. Further, bidirectional interference between expression and gaze was found, determining that the processing of each of the two dimensions is dependent on the other.

A similar study found an asymmetrical pattern of Garner interference, where expression interfered with gaze judgment but gaze did not interfere with expression judgment (Graham & LaBar, 2007). However, when less intense and less identifiable facial expressions were used, a symmetrical pattern of Garner interference was restored. On the basis of these findings, it is plausible to conclude that when it is hard to discriminate, the emotional expression is processed slowly, and gaze interferes with emotion judgments. Therefore, even though it is clear that judgments on these two dimensions interfere to some extent, such interference is stimulus-bound.

Recent research has also focused on the effect of facial expression on the automatic orienting cued by gaze direction. These studies employed Posner’s classic spatial-cueing paradigm (Posner, 1980), that is, in a target detection task, location cues can facilitate reaction times depending on whether or not the cue is a valid indicator of the target’s location. In the case of facial expressions, emotional faces that orient their gaze are used as the central cue to facilitate reaction times depending on whether or not the direction of the gaze is towards the target’s location or towards the opposite location.

The studies that reported an interaction between the facial expression and the congruency of the gaze direction with the target location revealed faster attention shifting in the presence of a fearful expression (Fichtenholtz, Hopfinger, Graham, Detwiler, & LaBar, 2007; Tipples, 2006) or of a happy one (Hori et al., 2005) relative to neutral faces. However, other studies did not replicate this effect (Bayliss, Frischen, Fenske, & Tipper, 2007; Hietanen & Leppänen, 2003), or found the effect only with special populations (Mathews, Fox, Yiend, & Calder, 2003). These diverse outcomes might have been due to method factors, such as the number of face identities used or the sequential versus the simultaneous modification of the dimensions of gaze direction and facial expression of the central cue. However, the diverse outcomes might also reflect substantive factors. A recent study suggests that the search target has an effect on such tasks (Kuhn & Tipples, 2011). In fact, participants were more likely to follow the gaze on a fearful face compared to a happy face when searching for a threatening target. This stronger cueing effect for fearful faces disappeared when searching for a pleasant target. Thus, it can be concluded that facial expression modulates and facilitates attention orienting by eye gaze cues only under specific conditions.

Neuroimaging Studies

Two brain regions have been found to be particularly involved in the processing of gaze direction and facial expression: the amygdala and the superior temporal sulcus (STS). For example, a functional magnetic resonance imaging (fMRI) study investigated the role of the amygdala in processing threat-related ambiguity by presenting adults with angry and fearful facial displays with direct and averted gaze (Adams, Gordon, Baird, Ambady, & Kleck, 2003). Based on their previous results (Adams & Kleck, 2003), which suggested that expression and gaze are associated on the basis of their communicated intent, the authors reasoned that angry faces with averted gaze and fearful faces with direct gaze corresponded to ambiguous stimuli, and therefore they would have enhanced activity in the amygdala. The findings were consistent with such predictions and demonstrated an important interaction of expression and gaze on amygdala functioning.

However, more recent findings suggest that in emotion perception the role of the amygdala is secondary and possibly restricted to that of seeking out the eyes, that is, driving saccades towards the eye region (see Atkinson & Smithson, 2013). Accordingly, the role of the amygdala might be that of more general and abstract information processing, including processing of salient and biological significant stimuli via its broad connectivity with the cortex and other subcortical structures (Pessoa & Adolphs, 2010). In the case of Adams et al.’s (2003) study, therefore, their ambiguous stimuli might have represented salient and biological significant stimuli in order to enhance activity in the amygdala.

The STS has been found to contain dissociable representations for the perception of facial expression and gaze. For instance, an fMRI study found a greater response for emotional than for neutral faces with direct gaze in the bilateral STS, the right inferior frontal gyrus, and the bilateral occipital lobe (Engell & Haxby, 2007). A stronger activation to neutral faces with averted gaze than with direct gaze was observed in the right STS. Finally, a greater activation to emotional than to neutral faces with averted gaze was observed within the inferior occipital gyrus which extended bilaterally into the lateral fusiform gyrus.

Similarly, dissociable roles for processing emotional expression and gaze have been observed in the somatosensory and superior temporal cortices (posterior STS) respectively (Pourtois et al., 2004). Transcranial magnetic stimulation (TMS) was used to deliver single pulses over these two regions of interest to interfere selectively with the emotional and gaze judgment task. TMS over somatosensory areas interfered with emotion expression recognition, especially with the expression of fear, whereas TMS over superior temporal cortex interfered with the perception of gaze. Altogether, such findings suggest some crucial anatomical–functional segregation between these neural systems for the processing of facial expression and gaze direction. Nevertheless, it is unclear whether co-occurring dimensions of face stimuli, such as expression and direction of gaze, are processed jointly or independently by anatomically and functionally segregated neural structures.

Electrophysiological Studies

The behavioural results reported earlier correspond well with electrophysiological findings. In one study, event-related potentials (ERPs) were recorded from adults and 4-month-old infants while they watched pictures of faces that varied in emotional expression (happy and fearful) and in gaze direction (direct and averted) (Rigato, Farroni, & Johnson, 2010). In adults, the results showed an expression–gaze interaction over a posterior channel location reflected as a greater response to fearful expressions with averted gaze (avoidance-oriented emotion), and to happy faces with direct gaze (approach-oriented emotion), in line with the combinations suggested by the “shared signal” hypothesis.

Interestingly, the effects of facial expression on visual evoked potentials were observed at shorter latencies than those of gaze direction, and the interactions between the processing of emotional expressions and gaze direction appeared at longer latencies. This pattern of results could indicate an initial phase of independent processing of the two dimensions prior to the stage at which the information is integrated. Nonetheless, the main effect of gaze occurred at the same latency as its interaction with facial expression. This means that it is not possible to conclude definitively that gaze processing is independent from facial expression processing; on the contrary, it suggests that gaze is encoded in facial expressions.

In infants, an enhanced ERP response to a happy expression at the frontocentral negative component (Nc) was found, and this was due to the direct gaze condition only. This effect, according to the “shared signal” hypothesis, might reflect an early sensitivity to the approach-oriented emotions; however, it may also represent familiarity with such a stimulus (de Haan & Nelson, 1997) or allocation of attention processes (Courchesne, Ganz, & Norcia, 1981; Nelson, 1994). In another study, infants of 7 months of age showed enhanced responses of the Nc to angry faces with direct gaze, but not with averted gaze, possibly driven by the novelty of the stimulus that facilitates the allocation of attentional resources (Hoehl & Striano, 2008).

Accordingly, other ERP studies have confirmed that from an early age infants are sensitive to socially relevant stimuli and allocate greater attention to them. In one study, infants who saw angry faces with direct gaze had a larger positive slow wave (PSW) amplitude, suggesting an enhanced neural processing of these emotional displays (Striano, Kopp, Grossmann, & Reid, 2006). Increased PSW amplitude may be involved in maintaining information over a period of time, and it has been found to reflect the novelty of the stimulus (de Haan & Nelson, 1999; Nelson, 1997) and allocation of attention (Reid, Striano, Kaufman, & Johnson, 2004).

Therefore already in young infants a direct gaze not only enhances the processing of a happy expression, which is generally the most familiar to the infant, but it also enhances the processing of a less familiar angry face. This might suggest that the familiarity, and therefore the experience with the stimulus, plays a secondary role in the processing of the interaction of certain combinations of expression and gaze. However, the more general development of facial expression processing is affected by experience. For example, studies with infants and children of clinically depressed mothers (Dawson et al., 2003) revealed that an atypical emotional environment involving lower frequency of positive emotion and higher frequency of flat or negative emotion affects this process (de Haan, Belsky, Reid, Volein, & Johnson, 2004).