Neuroscience and Facial Expressions of Emotion: The Role of Amygdala–Prefrontal Interactions

Facial Expressions are Naturally Conditioned Stimuli

Facial expressions mediate a critical portion of our nonverbal communication. From the expressions of others we can glean information about their internal emotional state, their intentions, and/or their reaction to contextual events in our immediate environment. Facial expressions of emotion have predicted important events for us in the past, and we can use this information about previous associated outcomes to respond appropriately to expressions we encounter subsequently. Considered in this light, facial expressions constitute conditioned stimuli (CSs). Likening facial expressions of emotion to conditioned stimuli allows us to draw parallels between studies of social learning based on the facial expressions of others (see, e.g., Davis, Johnstone, Mazzulla, Oler, & Whalen, 2010) and the rich Pavlovian conditioning literature.

Fear, Fearful Facial Expressions, and the Human Amygdala

Numerous nonhuman animal and human studies of aversive Pavlovian conditioning have documented that the amygdala is critical to the acquisition and expression of conditioned behaviors (e.g., freezing) indicative of a learned state of fear (LeDoux, 1996). When it was documented that patients with bilateral lesions of the amygdala showed greater deficits in processing fearful facial expressions, (Adolphs, Tranel, Damasio, & Damasio, 1994), these data were largely interpreted as consistent with the role of the amygdala in fear conditioning. Subsequent demonstrations that presentations of static photographs of fearful expressions produced reliable activation of the human amygdala were also interpreted as consistent with the fear model (Breiter et al., 1996; Morris et al., 1996).

However, lesions of the amygdala also block associative orienting responses (Gallagher & Holland, 1994; Kapp, Whalen, Supple, & Pascoe, 1992; see also Whalen, 1998). Associative orienting responses are defined as any autonomic or somatic responses that reflect increased vigilance or attention, thereby facilitating the animal’s rate of learning during acquisition (see Kapp et al., 1992)—for example, a left head turn observed in a rat being trained that the occurrence of a light to the left predicts the occurrence of a shock. Critically, these responses are observed during the early stages of acquisition during Pavlovian conditioning, as well as any time during learning where the outcome predicted by a particular cue suddenly changes (see Whalen, 1998). Thus, amygdala lesions can ultimately attenuate learned conditioned responses, because animals with such lesions do not attentively engage the environment—they are poor consumers of potentially predictive information.

Indeed, evidence suggests that bilateral lesions of the human amygdala do not prevent individuals from learning about fear faces; rather, these lesions make them inefficient in orienting to the most informative feature of the face. Operationally, this manifests in these subjects not looking to the eye region of the face (Adolphs et al., 2005). Thus, amygdala lesions affect orienting to the place on the face where the best potentially predictive information can be found—suggesting the amygdala is critical to normal face processing and may play a role in the aberration in eye region processing, observed in disorders such as autism (Pelphrey et al., 2002).

Subsequent studies have demonstrated the sensitivity of amygdala responses to fearful expressions by observing similar effects, though fearful expressions were presented using techniques that mitigated subjective awareness (e.g., backward masking, binocular suppression; Etkin et al., 2004; M. J. Kim et al., 2010; Whalen et al., 1998; Williams, Morris, McGlone, Abbott, & Mattingly, 2004). Since a fearful face contains an immense amount of configural information (e.g., raised brows, wide eyes, slightly open mouth, etc.), it is likely that the amygdala does not compute all this information in such a short time frame. Indeed, the presentation of fearful eye whites using backward masking has been shown to be sufficient to produce amygdala activation (Whalen et al., 2004). These data suggest that the amygdala may use widened eyes as a crude proxy for the presence of fearful faces and offer a mechanism for this more automatic response to these complex social stimuli (see Whalen et al., 2009). Though the automaticity of these responses might suggest “innateness” (see Whalen & Kleck, 2008), we assume these amygdala responses are related to previous learning—this “shape” (i.e., eye whites) has predicted important outcomes in the past and now commands the respect of this system.

Amygdala Responses to Other Primary Facial Expressions

Further research has shown that in addition to fearful expressions, the human amygdala is responsive to all primary facial expressions, including anger (Fitzgerald, Angstadt, Jelsone, Nathan, & Phan, 2006; Whalen, Shin, McInerney, & Fischer, 2001; Yang et al., 2002), surprise (H. Kim et al., 2003; H. Kim et al., 2004), disgust (Fitzgerald et al., 2006; Phillips et al., 1997), sadness (Fitzgerald et al., 2006; Yang et al., 2002), and happiness (Fitzgerald et al., 2006; Yang et al., 2002). Taken together, we can conclude from these studies that the human amygdala is responsive to the potentially predictive value of all facial expressions in general, consistent with data showing that increased amygdala activation is not restricted to threat-related information (e.g., Gallagher & Holland, 1994; Paton, Belova, Morrison, & Salzman, 2006).

That said, one caveat when considering such data is whether amygdala activations observed in response to a particular expression are in any way causal to a behavioral outcome. For example, patients with bilateral lesions of the amygdala are more greatly impaired in their processing of fearful expressions but show no behavioral deficit to happy expressions (Adolphs et al., 1994). Amygdala activation in response to happy expressions may be a case of the amygdala monitoring the presence of these expressions in the environment without necessarily having a causal impact on behavior. Alternatively, it is possible that we simply have not yet figured out the behaviors that should be measured to document such a causal link.

In the interim, a number of investigators have adopted the strategy of using fearful expressions as their starting point—directly pitting fearful faces against other primary expressions such as angry (Whalen et al., 2001), happy (Morris et al., 1996), disgusted (Phillips et al., 1997), or surprised (H. Kim et al., 2003) faces. Indeed, in the following sections we describe our efforts to compare fear to other expressions in order to isolate the meaning of amygdala responses to fearful expressions, and more generally the fundamental role of the human amygdala in processing biologically relevant predictive stimuli.

Using Fearful and Angry Facial Expressions to Demonstrate the Amygdala’s Role in Resolving Predictive Uncertainty

Since angry and fearful faces both signal the presence of threat, activity within the amygdala will increase in response to this negative predictive value. But because angry expressions also provide information about the source of threat, there will be a greater extent of amygdala activation observed in response to fearful expressions when directly contrasted with anger (Whalen et al., 2001). A recent behavioral demonstration supported this hypothesis (Davis et al., 2011; see Figure 1). Subjects were presented with pictures of individuals with either fearful or angry expressions in presentation blocks that included alternating neutral words. After passively viewing the fearful face/neutral word blocks and angry face/neutral word blocks, subjects were given recognition tests to assess their memory for the words and faces. Subjects recognized more words that alternated with fearful face presentations compared to angry faces—consistent with the notion that the predictive uncertainty of fearful expressions diffuses attention, thereby increasing memory for the surrounding context. When tested for their recognition of the presented faces, subjects recognized more angry than fearful faces—consistent with the notion that angry faces capture attention since they embody a direct source of threat (Davis et al., 2011). The fact that these two expressions can be equated for their intensity of valence and arousal value subjectively (Davis et al., 2011) and arousal objectively (heart rate; Ekman, Levenson, & Friesen, 1983, and skin conductance; Johnsen, Thayer, & Hugdahl, 1995) suggests that the additional amygdala activation observed to fear is not related to the dimensions of valence or arousal, but another dimension related to information value—or, in this example, predictive uncertainty (Whalen, 1998). These data are consistent with the notion that the predictive uncertainty of a facial expression will produce amygdala activation above and beyond that observed to the detection of predicted negativity per se. This amygdala activation subserves associative orienting—the facilitation of processing in other brain systems that might disambiguate the environmental source of this expressive change (see Whalen, 1998). It is, of course, possible that amygdala responses to fearful faces have more to do with the uncertainty of the expression itself (e.g., “What expression is that?”) rather than what the expression is predicting (as we have argued here). If the assertion that amygdala activation is related to predictive uncertainty has merit, then a compelling demonstration would involve showing a similar amygdala signal increase to another facial expression that has a similar ambiguity of source, but which is not necessarily negatively valenced.

OPEN IN VIEWER

Figure 1.

Davis et al. (2011), experimental design. In a between-participants design, 50% of participants viewed blocks of fearful faces alternating with neutral words and blocks of neutral faces alternating with neutral words. The other 50% of participants viewed blocks of angry faces alternating with neutral words and blocks of neutral faces alternating with neutral words. Faces were presented for 200 ms, and words were presented for 800 ms. A 5-s fixation period separated blocks. Images from Pictures of Facial Affect, by P. Ekman and W. V. Friesen (1976), Palo Alto, CA: Consulting Psychologists Press. Copyright 1976 by P. Ekman and W.V. Friesen. Reprinted with permission from Paul Ekman, Ph.D./Paul Ekman Group, LLC.

Comparing Fearful and Surprised Expressions

One such expression is that of surprise, which provides a critical comparison expression for fear. Though neither expression indicates the exact nature of its eliciting event, fearful expressions do provide additional information concerning predicted negative valence. Surprise, on the other hand, can be interpreted either positively or negatively (Tomkins & McCarter, 1964). For example, a surprised expression might be observed in response to an oncoming car (negative) or an unexpected birthday party (positive). Thus, surprised facial expressions can be used to reveal important individual differences in both (a) the propensity to subjectively ascribe positive or negative valence to an ambiguous predictor and (b) the relationship between this bias and functional magnetic resonance imaging (fMRI) signal changes.

Using Surprised Expressions to Assess Negativity–Positivity Bias

While viewing surprised faces during neuroimaging, high signal change magnitudes were observed in the amygdala in subjects who interpreted these expressions negatively, while lower signal change magnitudes were observed in subjects who interpreted these expressions positively (H. Kim et al., 2003). Critically, these valence-related activations were observed in one portion of the amygdala (i.e., ventral), while another region (i.e., dorsal amygdala/substantia innominata [SI]) showed comparable signal increases in all subjects to surprised faces despite their differing valence interpretations. Thus, within a single group of subjects the amygdala tracked both valence (i.e., positive or negative) and uncertainty (i.e., what is she reacting to?).

That some individuals would show such a positivity bias associated with lower amygdala activity might seem a bit surprising for a brain region that functions to implicitly monitor the environment for potential threat. That is, one might have thought that the amygdala would have responded to the potential negativity of the surprised faces similarly in all subjects. Individual differences of this type suggested to us that another region of the brain might be exerting a regulatory influence over the amygdala while viewing surprised expressions. Accordingly, we observed two regions of the medial prefrontal cortex (mPFC) that were correlated with subjects’ valence interpretations of surprised faces (see Figure 2). Like the amygdala, a dorsal region of the mPFC (specifically the rostral, dorsal anterior cingulate cortex [ACC]; see H. Kim et al., 2003) displayed a positive relationship with negative valence ratings (i.e., higher activity with more negative ratings). A ventral region of the mPFC (ventral ACC]; see H. Kim et al., 2003) showed an opposite relationship with valence ratings of surprised faces compared to the amygdala and dorsal mPFC (i.e., higher activity with more positive ratings). Critically, this inversely correlated ventral mPFC–amygdala fMRI activity was observed during passive viewing of these expressions, predicting ratings given after the scan session. Thus, these amygdala–prefrontal activations that matched subsequent valence interpretations were not task driven and thereby appear to be relatively automatic/implicit (H. Kim et al., 2003). Further, the bias represented by these judgments appears to be more trait-than state-like, as subjects who were tested a year later gave similar ratings (Neta, Norris, & Whalen, 2009).

OPEN IN VIEWER

Figure 2.

(a) 3-dimensional depiction of the correlational results of Kim et al. (2003). Amygdala and dorsal mPFC loci that showed a positive correlation with valence ratings of surprised faces (colored in orange) are positively correlated (red arrow; r=+.66). The ventral mPFC locus that showed a negative correlation with valence ratings of surprised faces (colored in blue) is negatively correlated with amygdala (blue arrow; r=-.69) and the dorsal mPFC (blue arrow; r=-.62); (b) An example of a surprised face stimulus and the valence rating scale used in Kim and colleagues (2003) paradigm.

One interpretation of these data is that, in response to ambiguous surprised expressions, a regulatory override message from the ventral region of the mPFC is required to interpret these faces as positively valenced. Inherent in this assertion is the presumption that initial amygdala activation is part of an early default negative interpretation of surprised faces in all subjects, after which some subjects are able to regulate and respond more positively. Such a hypothesis is consistent with data showing that subjects take longer to ascribe a positive rating to these faces compared to a negative rating (Neta et al., 2009), in that the prefrontal response would require additional time to reverse the initial amygdala reaction.

Note that in this same study, skin conductance responses to negatively and positively rated surprised expressions did not differ, suggesting that the reaction time difference was not related to any difference in the perceived arousal value of positive versus negative surprise. A subsequent study showing that low spatial frequency (LSF) versions of surprised faces are rated as more negative compared to high spatial frequency (HSF) versions (Neta & Whalen, 2010) is also consistent with this hypothesized default negativity of surprise in that LSF versions of fearful expressions have been shown to more readily engage the amygdala (Vuilleumier, Armony, Driver, & Dolan, 2003). Finally, when surprised faces serve as the infrequent stimulus in an oddball paradigm, they are detected more readily when the more frequent expression presented is a happy expression compared to an angry expression—presumably because the default negativity of surprise makes them more of an “oddball” adjacent to happy versus angry expressions (Neta, Davis, & Whalen, 2011). These findings all utilize surprised faces—future work could address whether the role of the amygdala in default negativity can be generalized to other expressions (e.g., neutral faces; Somerville et al., 2004) and other social facial assessments (e.g., trustworthiness; Adolphs, Tranel, & Damasio, 1998).

Given the aforementioned data supporting the notion that initial amygdala responses to surprised expressions are part of a default negativity response, it follows that individual differences in one’s positivity–negativity bias will arise from differences in the strength of the prefrontal regulatory signal that would sometimes function to reverse this initial interpretation. Pragmatically, these data suggest that future studies could utilize surprised faces as presented stimuli as part of a simple, innocuous strategy to engage this prefrontal–amygdala circuitry thought to be critical for regulating negative responses. Failure of such regulation is thought to be at the heart of the anxiety disorders (e.g., Shin et al., 2005; Whalen et al., 2008) and the negativity bias that accompanies major depression (e.g., Alloy & Abramson, 1979; Bouhuys, Geerts, & Gordijn, 1999).

Using Facial Expressions to Assess Amygdala–Prefrontal Interactions

Imagine you are at the zoo. In one of the cages you see a snake coiled up, seemingly ready to strike at anyone who gets close. An initial, instinctive reaction is evoked at the sight of this snake, but it is quickly altered (suppressed, in this example) when you realize the fact that the snake is behind a sheet of Plexiglas and thus no longer poses a threat. Psychologists use the terms bottom–up and top–down processing to explain such phenomena. Bottom–up processing is driven by stimulus characteristics, such as the snake’s appearance in our example. Top–down processing, on the other hand, integrates contextual information that changes the meaning of the given situation. In our example, knowing that a sheet of Plexiglas is shielding you from the snake made you worry less about the actual threat, which in turn may have kept you from running away (Freese & Amaral, 2009; Whalen et al., 2009). Or we can easily imagine a scenario where you encounter the same snake in the wilderness, which would initiate a bottom–up freezing response followed by a completely different type of top–down control in the form of running, crying, screaming, or assuming the fetal position. Thus, it is the interaction between bottom–up and top–down processes that determines the ultimate adaptiveness of behavior in a given situation.

The amygdala and mPFC, among other brain regions, play a central role in behavioral phenomena that are highlighted by the competition between bottom–up and top–down processes, such as emotion regulation, fear conditioning, and extinction (Bishop, 2007; Ochsner & Gross, 2005; Quirk & Beer, 2006; Whalen, 2007). Importantly, the amygdala is known to be heavily interconnected with multiple regions within the mPFC, including the orbital cortex and anterior cingulate cortex (see Freese & Amaral, 2009; Ghashghaei, Hilgetag, & Barbas, 2007). It is hypothesized that the mPFC regulates and controls amygdala output, a top–down control mechanism that keeps bottom–up signals in check (Bishop, 2007; Morgan, Romanski, & LeDoux, 1993; Ochsner & Gross, 2005; Quirk & Beer, 2006). While numerous studies have assessed the separate contributions the amygdala and mPFC make to bottom–up and top–down interactions in emotion, respectively (Bishop, 2007; Simpson, Drevets, Snyder, Gusnard, & Raichle, 2001), more recent studies suggest that the structural and functional connectivity between these two regions is a better predictor of these outcomes than the activity of either region alone (M. J. Kim, Gee, Loucks, Davis, & Whalen, 2011; M. J. Kim & Whalen, 2009; Pezawas et al., 2005).

Conclusions

Studies reviewed here sought to define dimensional constructs (e.g., valence, arousal, predictive uncertainty) that might explain human amygdala responses to specific facial expressions of emotion (i.e., fearful, angry, and surprised). Existing data show that the amygdala can solely track arousal in some instances (Anderson, Christoff, Panitz, De Rosa, & Gabrieli, 2003; Canli, Zhao, Brewer, Gabrieli, & Cahill, 2000; Demos, Kelley, Ryan, Davis, & Whalen, 2008; Williams et al., 2004) and valence in others (Anders, Lotze, Erb, Grodd, & Birbaumer, 2004; H. Kim et al., 2003; H. Kim et al., 2004; Pessoa, Padmala, & Morlan, 2005). With specific reference to facial expressions, surprised expressions were utilized to demonstrate that the amygdala can simultaneously track both arousal and valence (H. Kim et al., 2003; see also Whalen et al., 1998; Winston, Gottfried, Kilner, & Dolan, 2005). One study even set arousal and valence aside for a moment by equating presented fearful and angry faces on these dimensions, to show that predictive uncertainty per se can also modulate amygdala responsivity (Whalen et al., 2001; see also Herry et al., 2007).

The main aim of this review was to show the fruitfulness of using facial expressions as experimental stimuli in order to study how neural systems support biologically relevant learning as it relates to social interactions. Though use of these stimuli means we will lack the ability to control for reinforcement history, it is this history that will give rise to individual differences in neural responsivity and subsequent behavior. Finally, facial expressions offer a relatively innocuous strategy with which to investigate normal variations in affective processing, as well as the promise of elucidating what role the aberrance of such processing might play in emotional disorders (Bouhuys et al., 1999; Shin et al., 2005; Whalen et al., 1998).