Reading What the Mind Thinks From How the Eye Sees

Abstract

Human eyes convey a remarkable variety of complex social and emotional information. However, it is unknown which physical eye features convey mental states and how that came about. In the current experiments, we tested the hypothesis that the receiver’s perception of mental states is grounded in expressive eye appearance that serves an optical function for the sender. Specifically, opposing features of eye widening versus eye narrowing that regulate sensitivity versus discrimination not only conveyed their associated basic emotions (e.g., fear vs. disgust, respectively) but also conveyed opposing clusters of complex mental states that communicate sensitivity versus discrimination (e.g., awe vs. suspicion). This sensitivity-discrimination dimension accounted for the majority of variance in perceived mental states (61.7%). Further, these eye features remained diagnostic of these complex mental states even in the context of competing information from the lower face. These results demonstrate that how humans read complex mental states may be derived from a basic optical principle of how people see.

Keywords

facial expressions emotions social perception

Humans can read highly complex mental states from the eyes of other humans (Baron-Cohen, Wheelwright, Hill, Raste, & Plumb, 2001). Eye reading is a critical ability associated with people’s capacity for mentalizing and appreciating that other individuals have minds, intentions, and feelings (Baron-Cohen, 1995). The additional contrast afforded by the white sclera in human eyes, unique among primates (Kobayashi & Kohshima, 1997), highlights how people’s eyes have physically evolved to support their salient role in human social and emotional communication (Lee, Susskind, & Anderson, 2013). Further, the superior temporal sulcus and gyrus, regions that are responsive to the eyes (Allison, Puce, & McCarthy, 2000; Calder et al., 2007), are also neighbors of regions that support theory of mind (Saxe & Powell, 2006). Despite the prominence of the eyes and surrounding tissue in mental-state signaling, the physical features that convey such complex states, and how they came to do so, remains unknown.

Although the eyes may be the proverbial windows to the soul, they originated as conduits of light to the retina. Expressive emotional features that reconfigure the eyes’ appearance may have arisen as sensory adaptations for the expresser that were organized according to Darwin’s principle of antithesis, which posits opposing form and function (Darwin, 1872/1998; Susskind et al., 2008). We have shown recently that expressive eye widening (e.g., in fear) and eye narrowing (e.g., in disgust) are associated with opposing optical consequences and serve opposing perceptual functions (Lee, Mirza, Flanagan, & Anderson, 2014). Operating by the same physical principles as the lens aperture of a camera, eye widening enhances sensitivity, gathering more light information for vigilance (e.g., during fear; Öhman & Mineka, 2001; Whalen, 1998), whereas eye narrowing enhances acuity, exerting sharper focus of light information for discrimination (e.g., during disgust; Chapman & Anderson, 2012; Sherman, Haidt, & Clore, 2012). These opposing sensitivity and discrimination functions dovetail with a division of information-processing needs that is present throughout the visual system, from retinal rods and cones to the crude but fast magnocellular system and slow but sharp parvocellular system (Livingstone & Hubel, 1987), which channel information to dorsal “where” streams and ventral “what” streams, respectively (Ungerleider & Mishkin, 1982).

The notion that the form of facial expressions originates in adaptive sensory function for the sender (Lee et al., 2013, 2014; Susskind et al., 2008) supports the idea that there would be similarity in signaling across cultures (Ekman, Sorenson, & Friesen, 1969; for variations across cultures, see also Nelson & Russell, 2013) and contexts: Eye-narrowing disgust expressions are sometimes confused with physically similar anger expressions but are less often confused with eye-widening fear (Aviezer et al., 2008; Susskind, Littlewort, Barlett, Movellan, & Anderson, 2007). These emotionally expressive effects occur along a physical continuum (Lee et al., 2014), which suggests a fundamental dimension of appearance and meaning that is not restricted to just categories of fear and disgust. We thus hypothesized that expressive features associated with this eye-opening dimension that dynamically regulates the quality of how humans see may help explain how people have come to read a variety of mental states from the eyes. Our thesis is that the opposing effects of eye widening and narrowing on the expresser’s visual perception have been socially co-opted to denote opposing mental states of sensitivity and discrimination, respectively, such that even opposing complex mental states may originate from this simple perceptual opposition.

We grounded our investigation in basic emotional expressions (Ekman et al., 1969) because similar facial features have been shown to communicate not just basic emotions but also complex social information such as personality traits (Gill, Garrod, Jack, & Schyns, 2014; Said, Sebe, & Todorov, 2009). We thus modeled the eyes and surrounding features of six basic expressions, creating an exemplar for each expression by averaging across affective faces from two databases (Ekman & Friesen, 1976; Matsumoto & Ekman, 1988). In Experiment 1, we collected participant ratings of 50 different mental states, including six basic emotions. We then took a dimensional approach (Oosterhof & Todorov, 2008; Russell & Barrett, 1999; Susskind et al., 2008) to understanding how people read mental states from physical eye features. Aligned with our thesis that mental states are rooted in features shaped for sensory function, our hypothesis was that mental states conveying sensitivity would group along eye-widening features, whereas those conveying discrimination would group along eye-narrowing features, and that these mental-state groups would oppose one another. Although eyes were presented in isolation in Experiment 1, we examined their influence in the context of full faces in Experiment 2. If the eyes are windows into inner mental states, then eye-widening and eye-narrowing features should convey mental states of sensitivity and discrimination, both alone and in the context of competing facial information.

General Method: Expression and Feature Modeling

Examining how mental states are read from the eyes began with a simple but important methodological approach. We based our examination on the physical similarity of features (rather than the conceptual similarity of mental states). This is important for two reasons. First, the grounding of expressions in sensory function implies that the appearance of expressions is not arbitrary; they look as they do because they are adaptive interfaces with the environment (Darwin, 1872/1998). Thus, some mental states that are judged as conceptually similar (e.g., fear is similar to disgust in their negative valence; Russell, 1980) may diverge in their expressive form and function (e.g., fear opposes disgust; Susskind et al., 2008). Second, because we are examining how mental states are transmitted, the physical properties (not conceptual properties) are paramount because they constitute the initial signals sent to be decoded by the receiver.

We thus created averaged statistical models of the eyes’ appearance (Susskind et al., 2008) for the six basic expressions using cross-ethnic facial-identity databases (Ekman & Friesen, 1976; Matsumoto & Ekman, 1988; for details, see the Supplemental Material available online). Although participants rated these exemplar expressions for perceived mental states, our analysis examined the expressive features (rather than the emotion categories) underlying these mental-state perceptions. For these analyses, we extracted seven eye features from each exemplar: vertical eye aperture, eyebrow distance, eyebrow slope, eyebrow curvature, nasal wrinkles, temporal wrinkles, and wrinkles below the eyes (Fig. 1). Euclidean coordinates from the appearance models were used to compute eye aperture (distance from top to bottom of eye), eyebrow distance (distance from top of eye to eyebrow), eyebrow slope (slope from start to end of eyebrow), and eyebrow curvature (angular change from start to end of eyebrow). Wrinkle features were computed as the amount of high-spatial-frequency information extracted from corresponding areas (see the Supplemental Material for details).

Fig. 1.

Illustration of the eyes and eye features used in the experiments. From left to right, the images (extracted from expressions of disgust and fear) illustrate differences in eye aperture and eyebrow slope, differences in eyebrow distance and eyebrow curve, and the regions from which wrinkle information was extracted.

Experiment 1: Mental-State Map

To examine the relationship between eye features and mental states, we collected participants’ ratings of the six exemplar eyes for 50 different mental-state terms (six basic emotions and 44 complex mental states that expanded on Plutchik, 1980; for definitions of mental states used in this experiment, see Table S1 in the Supplemental Material). We then analyzed the ratings’ multivariate relationship to the seven eye features.

Twenty-eight undergraduates at the University of Toronto participated for course credit (approximately 30 participants were planned, matching previous behavioral studies). This was an ethnically diverse population sample (for demographics, see the Supplemental Material).

Participants rated the eyes in a within-subjects, randomized, full factorial design (6 exemplars × 50 mental states × 2 repetition blocks; repetitions were averaged). Each trial first showed a fixation cross (500 ms), followed by eyes (subtending 7.5° × 4.7° from a viewing distance of 40 cm). Below that was a mental-state term and a rating scale (from 1, not at all, to 9, very strongly). The target screen remained on for 8,000 ms or until response. A break of up to 1 min was provided halfway through. The experiment was run on an IBM-compatible computer running E-Prime (Version 1.0; Psychology Software Tools, Pittsburgh, PA).

We first examined whether the eye region alone was diagnostic of basic emotions. We computed the basic-emotion recognition accuracy for each eye stimulus according to whether the matching basic-emotion rating or some other basic-emotion rating was higher in pairwise comparison (e.g., accuracy for fear eyes was the percentage of times the fear rating was higher than all of the remaining five basic-emotion ratings, with ties counting as chance, 50%). Averaging the percentage of correct recognition across participants revealed a high mean accuracy of 90.4% (95% confidence interval, or CI = [86.2%, 94.5%]), across all basic-expression eyes, one-sample t test, t(27) = 20.0, p < .0001, r² = .937. Mean accuracy ranged from 84.6% (95% CI = [75.6%, 93.7%]) for anger, t(27) = 7.9, p < .0001, r² = .696, to 96.4% (95% CI = [91.6%, 100%]) for sadness, t(27) = 19.6, p < .0001, r² = .934. We conducted an extended test of the eye region’s ability to convey basic emotions (i.e., we compared each eye stimulus’s matching basic-emotion rating with those for all remaining 49 mental states in pairwise comparison), and this test produced similar results, with mean overall accuracy of 88.8%, t(27) = 24.2, p < .0001. Thus, the eye region reliably conveyed diagnostic information about discrete mental states.

Next, we aimed to map, on the basis of eye features, how all 50 mental states were perceived. To create this eye-feature mental-state map, we first computed independent correlations across participants for each of the 350 combinations of seven features and 50 mental states. The resulting 7 × 50 matrix of r values was treated as 50 coordinates, one for each mental state, in a seven-dimension eye-feature space. We then computed the dissimilarities (euclidean distance) among the ratings of all mental-state terms in this space to capture the relationships between perceived mental states in terms of features (by contrast, examining the relationships between ratings without features reveals their conceptual similarity; Russell, 1980). We created a map of the eyes’ capacity to communicate various mental states by using circular unidimensional scaling (distance variance accounted for 54.0%; Hubert, Arabie, & Meulman, 2006) to chart a navigable map of mental states based on features (Fig. 2b). Two mental states that were similarly correlated across features were closer together relative to two mental states that were dissimilarly correlated.

Fig. 2.

Results from Experiment 1: relationship between mental states and eye features. In the map of mental states shown in (a), the perceived similarity of mental states across eye features is indicated by the mental states’ proximity to one another. The k-means cluster analysis identified four clusters of mental states, which are denoted by different colors. For reference, the labels for the basic emotions matching the exemplar eye stimuli are in larger type than the labels for complex mental states. The opposition of disgust and anger (in which eye narrowing enhances discrimination) to fear and surprise (in which eye widening enhances sensitivity) is illustrated by the fact that the former were maximally distant from the latter on the circle. The diagram in (b) summarizes the eye-feature patterns for each cluster, color-coded as in (a). Correlation of mental states to each eye feature was averaged across all mental states within each cluster. The brighter the square, the greater the magnitude of the Pearson r value (|r|); the plus and minus signs indicate the direction of correlation of each eye feature in each cluster. The illustration of eyes of disgust and fear (c) highlight key features of those expressions (i.e., brow slope and nasal wrinkles for disgust and brow distance and eye aperture for fear).

To calculate the placement of mental-state groups and their boundary transitions along the mental-state circle, we submitted the same dissimilarity data to a k-means cluster analysis. This revealed four distinct clusters of mental states (indicated by distinct colors in Fig. 2a; for further information, see Fig. S1 in the Supplemental Material). The mental-state map revealed that eye-narrowing features of disgust were aligned with a cluster of mental states that convey social discrimination, such as hate, suspicion, aggressiveness, and contempt. On the opposite side of the circle, eye-widening features were aligned with a cluster of mental states that convey information sensitivity, such as awe, anticipation, cowardice, and interest. These clusters were characterized by distinct eye-feature combinations that opposed each other and had inverse correlation patterns (Fig. 2b). Largely orthogonal to this opposition was a dimension anchored by the two remaining clusters. Eye features associated with joy aligned with a cluster of positively valenced mental states, whereas distinctly opposing eye features of sadness aligned with a cluster of negatively valenced mental states. To examine whether mental-state distance was better captured by the dimension of eye widening versus eye narrowing or by the dimension of valence, we conducted a principal components analysis of the (7 × 50) eye-feature mental-state dissimilarity relationships. The analysis indicated that two components met the critical eigenvalue threshold of 1. The widening-narrowing dimension was the primary component, explaining 61.7% of the variance; the valence dimension was the secondary component, explaining 26.8% of the variance. Eye aperture and eyebrow distance loaded on the widening-narrowing component, and temporal wrinkles, found in joy eyes, and eyebrow curvature, found in sad eyes, loaded on the valence component. Together, these components captured 88.8% of the total variance. For detailed results, see the Supplemental Material.

The distribution along the mental-state map and the k-means cluster analysis revealed that the sensitivity cluster had the fewest number of mental states and was more isolated than the other clusters. To measure cluster isolation, we first computed an isolation index for each mental state (i.e., we measured the distance from that mental state to each of the 49 other states and calculated the mean distance). We then grouped these isolation indices by cluster and submitted the four groups to a one-way analysis of variance, which revealed a significant difference in cluster isolation, F(3, 46) = 14.7, p < .0001, η² = .489. A Bonferroni post hoc analysis showed that mental states in the sensitivity cluster were the most isolated (ps < .0001; see Fig. S1a in the Supplemental Material), whereas the remaining clusters were similarly distant from each other (ps > .28). These results show that the sensitivity cluster had a smaller set of mental states and was more isolated from other mental-state clusters, which suggests greater diagnosticity (i.e., less ambiguity) of their perceived mental states. By contrast, the opposing discrimination cluster had more mental states that were closer together, which suggests greater discrimination required to read their meaning. Therefore, the mental contents conveyed by expressions (sensitivity or discrimination) were mirrored by the perceptual needs of the observer—narrower eyes required more discrimination for the observer to diagnose a particular mental state.

Supporting this relative distance, a complete analysis of all mental-state distances across all participants revealed that hate and awe were the most opposed complex mental states communicated through physical eye features. The distance between hate and awe, 1.15, was significantly different from the mean distance between mental states, t(27) = 8.32, p < .0001, r² = .720, 95% CI = [0.868, 1.44], after distance scores across all 946 mental-state pairs (44 mental states × 43/2) were mean-centered at 0.

Experiment 2: Reading Eyes in Full Face Expressions

These results reflect judgments of eyes in isolation. We next investigated the importance of the eyes’ contributions toward evaluating the mental contents in complex, full-face expressions. If eye features associated with widening versus narrowing are used as diagnostic signals of mental states of sensitivity versus discrimination, then observers should rely on them even when the lower face is visible. To test this, we created a set of 36 mixed expressions (Fig. S2 in the Supplemental Material), seamlessly combining the six upper regions (eyes) and six lower regions (nose and mouth) of the expression exemplars modeled in Experiment 1. We then selected a subset of appropriate mental-state terms that uniformly covered the mental-state map in Figure 2a: six basic states (disgust, anger, fear, surprise, joy, and sadness), six complementary complex states (hate, suspicion, awe, cowardice, admiration, and apprehension), and four in-between states (interest, boredom, pride, and remorse).

Twenty-nine new undergraduates at the University of Toronto participated for course credit (approximately 30 participants were planned). Participants rated all 36 faces on all 16 mental-state terms in a within-subjects, randomized, full-factorial design (for details, see the Supplemental Material). Faces subtended 7.5° × 7.5°. The experimental setup and trial structure were identical to the setup and structure used in the previous experiment.

We hypothesized that wide-eyed and narrow-eyed full expressions would convey opposing mental states of sensitivity and discrimination, respectively, as did wide and narrow eyes alone in Experiment 1. We first tested this for congruent expressions (wide-eyed fear and surprise vs. narrow-eyed disgust and anger). We then compared ratings between wide-eyed and narrow-eyed expressions with incongruent lower-face features (i.e., eyes of fear and surprise with all nonmatching lower faces vs. eyes of disgust and anger with all nonmatching lower faces), hypothesizing that the faces would convey the mental states associated with the eyes despite the incongruence. Finally, to determine the importance of the eyes, we examined the converse situation, comparing ratings between wide-mouthed and narrow-mouthed expressions with incongruent eyes (i.e., mouths of fear and surprise with all nonmatching eyes vs. mouths of disgust and anger with all nonmatching eyes), hypothesizing that they would not as strongly convey the same mental states of sensitivity and discrimination.

Congruent expressions

We subtracted ratings for narrow-eyed expressions from those for wide-eyed expressions to calculate difference scores. We first compared congruent wide-eyed, full-face expressions (fear and surprise) with narrow-eyed, full-face expressions (disgust and anger). For mental states of discrimination, averaged ratings of basic emotions (disgust and anger) and complex mental states (hate and suspicion) were highly significantly greater for expressions with narrow eyes than for expressions with wide eyes (Fig. 3). Paired-samples t tests revealed significant differences between wide-eyed and narrow-eyed expressions for basic-emotion ratings, difference score = −3.56, SD = 1.62, 95% CI = [−4.18, −2.94], t(28) = −11.8, p < .0001, r² = .833, and for complex-mental-state ratings, difference score = −2.58, SD = 1.62, 95% CI = [−3.19, −1.96], t(28) = −8.59, p < .0001, r² = .725. Conversely, for mental states of sensitivity, averaged ratings of basic emotions (fear and surprise) and complex mental states (awe and cowardice) were highly significantly greater for expressions with narrow eyes than for expressions with wide eyes (Fig. 3). Paired-samples t tests revealed significant differences between wide-eyed and narrow-eyed expressions for basic-emotion ratings, difference score = 4.43, SD = 1.99, 95% CI = [3.68, 5.19], t(28) = 12.0, p < .0001, r² = .838, and for complex mental-state ratings, difference score = 2.85, SD = 1.84, 95% CI = [2.15, 3.55], t(28) = 8.36, p < .0001, r² = .714. These results were similar to those found for the eyes alone in Experiment 1 (Fig. 3; for statistics of differences in eyes alone, see the Supplemental Material), which suggests that features of eye opening convey similar basic and complex mental states whether the eyes are shown alone or in the context of the full face.

Fig. 3.

Effect of the eyes on mental-state perception alone and in full-face expressions. The graphs show difference scores for averaged ratings of (a) basic emotions and (b) complex mental states when participants viewed eyes only (Experiment 1) and congruent expressions and incongruent expressions (Experiment 2). Difference scores for eyes only and for congruent faces were calculated by subtracting participants’ ratings of mental states when viewing expressions with narrow eyes of disgust and anger from their ratings of mental states when viewing expressions with wide eyes of fear and surprise. Difference scores for incongruent faces were calculated in two ways: by subtracting participants’ ratings of expressions with narrow eyes from ratings of expressions with wide eyes, and by subtracting ratings of expressions with narrow mouths from ratings of expressions with wide mouths. Error bars represent ±1 SEM. Asterisks indicate significant differences between expressions (*p < .05, **p < .001).

Incongruent expressions

Eyes

We next addressed whether features of eye opening could maintain signal value in the context of competing information from the lower face. To examine incongruent expressions, we averaged our data set across the lower face dimension (excluding congruent expressions; see Fig. S2 in the Supplemental Material). Using these ratings of incongruent expressions, which were defined according to incongruence with the eyes, we conducted the same analyses as above. The results confirmed our hypothesis that wide eyes convey sensitivity and narrow eyes convey discrimination, even in the presence of incongruent lower-face information (Fig. 3). Although there was a small reduction in endorsement of incongruent faces compared with congruent faces, the results followed the same overall pattern as found in Experiment 1 and in the congruent full-face expressions. Paired-samples t tests revealed highly significant differences between wide-eyed and narrow-eyed expressions for disgust and anger ratings, difference score = −2.6, SD = 1.22, 95% CI = [−3.06, −2.14], t(28) = −11.5, p < .0001, r² = .825, and for hate and suspicion ratings, difference score = −2.25, SD = 1.10, 95% CI = [−2.67, −1.83], t(28) = −11.0, p < .0001, r² = .811. Paired-samples t tests revealed highly significant differences between wide-eyed and narrow-eyed expressions for fear and surprise ratings, difference score = 2.21, SD = 1.17, 95% CI = [1.76, 2.66], t(28) = 10.1, p < .0001, r² = .786, and for awe and cowardice ratings, difference score = 1.69, SD = 1.25, 95% CI = [1.22, 2.17], t(28) = 7.31, p < .0001, r² = .656. Therefore, the wide-versus-narrow dimension of eye features continued to convey the sensitivity-versus-discrimination dimension of mental states despite the incongruent features from the lower face.

Mouth

If the eyes are indeed most important for conveying these mental states, lower faces from the same expressions (e.g., fear grimace) should not have the same influence. In the context of conflicting information from the eyes, lower-face information should convey less about sensitivity or discrimination. To test this hypothesis, we conducted another analysis, but we focused on the lower face, averaging our data set across the eyes dimension (again excluding congruent expressions; see Fig. S2 in the Supplemental Material). The results confirmed this hypothesis (Fig. 3).

Examining mental states of discrimination, paired-samples t tests revealed no significant differences between wide- and narrow-mouth expressions for disgust and anger ratings, t(28) = −0.49, p = .63, or for complex hate and suspicion ratings, t(28) = −1.58, p = .12. Examining mental states of sensitivity, paired-samples t tests revealed that comparing wide-versus-narrow-mouth expressions did distinguish basic emotions of disgust and anger, difference score = 0.84, SD = 0.77, 95% CI = [0.55, 1.14], t(28) = 5.87, p < .0001, r² = .552, as well as complex mental states of awe and cowardice, albeit to a lesser degree, difference score = 0.27, SD = 0.7, 95% CI = [0.01, 0.54], t(28) = 2.11, p < .05, r² = .137.

However, directly comparing results for incongruent expressions revealed a stronger influence of the eyes than of the lower face on perception of mental states related to sensitivity. For this analysis, we subtracted ratings of expressions that were incongruent with the mouth from ratings of expressions that were incongruent with the eyes to calculate difference scores. Paired-samples t tests revealed that, compared with the mouth, the eyes had a greater influence on fear and surprise ratings, difference score = 1.37, 95% CI = [0.84, 1.89], t(28) = 5.32, p < .0001, r² = .502, and on awe and cowardice ratings, difference score = 1.42, 95% CI = [0.83, 2.01], t(28) = 4.92, p < .0001, r² = .464. In sum, the results from Experiment 2 show that opposing mental states were attributed to eye widening and eye narrowing, even in the context of potentially incongruent information from the rest of the face.

Discussion

The eye features observers use in attributing mental states are not arbitrary but appear to reflect a basic structural logic. Opposing features around the eyes that harness invariant properties of light to bias sensory acquisition and perception in opposing ways (Lee et al., 2014) appear to be socially co-opted to convey our inner thoughts, feelings, and intentions. This result follows from the theory that emotional expressions may originate from a dimension of opening versus closing of the sensory apertures that have egocentric perceptual functional consequences (Darwin, 1872/1998; Lee et al., 2014; Susskind et al., 2008) and now serve as social signals. Hence, the functions and thus the appearance of these features may remain relatively invariant across cultures (Ekman et al., 1969) but may undergo exaptation for more complex social meaning (Shariff & Tracy, 2011) and may be further shaped by specific cultures (Elfenbein, 2013).

We first showed that specific features of the eye region were highly diagnostic of basic emotions, which suggests a coupling logic between the expressive form of the eyes and their conveyed social signals. We then showed that the dimensional variance of these eye features communicated not only basic emotions but also a wide variety of complex mental states, which corroborated previous findings (Baron-Cohen, Wheelwright, & Jolliffe, 1997; Baron-Cohen et al., 2001). To gain further understanding of how the eyes convey these mental states, we used multiple, continuous feature dimensions, along which basic expressions’ features were used as points of anchor. This dimensional perspective was supported by our previous findings that the opposing optical effects of eye apertures associated with fear and disgust were not discrete but exerted along an eye-opening continuum (Lee et al., 2014). Our multidimensional analyses revealed that eye-widening and eye-narrowing features associated with fear and disgust likewise supported how the eyes convey other opposing complex mental states, such as awe and hate.

The reading of emotions and mental states can be understood in part from the perceptual needs of the expresser. Opposing facial actions may bias perception toward the magnocellular visual system (sensitivity to where) or to the parvocellular visual system (discrimination of what; Lee et al., 2014). In the current study, this sensitivity-discrimination organizational principle was the primary dimension accounting for the ability to read mental states from the eyes, and this finding underscores their link to the sensory origins of their expressiveness (Darwin, 1872/1998; Susskind et al., 2008). This expressive antagonism may be engaged in contexts far removed from its sensory origins, associated with complex states, such as suspicion toward and discrimination of potentially unfair social transactions, which evoke sociomoral disgust (Chapman, Kim, Susskind, & Anderson, 2009). This highlights how expressive eye aperture has been socially exapted for purposes beyond its role in biasing visual encoding. Squinting when suspicious of another embodies human mental states in basic perception and action states (Niedenthal, 2007), even if it does not confer enhanced perceptual discrimination. Opening the eyes wide not only enhances the sensitivity of the visual system but also embodies complex mental states related to the need for sensitivity to incoming information, such as interest or awe.

Although we have shown that such expressive actions have direct perceptual consequences for the expresser (Lee et al., 2014), they may be, as Darwin (1872/1998) suggested, largely vestigial or ritualized (Sheperd, Lanzilotto, & Ghazanfar, 2012) in their modern usage, in which their social communicative role is paramount. In this sense, our results are limited to the mental states the receiver reads from these physical features and cannot speak to the true accuracy of these attributions to the sender’s mental states.

Orthogonal to the primary dimension in our mental-state map (Fig. 2a) was a positive-negative valence dimension anchored by joy and sadness, which indicates that a sensitivity-discrimination opposition is not the only dimension by which the eyes communicate internal states. These opposing clusters of positive and negative mental states were expressed by key eye features such as temporal wrinkles (e.g., joy eyes) and eyebrow curvature (e.g., sad eyes; Fig. 2b; see the Supplemental Material online). Reinforcing the dimensional perspective of how people perceive facial information (Oosterhof & Todorov, 2008; Russell & Barrett, 1999; Susskind et al., 2008), a broader exploration of the mental-state map integrating the primary and secondary dimensions revealed interesting potential accounts of other oppositional mental states. For instance, pride and boastfulness were opposite submission and remorse; submission and remorse were between fear and sadness, and pride and boastfulness were between disgust and joy. Likewise, boredom (between sadness and disgust) was opposite interest (between joy and surprise). A variety of nuanced mental states arises from a combination of these two basic dimensions of eye features.

Another notable aspect of the mental-state map (Fig. 2b) is its salient asymmetry. It is possible that the relative abundance of mental states on one side reflects a sampling bias in the mental states we selected for the experiment. Alternatively, a true asymmetry suggested by our cluster analysis, that mental states related to sensitivity were fewer and more isolated than those related to discrimination (see Fig. S1a in the Supplemental Material), may reflect the physical signal bias involving humans’ unique eye whites (Kobayashi & Kohshima, 1997). It has been shown that revealing more sclera in eye widening enhances processing of the eyes (Adolphs et al., 2005; Lee et al., 2013; Whalen et al., 2004). This increase in low-level contrast and luminance sends a physically stronger signal, which may enhance signal detection. (Among our exemplars, there were increases of 3.1 SD in contrast and 2.7 SD in luminance from lowest disgust to highest fear.) Conversely, by shifting the salience away from eye whites to other features surrounding the eyes, eye narrowing requires finer discrimination of higher-frequency information (e.g., wrinkles) across a wider permutation of possible configurations. Although wide eyes may represent a relatively unambiguous signal and unambiguous associated mental states, narrow eyes require greater discrimination to differentiate among underlying mental states. Thus, the opposing expressions (eye widening versus eye narrowing) that enhance visual sensitivity versus discrimination in the sender (Lee et al., 2014) may resonate in how the receiver decodes them (Lee et al., 2013). Wide eyes are strong physical signals and are more diagnostic of mental states of sensitivity, lessening the need for discrimination, whereas narrow eyes are weaker signals and are much less diagnostic, requiring greater scrutiny.

It is worth noting that the eye features we used here are not the only variables that influence how humans read mental states. For example, our examination held direction of eye gaze constant; direction of gaze can influence emotional expression perception (Adams & Kleck, 2005) and is an important contributor to mental-state decoding (Baron-Cohen, 1995). Evidence also suggests that eye features have been co-opted in the service of more complex social pressures. For example, widening the eyes makes the face appear more juvenile and narrowing the eyes makes the face appear more mature (Marsh, Adams, & Kleck, 2005), which may explain why similarly negative expressions, such as fear and anger expressions, elicit asymmetric approach and avoidance responses, respectively, from perceivers (Marsh, Ambady, & Kleck, 2005).

Our results suggest that even when the full face is visible, the eyes receive differential weighting in conveying complex mental states, and the eye region has the most weight. Relative to the eyes, lower face features canonically associated with sensory aperture narrowing (e.g., nose wrinkle and lip raise in disgust; Susskind et al., 2008) or widening (e.g., open mouth in fear and surprise; Susskind & Anderson, 2008; Susskind et al., 2008) conveyed significantly less sensitivity or discrimination, respectively. This finding underscores how the origins of decoding mental states from the eyes relate in part to how the eyes see. The eyes are conduits of light to the retina as well as windows into inner mental states. Conspicuous facial features that harness the physical principles of light may explain how people convey the eye-widening experience of awe or the narrow-mindedness of hate. How expressions alter the way the eyes see reveals an important organizing principle underlying people’s ability to read the complex states of another person’s mind.

Footnotes

Acknowledgements

We thank R. Mahmood and Z. Katsiapis for conducting the experiments and J. Susskind for providing feedback on the facial-expression models.

Action Editor

Alice J. O’Toole served as action editor for this article.

Declaration of Conflicting Interests

The authors declared that they had no conflicts of interest with respect to their authorship or the publication of this article.

Funding

This work was supported by the Canada Research Chairs program, by a Natural Sciences and Engineering Research Council (NSERC) grant (to A. K. Anderson), and by an NSERC postgraduate fellowship (to D. H. Lee).

Supplemental Material

Additional supporting information can be found at

References

Adams

R. B.

Jr. Kleck

R. E.

(2005). Effects of direct and averted gaze on the perception of facially communicated emotion. Emotion, 5, 3–11. doi:10.1037/1528-3542.5.1.3

Adolphs

Gosselin

Buchanan

T. W.

Tranel

Schyns

Damasio

A. R.

(2005). A mechanism for impaired fear recognition after amygdala damage. Nature, 433, 68–72. doi:10.1038/nature03086

Allison

Puce

McCarthy

(2000). Social perception from visual cues: Role of the STS region. Trends in Cognitive Sciences, 4, 267–278. doi:10.1016/S1364-6613(00)01501-1

Aviezer

Hassin

R. R.

Ryan

Grady

Susskind

J. M.

Anderson

A. K.

, . . . Bentin

(2008). Angry, disgusted, or afraid? Studies on the malleability of emotion perception. Psychological Science, 19, 724–732. doi:10.1111/j.1467-9280.2008.02148.x

Baron-Cohen

(1995). Mindblindness: An essay on autism and theory of mind. Boston, MA: MIT Press/Bradford Books.

Baron-Cohen

Wheelwright

Hill

Raste

Plumb

(2001). The “reading the mind in the eyes” test revised version: A study with normal adults, and adults with Asperger syndrome or high-functioning autism. Journal of Child Psychology and Psychiatry, 42, 241–251. doi:10.1017/S0021963001006643

Baron-Cohen

Wheelwright

Jolliffe

(1997). Is there a “language of the eyes”? Evidence from normal adults, and adults with autism or Asperger syndrome. Visual Cognition, 4, 311–331. doi:10.1080/713756761

Calder

A. J.

Beaver

J. D.

Winston

J. S.

Dolan

R. J.

Jenkins

Eger

Henson

R. N. A.

(2007). Separate coding of different gaze directions in the superior temporal sulcus and inferior parietal lobule. Current Biology, 17, 20–25. doi:10.1016/j.cub.2006.10.052

Chapman

H. A.

Anderson

A. K.

(2012). Understanding disgust. Annals of the New York Academy of Sciences, 1251, 62–76. doi:10.1111/j.1749-6632.2011.06369.x

10.

Chapman

H. A.

Kim

D. A.

Susskind

J. M.

Anderson

A. K.

(2009). In bad taste: Evidence for the oral origins of moral disgust. Science, 323, 1222–1226. doi:10.1126/science.1165565

11.

Darwin

(1998). The expression of the emotions in man and animals. New York, NY: Oxford University Press. (Original work published 1872)

12.

Ekman

Friesen

(1976). Pictures of facial affect. Palo Alto, CA: Consulting Psychologists Press.

13.

Ekman

Sorenson

E. R.

Friesen

W. V.

(1969). Pan-cultural elements in facial displays of emotion. Science, 164, 86–88. doi:10.1126/science.164.3875.86

14.

Elfenbein

H. A.

(2013). Nonverbal dialects and accents in facial expressions of emotion. Emotion Review, 1, 90–96. doi:10.1177/1754073912451332

15.

Gill

Garrod

O. G. B.

Jack

R. E.

Schyns

P. G.

(2014). Facial movements strategically camouflage involuntary social signals of face morphology. Psychological Science, 25, 1079–1086. doi:10.1177/0956797614522274

16.

Hubert

Arabie

Meulman

(2006). The structural representation of proximity matrices with MATLAB. Philadelphia, PA: ASA-SIAM.

17.

Kobayashi

Kohshima

(1997). Unique morphology of the human eye. Nature, 387, 767–768.

18.

Lee

D. H.

Mirza

Flanagan

J. G.

Anderson

A. K.

(2014). Optical origins of opposing facial expression actions. Psychological Science, 25, 745–752. doi:10.1177/0956797613514451

19.

Lee

D. H.

Susskind

J. M.

Anderson

A. K.

(2013). Social transmission of the sensory benefits of fear eye widening. Psychological Science, 24, 957–965. doi:10.1177/0956797612464500

20.

Livingstone

M. S.

Hubel

D. H.

(1987). Psychophysical evidence for separate channels for the perception of form, color, movement, and depth. The Journal of Neuroscience, 7, 3416–3468.

21.

Marsh

A. A.

Adams

R. B.

Jr. Kleck

R. E.

(2005). Why do fear and anger look the way they do? Form and social function in facial expressions. Personality and Social Psychology Bulletin, 31, 73–86. doi:10.1177/0146167204271306

22.

Marsh

A. A.

Ambady

Kleck

R. E.

(2005). The effects of fear and anger facial expressions on approach- and avoidance-related behaviors. Emotion, 5, 119–124. doi:10.1037/1528-3542.5.1.119

23.

Matsumoto

Ekman

(1988). Japanese and Caucasian facial expressions of emotion (JACFEE) [Slides]. San Francisco, CA: San Francisco State University, Department of Psychology, Intercultural and Emotion Research Laboratory.

24.

Nelson

N. L.

Russell

J. A.

(2013). Universality revisited. Emotion Review, 5, 8–15. doi:10.1177/1754073912457227

25.

Niedenthal

P. M.

(2007). Embodying emotion. Science, 316, 1002–1005. doi:10.1126/science.1136930

26.

Öhman

Mineka

(2001). Fears, phobias, and preparedness: Toward an evolved module of fear and fear learning. Psychological Review, 108, 483–522. doi:10.1037/0033-295X.108.3.483

27.

Oosterhof

N. N.

Todorov

(2008). The functional basis of face evaluation. Proceedings of the National Academy of Sciences, USA, 105, 11087–11092. doi:10.1073/pnas.0805664105

28.

Plutchik

Kellerman

(Eds.). (1980). Emotion: Theory, research, and experience: Vol. 1. Theories of emotion. New York, NY: Academic Press.

29.

Russell

J. A.

(1980). A circumplex model of affect. Journal of Personality and Social Psychology, 39, 1161–1178. doi:10.1037/h0077714

30.

Russell

J. A.

Barrett

L. F.

(1999). Core affect, prototypical emotional episodes, and other things called emotion: Dissecting the elephant. Journal of Personality and Social Psychology, 76, 805–819. doi:10.1037/0022-3514.76.5.805

31.

Said

C. P.

Sebe

Todorov

(2009). Structural resemblance to emotional expressions predicts evaluation of emotionally neutral faces. Emotion, 9, 260–264. doi:10.1037/a0014681

32.

Saxe

Powell

L. J.

(2006). It’s the thought that counts: Specific brain regions for one component of theory of mind. Psychological Science, 17, 692–699. doi:10.1111/j.1467-9280.2006.01768.x

33.

Shariff

Tracy

(2011). What are emotion expressions for? Current Directions in Psychological Science, 20, 395–399. doi:10.1177/0963721411424739

34.

Sheperd

S. V.

Lanzilotto

Ghazanfar

A. A.

(2012). Facial muscle coordination in monkeys during rhythmic facial expressions and ingestive movements. The Journal of Neuroscience, 32, 6105–6116. doi:10.1523/JNEUROSCI.6136-11.2012

35.

Sherman

G. D.

Haidt

Clore

G. L.

(2012). The faintest speck of dirt: Disgust enhances the detection of impurity. Psychological Science, 23, 1506–1514. doi:10.1177/0956797612445318

36.

Susskind

J. M.

Anderson

A. K.

(2008). Facial expression form and function. Communicative & Integrative Biology, 1, 148–149. doi:10.4161/cib.1.2.6999

37.

Susskind

J. M.

Lee

D. H.

Cusi

Feiman

Grabski

Anderson

A. K.

(2008). Expressing fear enhances sensory acquisition. Nature Neuroscience, 11, 843–850. doi:10.1038/nn.2138

38.

Susskind

J. M.

Littlewort

Barlett

M. S.

Movellan

Anderson

A. K.

(2007). Human and computer recognition of facial expressions of emotion. Neuropsychologia, 45, 152–162. doi:10.1016/j.neuropsychologia.2006.05.001

39.

Ungerleider

L. G.

Mishkin

(1982). Two cortical visual systems. In Ingle

D. J.

Goodale

M. A.

Mansfield

R. J.

(Eds.), Analysis of visual behavior (pp. 549–586). Cambridge, MA: MIT Press.

40.

Whalen

P. J.

(1998). Fear, vigilance, and ambiguity: Initial neuroimaging studies of the human amygdala. Current Directions in Psychological Science, 7, 177–188. doi:10.1111/1467-8721.ep10836912

41.

Whalen

P. J.

Kagan

Cook

R. G.

Davis

F. C.

Kim

Polis

, . . . Johnstone

(2004). Human amygdala responsivity to masked fearful eye whites. Science, 306, 2061. doi:10.1126/science.1103617

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.37 MB