Visual attention mechanisms in happiness versus trustworthiness processing of facial expressions

Abstract

A happy facial expression makes a person look (more) trustworthy. Do perceptions of happiness and trustworthiness rely on the same face regions and visual attention processes? In an eye-tracking study, eye movements and fixations were recorded while participants judged the un/happiness or the un/trustworthiness of dynamic facial expressions in which the eyes and/or the mouth unfolded from neutral to happy or vice versa. A smiling mouth and happy eyes enhanced perceived happiness and trustworthiness similarly, with a greater contribution of the smile relative to the eyes. This comparable judgement output for happiness and trustworthiness was reached through shared as well as distinct attentional mechanisms: (a) entry times and (b) initial fixation thresholds for each face region were equivalent for both judgements, thereby revealing the same attentional orienting in happiness and trustworthiness processing. However, (c) greater and (d) longer fixation density for the mouth region in the happiness task, and for the eye region in the trustworthiness task, demonstrated different selective attentional engagement. Relatedly, (e) mean fixation duration across face regions was longer in the trustworthiness task, thus showing increased attentional intensity or processing effort.

Keywords

Attention eye movements dynamic facial expressions happy faces trustworthiness

Introduction

Facial happiness (i.e., an expresser’s smiling face) is significantly related to the perception of trustworthiness by observers. People showing happy expressions are judged as more trustworthy than those with non-happy faces (while facial anger is perceived as untrustworthy). This robust finding occurs for emotional faces (Centorrino, Djemai, Hopfensitz, Milinski, & Seabright, 2015; Engell, Todorov, & Haxby, 2010; Johnston, Miles, & Macrae, 2010; Krumhuber et al., 2007; Miles, 2009; Oosterhof & Todorov, 2009; Quadflieg, Vermeulen, & Rossion, 2013; Sutherland, Young, & Rhodes, 2017; Willis, Palermo, & Burke, 2011; Winkielman, Olszanowski, & Gola, 2015), as well as for “happier-looking” (or “angrier-looking”) neutral faces, which are judged as and trustworthy (or untrustworthy) (Brewer, Collins, Cook, & Bird, 2015; Hehman, Flake, & Freeman, 2015; see Said, Haxby, & Todorov, 2011). Relatedly, facial happiness enhances the effects of other factors (e.g., expressers’ gaze direction) on observers’ trustworthiness ratings (Manssuer, Roberts, & Tipper, 2015; Strachan, Kirkham, Manssuer, & Tipper, 2016). Such a relationship highlights the shared adaptive importance of happiness (or anger) and trustworthiness (or untrustworthiness) detection, as both serve crucial roles in identifying potential friends or foes. To this end, observers use facial information to figure out the intentions and emotions of other people, thereby inferring their level of trustworthiness.

The close relationship between happiness and trustworthiness judgements suggests that both could be driven by the same mechanisms (Engell et al., 2010; Said et al., 2011). This study investigates the similarities and differences in visual attention mechanisms underlying the assessment of facial happiness and trustworthiness. We focused on the observers’ deployment of overt attention (i.e., eye fixations) during evaluations of happiness or trustworthiness in face stimuli. There are three relevant aspects to be considered regarding the observers’ gaze behaviour: where (i.e., which face region of the observed expresser is selectively looked at), when (i.e., the time course of first entering each face region), and how much (i.e., the amount of allocated processing resources, as shown by gaze duration and number of fixations). Selective attention to a particular face region (e.g., the eyes), temporal prioritisation of attention to that region, and subsequent enhanced attentional engagement involve preferential processing of specific face cues. As a consequence, happiness and trustworthiness perception could vary as a function of where, when, and how much such expressive cues are selectively looked at, given that preferential overt attention to a particular region is highly related to how a facial expression is judged (see Calvo, Gutiérrez-García, Avero, & Lundqvist, 2013; Schurgin et al., 2014; Vaidya, Jin, & Fellows, 2014). We therefore wanted to explore whether visual attention regarding these three aspects will be similar or different in the perception of happiness and trustworthiness.

There is evidence for the involvement of specific mechanisms in the processing of facial happiness. The typical recognition advantage of happy expressions (for reviews, see Calvo & Nummenmaa, 2016; Nelson & Russell, 2013) has been explained in terms of the distinctive and salient smiling mouth (Calvo, Fernández-Martín, & Nummenmaa, 2012). First, as a distinctive feature, the smile is specifically associated with facial happiness, that is, the smile is generally present in happy faces but absent in non-happy faces. As a consequence, the smile becomes diagnostic of happiness (Calder, Young, Keane, & Dean, 2000; Nusseck, Cunningham, Wallraven, & Bülthoff, 2008; Smith, Cottrell, Gosselin, & Schyns, 2005). In fact, the smiling mouth region on its own enhances the activity of ERP (brain event-related potentials) components (P3b) related to expression categorization (Calvo & Beltrán, 2014), with a neural signature that is source-located at the right infero-temporal (IT, fusiform gyrus [FG]) and dorsal cingulate (CC) cortices (Beltrán & Calvo, 2015). Second, as a salient feature, the smile attracts more overt attention (i.e., eye fixations) during expression recognition than any other facial region of the basic six emotional expressions (Beaudry, Roy-Charland, Perron, Cormier, & Tapp, 2014; Bombari et al., 2013; Calvo & Nummenmaa, 2008). Furthermore, the smile salience is associated with early attentional capture (90-130 ms post-stimulus onset), as assessed by the N1 ERP component (Calvo, Beltrán, & Fernández-Martín, 2014), and a neural signature that is source-located at the left infero-temporal (IT, middle temporal gyrus [MTG]) cortex (Beltrán & Calvo, 2015). Thus, the smiling mouth becomes easily accessible to perception, which secures an early processing of this diagnostic cue, allowing for it to be used as a shortcut for quick categorization of a face as happy (Adolphs, 2002; Leppänen & Hietanen, 2007).

The literature reviewed above indicates that facial happiness processing is highly dependent on selective and enhanced visual attention to the smiling mouth. To our knowledge, however, these mechanisms have not been investigated in the processing of trustworthiness. In this study, we aimed to compare the visual attention mechanisms in happiness versus trustworthiness processing within the same experimental paradigm. In a first approach, Calvo, Álvarez-Plaza and Fernández-Martín (2017) found that the contribution of the smiling mouth was greater for happiness than trustworthiness judgements, and the mouth was especially visually salient for expressions favouring happiness judgements. It was argued that the categorization of facial happiness is more automatically driven by the visual salience of a single feature, that is, the smiling mouth, whereas the perception of trustworthiness is more strategic, with the eyes being necessarily incorporated into a configural face representation. Nevertheless, Calvo et al. (2017) did not collect eye-movement data. In a further step, the current study therefore investigated whether the smiling mouth selectively attracts overt attention (earlier and more frequent or longer eye fixations) when judging happiness while the eye region attracts more attention when judging trustworthiness.

We hypothesised that judgements of happiness and trustworthiness are highly related (see above), but reached through different attentional processes, based on the following rationale. First, happy people often smile (albeit not all smiles reflect happiness), whereas trustworthy people may or may not smile, which makes a smile diagnostic of happiness but not of trustworthiness. Unlike facial happiness, which involves an observable facial cue (i.e., the smile), trustworthiness has no such distinctive signal. To infer trustworthiness, the observer should accordingly rely less on the smile and allocate attention to other parts of the face instead, thereby evaluating the expressive congruence from different sources (e.g., the eye–mouth incongruence could be seen as a sign of untrustworthiness). As a result, the smiling mouth would attract less attention during the processing of trustworthiness relative to happiness. Second, although expressive changes in the eye region play a minor role (i.e., they are not necessary or sufficient) in the categorization of a face as happy (e.g., Calder et al., 2000; Calvo, Fernández-Martín, & Nummenmaa, 2014), they are critical for the affective processing of a smile as positively valenced, and for judging a face as genuinely happy (Johnston et al., 2010; Krumhuber, Likowski, & Weyer, 2014; McLellan, Johnston, Dalrymple-Alford, & Porter, 2010). To the extent that happy eyes contribute to the smile’s “authenticity” or “genuineness,” it is understandable that they convey trustworthiness, whereas non-happy eyes make a smile appear to be fake and in turn untrustworthy.¹ Given that trustworthiness is an essential component of positive face valence (Oosterhof & Todorov, 2008), we can expect trustworthiness processing to be particularly sensitive to the eye region which should receive greater attention.

The above argument leads us to predict differential selective overt attention to the mouth or the eyes of a face (i.e., where visual fixations are allocated), depending on whether happiness or trustworthiness is judged and on processing stage (orienting or engagement). First, attentional orienting (i.e., when each face region is fixated first) will not significantly differ between happiness and trustworthiness evaluations. Rather, the initial/early attentional capture by the mouth or the eyes will occur automatically in a similar way for happiness and trustworthiness. Orienting will be mainly determined by stimulus properties rather than the viewer’s task strategy. To assess selective orienting, we measured entry times (i.e., the time elapsed between stimulus onset and first fixation on a face region) and initial fixation thresholds (i.e., the earliest time each region was fixated more than the other regions). Second, after initial orienting, the viewer’s task strategy will guide attention allocation. Selective attentional engagement (i.e., how much each region is fixated) will be greater for the smiling mouth (which will attract more and/or longer eye fixations) when judging happiness, whereas the eye region will attract more and/or longer fixations when judging trustworthiness. To assess selective engagement, we measured fixation density (i.e., number of fixations) and mean fixation duration (i.e., for each individual fixation). Third, an additional aspect was considered, that is, scanpaths involving the number of fixations on a particular region coming from other regions. As argued above, the eyes should be incorporated into a configural face representation for trustworthiness processing. Accordingly, if the integration of features across the face is more important for trustworthiness than happiness, greater visual back-and-forth shifting between the eye and mouth regions would be expected when judging trustworthiness relative to happiness.

We used 2-s video-clips displaying facial expressions as stimuli, with different combinations of the mouth (smiling or neutral) and eye expression (happy or neutral). Morphed dynamic expressions (instead of static photographs) were used to mimic real-life expressions and to enhance measurement sensitivity (Calvo, Avero, Fernández-Martín, & Recio, 2016; Krumhuber & Scherer, 2016; for a review, see Krumhuber, Kappas, & Manstead, 2013). In six different types of expressions, the eyes and mouth unfolded—together or independently—from neutral to happy or vice versa. Participants judged how happy (happiness task) or trustworthy (trustworthiness task) the expressers appeared to be. Eye movements and fixations were recorded for different face regions and across periods of expression unfolding. This approach allowed us to determine the relative role of each major expressive source (i.e., the eyes and mouth regions) in the spatio-temporal oculomotor profiles associated with each task. To this end, blended expressions (i.e., with non-congruent eyes and mouth) were necessary, in addition to prototypical expressions (i.e., with congruent eyes and mouth), for combining expressive cues, and thus to determine their relative contribution in each type of task. We were particularly interested in potential interactions between face region and unfolding time, to examine similarities and differences in gaze behaviour between happiness and trustworthiness processing.

Method

Participants

A total of 40 psychology undergraduates (26 females, 14 males; aged 18-30 years) participated for course credit, after providing informed consent. Half of them (13 females, 7 males) were randomly assigned to a condition involving facial happiness judgements, and another half to a trustworthiness evaluation condition. The study was approved by the ethics committee of the University of La Laguna and conducted in accordance with the WMA Declaration of Helsinki 2008.

Stimuli

We used 2-s video-clips as stimuli. To generate the different stimulus conditions, we first selected photographs of prototypical neutral expressions (i.e., neutral eyes and mouth; henceforth, Neutral) and happy expressions (i.e., happy eyes and a smiling mouth; henceforth, Happy) of 24 posers (12 females, 12 males) from the Karolinska Directed Emotional Faces database (KDEF; Lundqvist, Flykt, & Öhman, 1998). Second, composite faces were constructed for each poser by combining the upper half of each happy face with the lower half of the neutral face, and vice versa (e.g., Tanaka, Kaiser, Butler, & Le Grand, 2012). This resulted in two types of blended expressions: (a) neutral eyes and smiling mouth (henceforth, Ne+Sm) and (b) happy eyes and neutral mouth (henceforth, He+Nm). Figure 1 shows an example of these expressions.

Figure 1.

Types of prototypical (Neutral: neutral eyes and mouth or Happy: happy eyes and mouth) and blended expressions (He+Nm: happy eyes and neutral mouth and Ne+Sm: neutral eyes and a smiling mouth), regions of interest, and resulting dynamic expression conditions. Left and right, from the viewer’s perspective (i.e., visual field). Faces adapted from the Karolinska Directed Emotional Faces Database. Copyright by Daniel Lundqvist, Anders Flykt, and Arne Öhman (1998). Reprinted with permission.

Third, the resulting photographic versions (Neutral, Happy, He+Nm, and Ne+Sm) were converted into 30-frame per second dynamic expressions by means of FantaMorph© software (v.5.4.2; Abrosoft). To this end, one photograph of each version was used as the first frame at the beginning of the sequence (e.g., Neutral) and another photograph (e.g., Happy) was used as the last frame of the sequence. FantaMorph generated a continuum that smoothly unfolded from one expression to the other. This yielded six experimental conditions of dynamic expressions (see Figure 1), depending on the type of expression at the beginning and end of the sequence. For example, Neutral→Happy: initial neutral eyes and mouth unfolding towards final happy eyes and (smiling) mouth; Neutral→Ne+Sm: initial neutral eyes and mouth unfolding towards final neutral eyes and a smiling mouth; and so on.² A total of 144 video-clip stimuli were used (24 posers by six stimulus conditions). Samples of video-clips for each condition are shown in the ESM_1.mpeg (Supplemental Materials: Sample Stimuli) electronic file.

Within each video-clip, the initial expression (e.g., Neutral) lasted for 500 ms (first period, static) and was followed by a 1,000-ms unfolding (second period, dynamic) towards the final expression (e.g., Happy), which remained still (third period, static) for 500 ms. The 1-s unfolding was established to approximate the typical and natural average speed in the recognition of dynamic expressions as achieved in prior research (see Hoffmann, Traue, Bachmayr, & Kessler, 2010). The same dynamic (expression unfolding) display duration (1,000 or 1,040 ms) was used by Schultz and Pilz (2009), Johnston, Mayes, Hughes, and Young (2013), and Wingenbach, Ashwin, and Brosnan (2016). Each face subtended a visual angle of 10.6° (height) × 8.0° (width) at a 70-cm viewing distance, which approximates the size of a real face (18.5 × 13.8 cm) viewed from 1 m.

Objective assessment of “happiness” in the eye and the mouth region

We assumed that our so-called “happy” face stimuli involve happy eyes and a smile. The operationalization of these facial features, however, requires objective measurement, particularly for the eye expression due to its subtle changes. To this end, we assessed morphological Action Units (AUs), according to Facial Action Coding System (FACS; Ekman, Friesen, & Hager, 2002), by means of Emotient FACET software (v6.1; see iMotions, 2016; https://imotions.com/blog/facial-expression-analysis/), which is an automated facial expression analysis tool (e.g., Bartlett & Whitehill, 2011; Cohn & De la Torre, 2015).

AUs are anatomically related to the movement of specific face muscles (e.g., AU12 involves the contraction of the zygomaticus major muscle, which draws the angle of the mouth superiorly and posteriorly to allow for smiling). To quantify each of 20 AUs, FACET provides evidence scores that are expressed in odds ratios in a decimal logarithmic scale, where positive values indicate that an AU is present; negative values, that it is not present; and a zero score indicates chance level. For the current study aims, we selected four AUs. Two of them typically characterise happy faces according to FACS: AU6 (cheek raiser, with the D-marker around the eye region) and AU12 (lip corner puller in the mouth region) (Ekman et al., 2002). Also, albeit of secondary importance as a morphological feature of happy eyes, AU7 (lid tightener; i.e., narrowing of the eye aperture and some tension of the eyelids) can be considered as a cue to happy face authenticity (Del Giudice & Colle, 2007), and AU25 (lips part), as a measure of the intensity of a smile in the mouth region. We assessed and quantified these four AUs in the current happy and neutral face stimuli.

Faces with happy eyes showed AU6 to a greater extent (M = 3.12; standard deviation [SD] = 0.67; in odds ratios, as provided by FACET) than faces of the same individuals with neutral eyes (M = −1.98; SD = 0.69), t(46) = 26.00, p < .0001, d = 7.51, and above the zero baseline, t(23) = 22.83, p < .0001. AU7 was also greater in faces with happy eyes (M = 0.46; SD = 0.55) than with neutral eyes (M = −0.63; SD = 0.51), t(46) = 7.10, p < .0001, d = 2.05, and above the zero baseline, t(23) = 4.09, p < .0001. Similarly, faces with a smile showed AU12 to a greater extent (M = 4.29; SD = 0.57) than with a neutral mouth (M = −1.95; SD = 0.63), t(46) = 36.17, p < .0001, d = 10.44, and above the zero baseline, t(23) = 37.14, p < .0001. AU25 was also greater for faces with a smile (M = 2.37; SD = 0.63) than with a neutral mouth (M = −1.91; SD = 0.59), t(46) = 24.21, p < .0001, d = 6.99, and above the zero baseline, t(23) = 18.39, p < .0001. This validates our operationalization of happy versus neutral eyes, and a smiling versus neutral mouth.

Procedure

Each participant was presented with all the 144 video-clips (24 of each of six stimulus conditions), in four blocks of 36 trials, following 16 practice trials. Experiment Centre and iView X software (SMI; SensoMotoric Instruments GmbH, Teltow, Germany) was used for stimulus presentation and data collection. Block order was counterbalanced, the number of trials in each stimulus condition was balanced for each block, and trial order was randomised for each participant. Participants were told that short videos of faces would be presented, with different expressions (otherwise unspecified). Participants were asked to judge either “how happy each expresser looked like over the course of the expression unfolding,” on a 1 (“negative feelings”) to 9 (“very happy”) scale (happiness task), or “how trustworthy each expresser looked like. . .” on a 1 (“untrustworthy”) to 9 (“very trustworthy”) scale (trustworthiness task), and to respond quickly by pressing a key on the top row of a computer keyboard.

The sequence of events on each trial is shown in Figure 2. After an initial 500-ms fixation cross at the centre of a screen, a video-clip appeared: a still initial expression (500 ms) was followed by a dynamic display unfolding towards the final expression (1,000 ms) and a still final expression (500 ms). Following face offset, the question “how happy”? (happiness judgement task) or “how trustworthy”? (trustworthiness judgement task) appeared. The selected response and reaction times were collected. A 1,250-ms blank screen served as an intertrial interval.

Figure 2.

Sequence of events on each trial. Faces adapted from the Karolinska Directed Emotional Faces Database. Copyright by Daniel Lundqvist, Anders Flykt, and Arne Öhman (1998). Reprinted with permission.

Experimental design

The experimental design involved an orthogonal combination of Task (2: Happiness vs Trustworthiness), as a between-subjects factor, and Dynamic Expression condition (6: see Figure 1 or Table 1), as a within-subjects factor. For Dynamic Expression, the different combinations of eye and mouth, along with their unfolding from an initial to a final expression, yielded six conditions. There were two prototypical expressions: Neutral→ Happy and Happy→Neutral (i.e., Nos 1 and 6 in Figure 1, respectively) and four blended expressions: He+Nm→Happy, Neutral→Ne+Sm, Happy→He+Nm, and Neutral→He+Nm (i.e., Nos 2, 3, 4, and 5, in Figure 1, respectively). Half of the expressions (Nos 1-3) ended with a smiling mouth, with either neutral or happy eyes at the beginning or at the end of the video sequence, and the other half (Nos 4-6) ended with a neutral mouth, with either neutral or happy eyes at the beginning or at the end.

Table 1.

Mean trustworthiness scores (9-point scale) and reaction times (RTs; ms), as a function of task and dynamic facial expression.

Task		Type of dynamic expression
		Final expression WITH a smile			Final expression WITHOUT a Smile
		Neutral to Happy	He+Nm to Happy	Neutral to Ne+Sm	Happy to He+Nm	Neutral to He+Nm	Happy to Neutral
Happiness
Response	M	6.92^a	6.36^b	6.11^b	2.97^c	2.96^c	3.02^c
	SD	0.67	1.08	1.06	0.83	0.78	0.93
RTs	M	728^a	888^b	867^b	843^{a b}	915^b	762^a
	SD	205	297	277	276	306	265
Trustworthiness
Response	M	6.78^a	5.93^b	5.89^b	2.75^c	2.61^c	3.14^c
	SD	0.73	1.21	1.30	0.77	0.69	1.20
RTs	M	804^a	916^b	918^b	850^{a b}	897^{a b}	863^{a b}
	SD	149	181	199	202	168	210

SD: standard deviation; Neutral: neutral eyes and neutral mouth. Happy: happy eyes and smiling mouth. He+Nm: Happy eyes and neutral mouth. Ne+Sm: Neutral eyes and smiling mouth.

Average scores with a different superscript are significantly different across type of expression; scores sharing a superscript are equivalent.

Eye-movement measures

Gaze behaviour was recorded via a 500-Hz (binocular; spatial resolution: 0.03°; gaze position accuracy: 0.4°) RED system eyetracker (SMI GmbH; Teltow, Germany). Six face regions of interest were defined: forehead, left eye and eyebrow (henceforth, left eye), right eye and eyebrow (henceforth, right eye), nose/cheek (henceforth, nose), mouth, and chin (see their shapes in Figure 1). Left and right eye are considered from the viewer’s perspective, that is, left-eye fixations refer to fixations made by viewers towards their left visual field (actually, the right eye of the expresser). Approximately 98% of total fixations occurred within these six regions. For statistical analyses, the forehead and the chin were excluded because less than 1% of fixations landed on these regions. Net gaze duration was obtained after saccades (M frequency per second = 5.24; M saccade duration = 43 ms) and blinks (mean frequency per second = 0.15; mean blink duration = 156 ms) were removed. For saccade and fixation detection parameters, we used a velocity-based algorithm with a 40°/s peak velocity threshold and 80 ms for minimum fixation duration.

Number of fixations and gaze duration were collected for each face region and period and converted into a fixation density measure, that is, the total number of fixations (of all the viewers) on each region at a given time, during each of 60 consecutive 33-ms time bins across the 2-s face display. This provided a detailed analysis of the gaze time course (see Bindemann, Scheepers, & Burton, 2009). Fixation density scores are independent from differences in the duration (i.e., 500 or 1,000 ms) of the three major periods (see section “Procedure”), as fixation density was adjusted to the number of 33-ms time bins in each period. Also, given that the size of face regions varied (left eye region = 6.22 pixels, right eye region = 6.22, nose-cheek = 8.55, and mouth = 7.28), the raw density scores were adjusted to size: (raw density scores/region size) × 100 (see Figures 3 and 4, and Graphical Abstract). This allowed us to make fixation density comparisons across periods and regions.

Figure 3.

Fixation density differences between the happiness and the trustworthiness judgement tasks, for each face region. Fixation density = (raw density scores/region size) × 100. Vertical lines in bars indicate standard errors of the mean.

Figure 4.

Fixation density across 60 33-ms bins during the 2-s face display, for each region of expressions unfolding to a final smile, or to a final neutral mouth. Left side: happiness task; right side: trustworthiness task. Arrows indicate the threshold, that is, the earliest time bin (onset, in ms; e.g., 300), at which a region (e.g., left eye) had significantly more fixation density than all the other regions. Two scores within a box indicate the amplitude, that is, the interval during which there was a fixation advantage for one task versus the other; for example, 1,000 to 1,400 indicates greater fixation density on the mouth region from 1,000 to 1,400 ms in the happiness than in the trustworthiness task, for expressions ending with a smile. Left and right eye (from the viewer’s perspective) refer to visual field. Fixation density = (raw density scores/region size) × 100.

From fixation density measures, we computed the thresholds for each region (i.e., the earliest 33-ms time bin at which each region was fixated first significantly more than all the other regions). This served as an index of early selective attentional orienting. We also computed fixation density amplitudes (i.e., the interval following initial orienting during which each region was fixated significantly more in one task or the other). This served as an index of selective attentional engagement. In addition, entry times (i.e., the time elapsed from face onset until first fixation on each region) were examined as a complementary measure of attentional orienting; and mean fixation duration (i.e., how long was each single fixation on average), as a complementary measure of attentional engagement.

An additional measure was included, which involved the scanpaths of fixations on a particular region coming from or going to other regions. To this end, we considered the number of fixations landing on each region (e.g., the eyes) that launched from each of the other major face regions (mouth and nose), and so on. This was aimed at detecting back-and-forth shifting between the eye and mouth regions when judging trustworthiness relative to happiness, as a configural integration processing strategy.

Results

Judgement performance

Judgement ratings and reaction times were analysed by means of a Task (2: happiness vs trustworthiness) × Dynamic Expression (6; see section “Experimental design”) analysis of variance (ANOVA).³ Bonferroni corrections (p < .05) were conducted for all post hoc contrasts involving multiple comparisons, for these measures and the eye-movement measures (unless otherwise indicated). Effects of expression emerged for response ratings, F(5, 190) = 205.66, p < .0001, $η_{p}^{2} = . 84$ , and reaction times, F(5, 190) = 7.85, p < .0001, $η_{p}^{2} = . 17$ . There were no significant effects of task or interactions (all Fs < 1.23, ps > .28). This implies that the pattern of effects was the same for happiness and trustworthiness judgements, which was confirmed by a significant correlation between happiness and trustworthiness ratings (r = .93; p < .0001; N = 144 stimuli).

For response ratings, post hoc contrasts showed that all the expressions with a final smiling mouth were judged as happier and more trustworthy than those ending with a neutral mouth (which did not differ from one another). In addition, within the former group (i.e., final smile), prototypical happy expressions (i.e., Neutral→Happy: initial neutral eyes and mouth unfolding to final happy eyes and a smile) were judged as happier and more trustworthy than blended expressions (He+Nm→Happy: initial happy eyes and neutral mouth unfolding to final happy eyes and a smile; and Neutral→Ne+Sm: initial neutral eyes and mouth unfolding to final neutral eyes and a smile), which did not differ from each other (see Table 1). Consistently, for reaction times, post hoc contrasts revealed that Neutral→Happy faces were responded to faster than He+Nm→Happy faces and Neutral→Ne+Sm faces, which did not differ from each other and the rest (see Table 1).

Eye-movement measures: attentional orienting

To examine attentional orienting (i.e., when each region was fixated first), we analysed entry times (see section “Eye-movement measures”) by means of a Task (2) × Dynamic Expression (6) × Region (4) ANOVA. Main effects of region appeared, F(3, 114) = 51.77, p < .0001, $η_{p}^{2} = . 58$ , with the nose being fixated earlier (M = 235 ms) than the left eye (M = 516), which was fixated earlier than the right eye (M = 818) and the mouth (M = 919), which did not differ from each other (after Bonferroni-correction, p < .05, post hoc contrasts). Importantly, neither the effects of task nor the interactions of task with region or with region and expression were statistically significant (all Fs < 1). This implies that task did not affect initial orienting to particular regions.

This was corroborated by an additional analysis of the orienting threshold for each region (see section “Eye-movement measures”). We conducted one-way (4: Region) ANOVAs on such thresholds for each 33-ms time bin, followed by Tukey t-tests for multiple post hoc comparisons across regions (p < .05). Thresholds clearly emerged for the nose (33 ms), the left eye (300 ms), and the mouth (happiness task: 933 ms, for expressions ending with a smile; 967 ms for those ending with a neutral mouth; no threshold in the trustworthiness task) at different time points (all Fs(3, 92) ⩾ 59.58, p < .0001, $η_{p}^{2} ⩾ . 66$ ) (see Figure 4; also Figures 5 and 6 of the Supplemental Materials). Importantly, thresholds were practically the same in the happiness and the trustworthiness judgement tasks, thus showing no differences between tasks.

Eye-movement measures: attentional engagement

To determine attentional engagement (i.e., how much each region was fixated across the 2-s display), we first analysed fixation density with a Task (2) × Dynamic Expression (6) × Region (4: left eye vs right eye vs nose/cheek vs mouth) × Interval (3: 0-to-500 [initial static period] vs 501-to-1,500 [dynamic period] vs 1,501-to-2,000 ms [final static period]) ANOVA. Main effects of task, F(1, 184) = 20.31, p < .0001, $η_{p}^{2} = . 10$ , region, F(3, 184) = 466.37, p < .0001, $η_{p}^{2} = . 88$ , and interval, F(2, 368) = 49.90, p < .0001, $η_{p}^{2} = . 21$ were qualified by a task by region interaction, F(3, 184) = 54.96, p < .0001, $η_{p}^{2} = . 47$ (see Figure 3). Pairwise independent sample t-tests compared the tasks against each other for each region. Fixation density was higher in the trustworthiness relative to the happiness task for the left eye, t(46) = 8.44, p < .0001, d = 2.44, and the right eye, t(46) = 7.21, p < .0001, d = 2.08. In contrast, the reverse occurred for the mouth, t(46) = 5.77, p < .0001, d = 1.67, and the nose region, t(46) = 4.02, p < .0001, d =1.16, with fixation density being higher in the happiness than in the trustworthiness task.

As a complementary measure, we analysed mean fixation duration (see section “Eye-movement measures”) in a Task (2) × Dynamic Expression (6) × Region (4) ANOVA. An effect of task, F(1, 38) = 5.35, p = .026, $η_{p}^{2} = . 12$ , showed that fixations were longer in the trustworthiness task (M = 243 ms) than in the happiness task (M = 195). Importantly, interactions of task with the other factors were not significant (all Fs ⩽ 1.31, ps ⩾ .28, ns). This implies that task affected attentional engagement, with more intense allocation of overt attention in the trustworthiness than in the happiness task.

Time course of eye fixations

The previous effects on fixation density were modulated by interval, as shown by interactions between interval and region, F(6, 368) = 795.46, p < .0001, $η_{p}^{2} = . 93$ , interval, region, and task, F(6, 368) = 29.99, p < .0001, $η_{p}^{2} = . 33$ , and interval, region, and expression, F(30, 1,840) = 24.63, p < .0001, $η_{p}^{2} = . 29$ . The role of interval is critical to determine the time course of selective allocation of visual attention to face regions as a function of task. Accordingly, to explore the time course in detail, we analysed fixation density across shorter (33 ms) periods over the 2-s stimulus display. This approach was particularly relevant for the aims of this study, to uncover the spatio-temporal oculomotor profile while judging happiness versus trustworthiness.

A Task (2) × Dynamic Expression (6) × Region (4) × Interval (60 consecutive 33-ms time bins) ANOVA yielded significant effects on fixation density: a task by region interaction, F(3, 184) = 52.02, p < .0001, $η_{p}^{2} = . 46$ , an interval by region by task interaction, F(177, 10,856) = 21.45, p < .0001, $η_{p}^{2} = . 26$ , and an interval by region by expression interaction, F(885, 54,280) = 18.66, p < .0001, $η_{p}^{2} = . 23$ . To examine the time course of gaze behaviour across different face regions for each task, we compared fixation densities in the happiness versus the trustworthiness task by means of t-tests for independent samples for each 33-ms time bin, expression, and region. Given the comparable patterns of effects for all the expressions ending with a smile, and for those ending with a neutral mouth, the average scores of each group of expressions are shown in Figure 4 (the time course patterns for each expression are detailed in Figures 5a to f and 6a to f of ESM_3A and B_Supplemental Materials).

The t-test comparisons showed no significant differences between tasks for the nose region. The left eye was fixated more in the trustworthiness than in the happiness task for expressions ending with a smiling mouth, between 467 and 1,233 ms from face onset, all ts(46) ⩾ 2.74, p ⩽ .009, d ⩾ 0.79, and for expressions ending with a neutral mouth, between 467 and 1,067 ms, all ts(46) ⩾ 3.36, p ⩽ .002, d ⩾ 0.97. The right eye was also fixated more in the trustworthiness than in the happiness task for expressions ending with a smile, although this occurred later, between 967 and 2,000 ms from face onset, all ts(46) ⩾ 2.87, p ⩽ .006, d ⩾ 0.83, and for expressions ending with a neutral mouth, between 1,033 and 2,000 ms, all ts(46) ⩾ 2.66, p ⩽ .011, d ⩾ 0.77. In contrast, the mouth was fixated more in the happiness than in the trustworthiness task for expressions with a final smile, between 1,000 and 1,400 ms from onset, all ts(46) ⩾ 2.65, p ⩽ .011, d ⩾ 0.77, and also for expressions ending with a neutral mouth (except for Neutral→He+Nm expressions, understandably, as the mouth did not change), between 367 and 633 ms, all ts(46) ⩾ 2.87, p ⩽ .006, d ⩾ 0.83, and between 1,067 and 1,800 ms, all ts(46) ⩾ 2.89, p ⩽ .006, d ⩾ 0.83. In sum, the fixation density amplitude (see section “Eye-movement measures”) was greater for the left eye region in the trustworthiness task, and for the mouth region in the happiness task.

Scanpaths of fixations from one face region to another

A Task (2) × Dynamic Expression (6) × Region scanpath (6: from eyes to mouth, mouth to eyes, eyes to nose, nose to eyes, mouth to nose, and from nose to mouth) ANOVA yielded significant main effects of task, F(1, 46) = 65.00, p < .0001, $η_{p}^{2} = . 59$ , expression, F(5, 230) = 43.79, p < .0001, $η_{p}^{2} = . 49$ , and region, F(5, 230) = 511.40, p < .0001, $η_{p}^{2} = . 92$ on probability of fixations. The effect of task is particularly relevant to the aims of this study. Importantly, t-tests for independent samples (happiness vs trustworthiness) showed more fixations on the eyes coming from the mouth (mean probability = 0.314 vs 0.293), t(46) = 2.14, p = .038, d = 0.62, and on the mouth coming from the eyes (mean probability = 0.427 vs 0.385), t(46) = 3.34, p = .002, d = 0.96, in the happiness task than in the trustworthiness task. The same was also the case for fixations going from the nose to the eyes or vice versa, t(46) = 2.50, p = .016, d = 0.72, and from the nose to the mouth or vice versa, t(46) = 5.11, p = .0001, d = 1.47.

Discussion

The pattern of judgement ratings and reaction times was equivalent for happiness and trustworthiness, and there was a significant correlation between tasks. This confirms the findings of prior research showing a consistent relationship between perceived happiness and trustworthiness: happy faces, and even “happy-looking” neutral faces, are judged as more trustworthy than non-happy faces (Brewer et al., 2015; Calvo et al., 2017; Centorrino et al., 2015; Engell et al., 2010; Hehman et al., 2015; Johnston et al., 2010; Krumhuber, Manstead, & Kappas, 2007; Miles, 2009; Oosterhof & Todorov, 2009; Quadflieg et al., 2013; Sutherland et al., 2017; Willis et al., 2011). Such an equivalence in the evaluation output for facial happiness and trustworthiness suggests that they could rely on the same mechanisms. In fact, both judgements involve the processing of positive affect (Oosterhof & Todorov, 2008) and share similar brain networks responsible for social-relevant (superior temporal sulci [STS]) and emotion-relevant (amygdala) information processing (Engell et al., 2010; Said et al., 2011). This study focused on visual mechanisms involving attention to the eyes and the mouth regions.

The presence (or the absence) of happy eyes and a smiling mouth affected happiness and trustworthiness judgements in the same way. Specifically (a) dynamic expressions ending with a smile were judged as both happier and more trustworthy than those ending with a neutral mouth, regardless of the eye expression; (b) a final smile in the presence of neutral eyes was judged both as less happy and trustworthy than in the presence of happy eyes; and (c) facial expressions with congruent happy eyes and a smile were judged as the most happy and trustworthy. This implies that (a) the smile plays a critical role for both judgements; (b) the eyes make a significant contribution, but only when they appear in a face with a smiling mouth; and (c) congruence between the eyes and the mouth is important for conveying happiness and trustworthiness. The final expression in the dynamic sequence seems crucial for both judgements. Nevertheless, the full dynamic display also makes a significant contribution: although the (1) Neutral-to-Happy and the (2) He+Nm-to-Happy conditions shared the same final expression, ratings were significantly higher and decision times were shorter, for the former than the letter, and this occurred for both tasks. This means that judgements are also sensitive to expressive changes from the beginning, prior to the final expression. By tracing the visual attention processes that precede such equivalent judgement products for both tasks backwards, we can obtain a detailed picture reflecting similarities as well as differences in the perception of happiness and trustworthiness.

In correspondence with the equivalent judgement ratings, we found some similarities in the visual attention processes. Attentional orienting (i.e., the time course of initial fixation on each face region) was comparable when judging happiness and trustworthiness, as shown by entry times (i.e., when the eyes and the mouth were fixated first) and initial fixation thresholds (i.e., the earliest time at which each region was fixated more than other regions). This suggests that initial orienting may be driven by an automatic mechanism that is mainly guided by stimulus characteristics, regardless of task relevance or processing strategies. This view is further strengthened by the systematic tendency to look earlier at the left visual field—particularly the left eye region—from the viewer’s perspective (thus the right side of the face), regardless of task. This reflects the natural and well-established leftward gaze bias in free-viewing tasks (Guo, Smith, Powell, & Nicholls, 2012; Peterson & Eckstein, 2012; Schurgin et al., 2014; Xiao, Quinn, Wheeler, Pascalis, & Lee, 2014). There is, however, one finding that might seem inconsistent with prior eyetracking research using static facial expressions, where the smiling mouth is generally likely to attract the initial fixation compared with any other region including the eyes (Beaudry et al., 2014; Bombari et al., 2013; Calvo & Nummenmaa, 2008). In this study, the eye region captured overt attention earlier than the mouth did. To explain these discrepancies, it must be noted that we used dynamic expressions, which, in addition, started with a smiling mouth only in 33% of trials. This implies that in most cases a smiling mouth unfolded late, and hence it could not affect initial orienting; in other words, the smiling mouth was fixated after the eyes because the smile was absent earlier.

Following the common initial orienting for happiness and trustworthiness processing, there were clear differences regarding attentional engagement. This was shown, first, by a greater fixation density on the eyes in the trustworthiness task relative to the happiness task, and greater fixation density on the mouth in the happiness task. Importantly, such selective fixation advantages extended over longer periods—as indicated by the amplitude index—for the respective task than for the other. Such selective attentional engagement as a function of the task seems plausible and can be explained in the light of prior research. The smiling mouth is a distinctive diagnostic feature of happy faces (Calder et al., 2000; Nusseck et al., 2008; Smith et al., 2005), and therefore it is understandable that visual attention is selectively allocated to the mouth when facial happiness is task-relevant. Observers tend to fixate preferentially on regions that maximise performance in determining the emotional expression, that is, the most diagnostic regions (Peterson & Eckstein, 2012; Schurgin et al., 2014). However, as a smile per se is unlikely to be diagnostic of trustworthiness, it attracts less attention when trustworthiness is task-relevant. Rather, given the importance of the eye expression for detecting the genuineness (e.g., the truly felt affect) of emotional expressions (Calvo et al., 2012; Johnston et al., 2010; Krumhuber et al., 2014; McLellan et al., 2010), and that trustworthiness is an essential component of positive face valence (Oosterhof & Todorov, 2008), it is understandable that attention is selectively allocated to the eye region when trustworthiness must be assessed. This suggests that attentional engagement mechanisms are strategic or goal-guided (Peterson & Eckstein, 2012; Schurgin et al., 2014).

A second attentional engagement difference was found for mean fixation duration, which was longer in the trustworthiness than in the happiness task. Mean fixation durations on face stimuli (e.g., Leder, Tinio, Fuchs, & Bohrn, 2010) and visual scenes (e.g., Mills, Hollingworth, Van der Stigchel, Hoffman, & Dodd, 2011) vary with task demands and increase with perceptual and cognitive processing difficulty (see Henderson, 2003; Rayner, 2009). The longer fixations in the trustworthiness task therefore suggest that trustworthiness evaluations involve a more resource-demanding process (due, for example, to insufficient information in each single fixation, and the need for integration). In contrast, facial happiness evaluations may involve easier processing, based mainly on the inspection of the smiling mouth. Thus, longer individual fixations across all face regions in the trustworthiness task would indicate more “intense” attention or effort. Given that face cues signalling trustworthiness are probably less evident than those signalling happiness, the processing “steps” (i.e., individual fixations) would in turn need enhanced attention when judging trustworthiness.

A related difference concerned the scanpaths showing a greater number of fixations from eyes to mouth and vice versa in the happiness than the in the trustworthiness task. Although initially unexpected, this finding is consistent with the fact that mean fixation durations were longer in the trustworthiness than in the happiness task: Within a limited 2-s display, longer fixations imply fewer re-fixations. Longer fixations were probably useful for configural integration of features instead of frequent re-fixations on other regions, considering that other regions could be (a) retained in iconic memory (~1 s) after one fixation, in an otherwise relatively short stimulus presentation (2 s) and (b) accessed in peripheral vision (⩽5° of visual angle between the eyes and the mouth), in an otherwise realistically sized (10.6° × 8.0°) face stimulus (see section “Stimuli”). In these conditions, re-fixations may not be necessary for configural integration as other processing strategies (longer mean fixations, probably helped by iconic memory and peripheral vision) could be used efficiently.

Another contribution of this study is the assessment of spatio-temporal oculomotor profiles for dynamic facial expressions. Measures of eye movements and fixations have been obtained in many prior studies using static facial expression stimuli (e.g., Beaudry et al., 2014; Bombari et al., 2013; Calvo & Nummenmaa, 2008; Eisenbarth & Alpers, 2011; Kanan, Bseiso, Ray, Hsiao, & Cottrell, 2015; Schurgin et al., 2014; Vaidya et al., 2014; Wells, Gillespie, & Rotshtein, 2016). Research using static expressions has found that the patterns of fixations are functional. That is, directing fixations to the facial features with greater diagnostic value predicts successful expression recognition (Peterson & Eckstein, 2012; Schurgin et al., 2014; Vaidya et al., 2014). Particularly, the first two fixations are critical for the recognition of emotional expressions (Schurgin et al., 2014) and also face identity (Hsiao & Cottrell, 2008). In the same vein, the probability that non-genuine smiles are accurately discriminated from genuine smiles depends on whether the eye or the mouth region is looked at earlier (Calvo et al., 2013). In this study, our approach involving dynamic expressions adds relevant information compared with static expressions in prior research. The spatio-temporal oculomotor profiles revealed that the amount of overt attentional engagement varies for happiness and trustworthiness processing. Differences in visual scanning suggest that the eyes are more diagnostic for trustworthiness evaluation, as can be inferred from the early and longer deployment of visual attention to this region. In contrast, the mouth expression seems more diagnostic for happiness evaluation, given the longer fixation on this region, relative to trustworthiness evaluation.

Conclusion

An unfolding smile (mainly) and happy eyes (to a lesser extent) enhance perceptions of both happiness and trustworthiness. This is reached through (only) partially overlapping visual processes for happiness and trustworthiness. Common mechanisms involve attentional orienting: entry times (i.e., time of initial fixation) and fixation thresholds (i.e., initial fixation on a region compared with others) were comparable for the eyes and mouth on both tasks. However, differences occurred in attentional engagement. First, selective visual attention patterns varied depending on the type of task, showing greater and longer fixation density on the mouth during happiness processing and on the eyes during trustworthiness processing. Second, more intense attention (mean fixation duration) was allocated to all face areas when evaluating trustworthiness than happiness, which implies the involvement of additional processing demands. In sum, selective visual attention is paid to the (smiling) mouth in judgements of happiness, whereas observers rely on selective visual attention to the eyes and allocate enhanced general (not selective) processing effort when judging trustworthiness.

Supplemental Material

ESM_1_Supplemental_Materials_Sample_Stimuli – Supplemental material for Visual attention mechanisms in happiness versus trustworthiness processing of facial expressions

Supplemental material, ESM_1_Supplemental_Materials_Sample_Stimuli for Visual attention mechanisms in happiness versus trustworthiness processing of facial expressions by Manuel G Calvo, Eva G Krumhuber and Andrés Fernández-Martín in Quarterly Journal of Experimental Psychology

Supplemental Material

ESM_2A_Supplemental_Dataset_Participants – Supplemental material for Visual attention mechanisms in happiness versus trustworthiness processing of facial expressions

Supplemental material, ESM_2A_Supplemental_Dataset_Participants for Visual attention mechanisms in happiness versus trustworthiness processing of facial expressions by Manuel G Calvo, Eva G Krumhuber and Andrés Fernández-Martín in Quarterly Journal of Experimental Psychology

Supplemental Material

ESM_2B_Supplemental_Dataset_Items_(Fixation_Density) – Supplemental material for Visual attention mechanisms in happiness versus trustworthiness processing of facial expressions

Supplemental material, ESM_2B_Supplemental_Dataset_Items_(Fixation_Density) for Visual attention mechanisms in happiness versus trustworthiness processing of facial expressions by Manuel G Calvo, Eva G Krumhuber and Andrés Fernández-Martín in Quarterly Journal of Experimental Psychology

Supplemental Material

ESM_3A_Supplemental_Materials_Figures_5A_to_F – Supplemental material for Visual attention mechanisms in happiness versus trustworthiness processing of facial expressions

Supplemental material, ESM_3A_Supplemental_Materials_Figures_5A_to_F for Visual attention mechanisms in happiness versus trustworthiness processing of facial expressions by Manuel G Calvo, Eva G Krumhuber and Andrés Fernández-Martín in Quarterly Journal of Experimental Psychology

Supplemental Material

ESM_3B_Supplemental_Materials_Figures_6A_to_F – Supplemental material for Visual attention mechanisms in happiness versus trustworthiness processing of facial expressions

Supplemental material, ESM_3B_Supplemental_Materials_Figures_6A_to_F for Visual attention mechanisms in happiness versus trustworthiness processing of facial expressions by Manuel G Calvo, Eva G Krumhuber and Andrés Fernández-Martín in Quarterly Journal of Experimental Psychology

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

This study was funded by Grant PSI2014-54720P, awarded to Manuel G. Calvo, from the Spanish Ministerio de Economía y Competitividad, Secretaría de Estado de Investigación, Desarrollo e Innovación.

Supplementary material

Supplementary material is available at .

Notes

References

Adolphs

(2002). Recognizing emotion from facial expressions: Psychological and neurological mechanisms. Behavioral and Cognitive Neuroscience Reviews, 1, 21–62.

Bartlett

Whitehill

(2011). Automated facial expression measurement: Recent applications to basic research in human behavior, learning, and education. In Calder

Rhodes

Johnson

Haxby

(Eds.), Handbook of face perception (pp. 489–513). Oxford, UK: Oxford University Press.

Beaudry

Roy-Charland

Perron

Cormier

Tapp

(2014). Featural processing in recognition of emotional facial expressions. Cognition & Emotion, 28, 416–432.

Beltrán

Calvo

M. G.

(2015). Brain signatures of perceiving a smile: Time course and source localization. Human Brain Mapping, 36, 4287–4303.

Bindemann

Scheepers

Burton

A. M.

(2009). Viewpoint and center of gravity affect eye movements to human faces. Journal of Vision, 927, 1–16.

Bombari

Schmid

P. C.

Schmid-Mast

Birri

Mast

F. W.

Lobmaier

J. S.

(2013). Emotion recognition: The role of featural and configural face information. The Quarterly Journal of Experimental Psychology, 66, 2426–2442.

Brewer

Collins

Cook

Bird

(2015). Atypical trait inferences from facial cues in alexithymia. Emotion, 15, 637–643.

Calder

A. J.

Young

A. W.

Keane

Dean

(2000). Configural information in facial expression perception. Journal of Experimental Psychology: Human Perception and Performance, 26, 527–551.

Calvo

M. G.

Álvarez-Plaza

Fernández-Martín

(2017). The contribution of facial regions to judgments of happiness and trustworthiness from dynamic expressions. Journal of Cognitive Psychology, 29, 618–625.

10.

Calvo

M. G.

Avero

Fernández-Martín

Recio

(2016). Recognition thresholds for static and dynamic emotional faces. Emotion, 16, 1186–1200.

11.

Calvo

M. G.

Beltrán

(2014). Brain lateralization of holistic versus analytic processing of emotional facial expressions. Neuroimage, 92, 237–247.

12.

Calvo

M. G.

Beltrán

Fernández-Martín

(2014). Early processing of happy facial expressions in peripheral vision: Neurophysiological evidence. Biological Psychology, 100, 60–70.

13.

Calvo

M. G.

Fernández-Martín

Nummenmaa

(2012). Perceptual, categorical, and affective processing of ambiguous smiling facial expressions. Cognition, 125, 373–393.

14.

Calvo

M. G.

Fernández-Martín

Nummenmaa

(2014). Facial expression recognition in peripheral versus central vision: Role of the eyes and the mouth. Psychological Research, 78, 180–195.

15.

Calvo

M. G.

Gutiérrez-García

Avero

Lundqvist

(2013). Attentional mechanisms in judging genuine and fake smiles: Eye-movement patterns. Emotion, 13, 792–802.

16.

Calvo

M. G.

Gutiérrez-García

Fernández-Martín

Nummenmaa

(2014). Recognition of facial expressions of emotion is related to their frequency in everyday life. Journal of Nonverbal Behavior, 38, 549–567.

17.

Calvo

M. G.

Nummenmaa

(2008). Detection of emotional faces: Salient physical features guide effective visual search. Journal of Experimental Psychology: General, 137, 471–494.

18.

Calvo

M. G.

Nummenmaa

(2016). Perceptual and affective mechanisms in facial expression recognition: An integrative review. Cognition & Emotion, 30, 1081–1106.

19.

Centorrino

Djemai

Hopfensitz

Milinski

Seabright

(2015). Honest signaling in trust interactions: Smiles rated as genuine induce trust and signal higher learning opportunities. Evolution and Human Behavior, 36, 8–16.

20.

Cohn

J. F.

De la Torre

(2015). Automated face analysis for affective computing. In Calvo

R.A.

DiMello

Gratch

Kappas

(Eds.), The Oxford handbook of affective computing (pp. 131–150). New York, NY: Oxford University Press.

21.

Del Giudice

Colle

. (2007). Differences between children and adults in the recognition of enjoyment smiles. Developmental Psychology, 43, 796–803.

22.

Eisenbarth

Alpers

G. W.

(2011). Happy mouth and sad eyes: Scanning emotional facial expressions. Emotion, 11, 860–865.

23.

Ekman

Friesen

W. V.

Hager

J. C.

(2002). Facial action coding system: Investigator’s guide. Salt Lake City, UT: Human Face.

24.

Engell

A. D.

Todorov

Haxby

J. V.

(2010). Common neural mechanisms for the evaluation of facial trustworthiness and emotional expressions as revealed by behavioral adaptation. Perception, 39, 931–941.

25.

Guo

Smith

Powell

Nicholls

(2012). Consistent left gaze bias in processing different facial cues. Psychological Research, 76, 263–269.

26.

Hehman

Flake

J. K.

Freeman

J. B.

(2015). Static and dynamic facial cues differentially affect the consistency of social evaluations. Personality and Social Psychology Bulletin, 41, 1123–1134.

27.

Henderson

J. M.

(2003). Human gaze control during real-world scene perception. Trends in Cognitive Sciences, 7, 498–504.

28.

Hoffmann

Traue

H. C.

Bachmayr

Kessler

(2010). Perceived realism of dynamic facial expressions of emotion: Optimal durations for the presentation of emotional onsets and offsets. Cognition & Emotion, 24, 1369–1378.

29.

Hsiao

J. H. W.

Cottrell

(2008). Two fixations suffice in face recognition. Psychological Science, 19, 998–1006.

30.

iMotions. (2016). Facial expression analysis: The complete pocket guide. Retrieved from https://imotions.com/blog/facial-expression-analysis/

31.

Johnston

Miles

Macrae

(2010). Why are you smiling at me? Social functions of enjoyment and non-enjoyment smiles. British Journal of Social Psychology, 49, 107–127.

32.

Johnston

Mayes

Hughes

Young

(2013). Brain networks subserving the evaluation of static and dynamic facial expressions. Cortex, 49, 2462–2472.

33.

Kanan

Bseiso

D. N. F.

Ray

N. A.

Hsiao

J. H.

Cottrell

G. W.

(2015). Humans have idiosyncratic and task-specific scanpaths for judging faces. Vision Research, 108, 67–76.

34.

Krumhuber

E. G.

Kappas

Manstead

A. S. R.

(2013). Effects of dynamic aspects of facial expressions: A review. Emotion Review, 5, 41–46.

35.

Krumhuber

E. G.

Likowski

K. U.

Weyers

(2014). Facial mimicry of spontaneous and deliberate Duchenne and non-Duchenne smiles. Journal of Nonverbal Behavior, 38, 1–11.

36.

Krumhuber

E. G.

Manstead

A. S. R.

Kappas

(2007). Temporal aspects of facial displays in person and expression perception: The effects of smile dynamics, head-tilt and gender. Journal of Nonverbal Behavior, 31, 39–56.

37.

Krumhuber

E. G.

Manstead

A. S. R.

Kappas

Cosker

Marshall

Rosin

P. L.

(2007). Facial dynamics as indicators of trustworthiness and cooperative behavior. Emotion, 7, 730–735.

38.

Krumhuber

E. G.

Scherer

K. R.

(2016). The look of fear from the eyes varies with the dynamic sequence of facial actions. Swiss Journal of Psychology, 75, 5–14.

39.

Leder

Tinio

P. P. L.

Fuchs

I. M.

Bohrn

(2010). When attractiveness demands longer looks: The effects of situation and gender. The Quarterly Journal of Experimental Psychology, 63, 1858–1871.

40.

Leppänen

Hietanen

J. K.

(2007). Is there more in a happy face than just a big smile? Visual Cognition, 15, 468–490.

41.

Lundqvist

Flykt

Öhman

(1998). The Karolinska directed emotional faces–KDEF. Stockholm, Sweden: Department of Clinical Neuroscience, Psychology section, Karolinska Institutet (CD-ROM).

42.

Manssuer

L. R.

Roberts

M. V.

Tipper

S. P.

(2015). The late positive potential indexes a role for emotion during learning of trust from eye-gaze cues. Social Neuroscience, 10, 635–650.

43.

McLellan

Johnston

Dalrymple-Alford

Porter

(2010). Sensitivity to genuine vs. posed emotion specified in facial displays. Cognition & Emotion, 24, 1277–1292.

44.

Miles

L. K.

(2009). Who is approachable? Journal of Experimental Social Psychology, 45, 262–266.

45.

Mills

Hollingworth

Van der Stigchel

Hoffman

Dodd

M. D.

(2011). Examining the influence of task set on eye movements and fixations. Journal of Vision, 11, 171–115. doi:10.1167/11.8.17

46.

Nelson

N. L.

Russell

J. A.

(2013). Universality revisited. Emotion Review, 5, 8–15.

47.

Nusseck

Cunningham

D. W.

Wallraven

Bülthoff

H. H.

(2008.). The contribution of different facial regions to the recognition of conversational expressions. Journal of Vision, 8, 1–23. doi:10.1167/8.8.1

48.

Oosterhof

N. N.

Todorov

(2008). The functional basis of face evaluation. Proceedings of the National Academy of Sciences of the United States of America, 105, 11087–11092. doi:10.1073/pnas.0805664105

49.

Oosterhof

N. N.

Todorov

(2009). Shared perceptual basis of emotional expressions and trustworthiness impressions from faces. Emotion, 9, 128–133.

50.

Peterson

M. F.

Eckstein

M. P.

(2012). Looking just below the eyes is optimal across face recognition tasks. Proceedings of the National Academy of Sciences of the United States of America, 109, 3314–3323. doi:10.1073/pnas.1214269109

51.

Quadflieg

Vermeulen

Rossion

(2013). Differential reliance on the Duchenne marker during smile evaluations and person judgments. Journal of Nonverbal Behavior, 37, 69–77.

52.

Rayner

(2009). The Thirty Fifth Sir Frederic Bartlett Lecture: Eye movements and attention in reading, scene perception, and visual search. The Quarterly Journal of Experimental Psychology, 62, 1457–1506.

53.

Said

C. P.

Haxby

J. V.

Todorov

(2011). Brain system for assessing the affective value of faces. Philosophical Transaction of the Royal Society, 366, 1660–1670.

54.

Schultz

Pilz

K. S.

(2009). Natural facial motion enhances cortical responses to faces. Experimental Brain Research, 194, 465–475.

55.

Schurgin

M. W.

Nelson

Iida

Ohira

Chiao

J. Y.

Franconeri

S. L.

(2014). Eye movements during emotion recognition in faces. Journal of Vision, 14, 1–16. doi:10.1167/14.13.14

56.

Smith

M. L.

Cottrell

G. W.

Gosselin

Schyns

P. G.

(2005). Transmitting and decoding facial expressions. Psychological Science, 16, 184–189.

57.

Strachan

J. W. A.

Kirkham

A. J.

Manssuer

L. R.

Tipper

S. P.

(2016). Incidental learning of trust: Examining the role of emotion and visuomotor fluency. Journal of Experimental Psychology: Learning, Memory, and Cognition, 42, 1759–1773.

58.

Sutherland

C. A. M.

Young

A. W.

Rhodes

(2017). Facial first impressions from another angle: How social judgements are influenced by changeable and invariant facial properties. British Journal of Psychology, 108, 397–415.

59.

Tanaka

J. W.

Kaiser

Butler

Le Grand

(2012). Mixed emotions: Holistic and analytic perception of facial expressions. Cognition & Emotion, 26, 961–977.

60.

Vaidya

A. R.

Jin

Fellows

L. K.

(2014). Eye spy: The predictive value of fixation patterns in detecting subtle and extreme emotions from faces. Cognition, 133, 443–456.

61.

Wells

L. J.

Gillespie

S. M.

Rotshtein

(2016). Identification of emotional facial expressions: Effects of expression, intensity, and sex on eye gaze. PLoS ONE, 11, e0168307. doi:10.1371/journal.pone.0168307

62.

Willis

M. L.

Palermo

Burke

(2011). Social judgments are influenced by both facial expression and direction of eye gaze. Social Cognition, 29, 415–429.

63.

Wingenbach

T. S. H.

Ashwin

Brosnan

(2016). Validation of the Amsterdam Dynamic Facial Expression Set—Bath Intensity Variations (ADFES-BIV): A set of videos expressing low, intermediate, and high intensity emotions. PLoS ONE, 11, e0147112. doi:10.1371/journal.pone.0147112

64.

Winkielman

Olszanowski

Gola

(2015). Faces in-between: Evaluations reflect the interplay of facial features and task-dependent fluency. Emotion, 15, 232–242.

65.

Xiao

N. G.

Quinn

P. C.

Wheeler

Pascalis

Lee

(2014). Natural, but not artificial, facial movements elicit the left visual field bias in infant face scanning. Neuropsychologia, 62, 175–183.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

1.09 MB

0.07 MB

0.48 MB

0.25 MB

0.24 MB