The role of facial expressions in attention-orienting in adults and infants

Abstract

Faces convey many signals (i.e., gaze or expressions) essential for interpersonal interaction. We have previously shown that facial expressions of emotion and gaze direction are processed and integrated in specific combinations early in life. These findings open a number of developmental questions and specifically in this paper we address whether such emotional signals may modulate the behavior in a following gaze context. A classic spatial cueing paradigm was used to assess whether different facial expressions may cause differential orienting response times and modulate the visual response to a peripheral target in adults and in 4-month-old infants. Results showed that both adults and infants oriented towards a peripheral target when a central face was gazing in the direction of the target location. However, in adults this effect occurred regardless of the facial expression displayed by the face. In contrast, in infants, the emotional facial expressions used, at least in the current study, did not facilitate the attention shift but tended to hold infants’ attention.

Keywords

development of face processing and emotion eye tracker infants

Eye gaze direction is an important indicator of another individual’s focus of attention (Kleinke, 1986). Several studies have demonstrated that another person’s eye gaze, as well as head position and pointing gestures, automatically shift the observer’s attention in the direction indicated by these cues (Friesen and Kingstone, 1998; Langton and Bruce, 1999; Langton, Watt, & Bruce, 2000). Studies based on Posner’s classic spatial cueing paradigm (Posner, 1980) found that lateralized targets are detected faster when a central face cue is gazing in the direction of the target location than when the eyes are directed to the opposite side.

Previous research has considered the interactions between gaze direction and emotional expression in attention shifting tasks but with inconsistent results. For instance, a few studies reported an interaction effect between facial expression and the congruence of the gaze cue with the target location (i.e., faster attention shifting selective of fearful expression in Putman, Hermans, & van Honk, 2006; and Tipples, 2006; faster attention shifting of happy expression in Hori et al., 2005; faster attention shifting of fearful and surprised expressions in Bayless, Glover, Taylor, & Itier, 2011). However, other studies found a beneficial effect of the emotional expression of fear only in anxious participants (e.g., Fox, Mathews, Calder, & Yiend, 2007). In contrast, many other works showed the gaze cueing effect to be independent of facial expression (Hietanen and Leppänen, 2003; Bayliss, Frischen, Fenske, & Tipper, 2007; Pecchinenda, Pes, Ferlazzo, & Zoccolotti, 2008). Additional lines of evidence suggest that rather than enhancing attention-orienting away from the face, emotionally expressive faces (e.g., angry faces, Fox et al., 2000) may inhibit orienting, suggesting that threat-related stimuli do not affect the shift component but rather the disengaging component of attention.

Even though some of the results from previous studies correlate, there are still big discrepancies in the findings and conclusions of earlier research, and the methodological causes are not yet clear. The same occurs in infancy research, where despite the evidence that infants are able to follow the gaze in a face-stimulus context, the relationship with affective facial expression is still uncertain. For instance, one study provided evidence that in 7-month-old infants, an adult’s averted gaze, combined with a neutral expression, was more effective in engaging attention than an emotional expression (e.g., happy or sad, Flom and Pick, 2005). The authors proposed that infants at this age may not comprehend the referential nature of affective expression or that they have difficulty in perceiving or attending to a target looked by another person when potentially competing information is also present, i.e., emotional display. Conversely, another study revealed beneficial effects of the emotional expression in attention-orienting tasks (De Groote, Roeyers, & Striano 2007); these authors showed that 3-month-old infants followed gaze more reliably when an experimenter displayed a disgusted expression rather than a happy expression. One interpretation of this result is that the infants looked away from the face displaying disgust more than from the happy face, so that they might have noticed the target by coincidence when gazing away. Six- and 9-month-old infants followed the gaze in both conditions, i.e., under happy and disgust expressions; however the younger group looked more quickly to the target when the experimenter posed a happy expression, while the older infants looked equally quickly in the two conditions. Facial expressions in this study were accompanied with corresponding vocal expressions, and the gaze direction was combined with head motion. These confounds may have affected the results, therefore direct comparisons with the Flom and Pick (2005) study is unwarranted. In a different paradigm, Phillips, Wellman and Spelke (2002) showed that 12-month-old infants, but not 8-month-olds, are able to connect the looking-behaviour of a person with his/her subsequent action when accompanied by a positive expression and a positive toned vocalization. Taken together, these works suggest that gaze following is a gradually developing skill deeply connected with the other emerging skills in infants, such as emotional expression perception, joint attention, deciphering the others’ affects and states of mind. Eye gaze cueing in infants seems to engage general mechanisms that become incorporated into a more specific system in adulthood. Noticeably, gaze orienting effects were found to depend on the perception of apparent eye motion in young infants (Farroni, Johnson, Brockbank, & Simion, 2000; Farroni, Mansfield, Lai, & Johnson, 2003) but not in adults (Hietanen & Leppänen, 2003; Schuller & Rossion, 2001, 2004).

In the current study, Posner’s classic spatial cueing paradigm was used to investigate whether the attention-orienting by eye gaze direction is modulated by facial expressions and to examine how this ability develops. Adults (Experiment 1) and 4-month-old infants (Experiment 2) were presented with three different facial expressions (neutral, happy and fearful), combined with averted gaze, which indicated the location of the target (valid condition) or the opposite location (invalid condition). The congruence between gaze direction and target location was expected to affect the reaction time, causing shorter responses when the eye gaze cue was directed to the same side as the target than to the opposite location in both groups of participants. In the light of the available literature, different outcomes could be predicted for a facial expression effect in adults. First, if emotional expressions have an advantage over neutral face stimuli in attention-orienting (Putman, Hermans, & van Honk, 2006; Tipples, 2006; Hori et al., 2005), faster reaction times to the target following a happy and a fearful rather than a neutral face can be expected. On the other hand, if emotional facial expressions tend to hold the viewer’s attention (Fox et al., 2000), shorter orienting responses following neutral faces than happy and fearful expressions can be expected. Finally, if eye gaze direction processing for attention shift is independent of facial expression analysis, then no effect of facial expression will be found (Hietanen and Leppänen, 2003). Since evidence from studies with infants indicate that either neutral or happy facial expressions are more effective in engaging their attention (Flom and Pick, 2005; De Groote et al., 2007), attention-orienting triggered by a fearful expression was predicted to result in an overall lengthening of the reaction times as compared with the responses cued by happy or neutral faces.

Experiment 1

Method

Participants

Thirty-two students at the University of Padua, Department of Psychology (20 females, 12 males), aged between 18 and 52 years (mean age 23.8 years), volunteered in the study. An additional 11 adults were tested, but were excluded from the final sample due to technical errors. All subjects had normal or corrected-to-normal vision.

Stimuli

The stimuli were four different female faces, displaying neutral, happy or fearful expressions, selected from the MacBrain Face Stimulus Set. In order to display different gaze directions, pictures have been modified with Adobe Photoshop 7.0. The face stimulus subtended a visual angle of 13 × 10.4 degrees.

Procedure

The participant sat on a chair at a distance of 90 cm from a high-resolution computer monitor (17”), with its eyes at the level of the monitor centre. A curtain was drawn so that the only visual stimulations that could attract his/her attention were the ones displayed on the screen.

The participant’s eye movements were recorded by a remote eye tracker system called the Eye Position Detector System (EPDS, Massaccesi, 2004).

Three markers in different positions (top-left, centre, bottom-right) of the screen were used to calibrate the participant’s saccades. After calibration, the trial began with a cross in the middle of the screen for 300 ms. Then, the first face stimulus appeared. This was a neutral face accompanied with direct gaze and displayed until the subject fixated on it, between 200 and 700 ms. Following this, the cue appeared for 300 ms. The cue was the same facial model as before but displaying either the same neutral expression or an emotional (fearful or a happy) expression, accompanied with averted gaze (left or right). Finally the target appeared, on the left or on the right of the screen, in a valid or an invalid position relative to the gaze cue. The participants were not aware of the aim of the study and were instructed to fixate on the center of the screen and orient towards the target face as soon as it appeared. They were also told that the gaze direction of the central face was not predictive of the target location. Targets were displayed until they were visually reached by the participants or for 800 ms at most (Figure 1). Each facial expression was randomly presented 16 times, while the gaze direction, the position of the target (left or right) and the congruence with the gaze cue (valid or invalid) were counterbalanced across the trials.

Figure 1.

Paradigm used in Experiment 1. Example of valid condition.

The reaction times spent by the subjects to visually reach the peripheral target, i.e., eye movement time from the central area to the target area of interest (AOI, Figure 2), were measured in 12 conditions (3 Emotional expressions × 2 Gaze direction × 2 Target locations), each repeated 4 times, for a total of 48 trials. The eye tracker software automatically coded the eye movement data during the experiment and all subjects were recorded by a video camera.

Figure 2.

Areas of Interest (AOI); AOI 1 and 5 are the target areas, AOI 3 is the central area.

Data processing and statistical analysis

Trials were excluded from the analysis if the response times to reach the target area were less than 80 ms or more than 800 ms. Furthermore, if more than 10 trials were rejected, the participant was excluded from the analysis. The reaction times were analysed by a 3 × 2 repeated-measure analysis of variance (ANOVA) with facial expression (neutral, happy and fearful) and condition (valid and invalid) as within-subject factors.

Results

The ANOVA revealed a main effect of the condition (F _(1,31) = 60.15, p < .001). The participants were faster to reach valid (mean = 278.24 ms) than invalid (mean = 308.92 ms) targets (see Table 1). No other significant effects were found.

Table 1.

Means and Standard Deviations of adults’ reaction time (Experiment 1).

Expression	Condition	Mean (ms)
Expression	Condition	Mean (ms)	SD (ms)
Neutral	Valid	280.55	37.74
Neutral	Invalid	310.3	34.92
Fearful	Valid	275.96	38.72
Fearful	Invalid	309.92	51.99
Happy	Valid	278.21	30.71
Happy	Invalid	306.55	44.68

This result confirmed the gaze cueing effect, as lateralized targets were detected faster when the central face was gazing in the direction of the target location (valid condition) than when the eyes were directed to the opposite side (invalid condition). There was no effect of facial expression, confirming that the emotional expression of the face producing the gaze shift does not influence the magnitude of cueing.

Experiment 2

Method

Participants

Thirty-three healthy 4-month-old infants, aged between 124 and 151 days (mean age 135 days) participated in the study (Experiment 2). Infants were selected from a list of the Birth Registry Office of Padua. The testing took place only if the infant was awake and in an alert state. Informed consent was obtained from the parents. After excluding 19 infants (9 due to fussiness, 2 due to technical errors and 8 because they reached the exclusion criterion), the final sample consisted of 14 infants (5 females and 9 males).

Stimuli

The stimuli were the same as in Experiment 1. The face stimulus subtended a visual angle of 19.5 × 15.6 degrees.

Procedure

The infant sat in an adapted infant car seat at a distance of 60 cm from a high-resolution computer monitor (17”), with his/her eye level aligned with the centre of the screen. Once the infant was positioned on the car seat, dynamic cartoons displayed in three different positions (top-left, centre, bottom-right) of the screen were used to attract infants’ attention in order to calibrate their saccades. The design and the procedure were the same as used in Experiment 1 but with slightly different stimulus presentation times in order to fit the infants’ age. The first stimulus was a fluttering colour cartoon, used to attract the infant’s attention and presented for 1000 ms. Then, a neutral face accompanied with direct gaze appeared for 700 ms. This period of mutual gaze was included to ensure the gaze cueing effect (Farroni et al., 2000, 2003). Following this, the cue appeared for 1500 ms. The cue was the same face model as before, but displaying either the same neutral expression, or an emotional (fearful or a happy) expression, accompanied with averted gaze (left or right). Finally the target appeared, on the left or on the right of the screen, in a valid or an invalid position relative to the gaze cue. Targets were different cartoon pictures shown until they were visually reached by the infant or for 1500 ms at most. Each facial expression was randomly presented 16 times, while the gaze direction, the position (left or right) of the target, and the congruence with the gaze cue (congruent or incongruent) were counterbalanced across the trials. Sounds were occasionally used to maintain infants’ attention throughout the experiment.

We measured the reaction times spent by the infants to visually reach the AOI of the peripheral target in 12 conditions, each repeated 4 times, for a maximum of 48 trials. Infants included in the final sample performed at least 2 trials per condition. The eye tracker software automatically coded the eye movement data during the experiment and all infants were recorded by a video camera.

Results

As for adults in Experiment 1, a 3 × 2 repeated measure analyses of variance (ANOVA) with facial expression (neutral, happy and fearful) and condition (valid and invalid) as within subject factors was performed. A significant effect of condition (F _(1,13) = 6.23, p < .03) showed faster response times to the targets in the valid (M = 303.45 ms) than in the invalid location (M = 339.53 ms). Additionally, a significant effect of facial expression was found (F _(2,26)= 5.649, p = .009), revealing faster reaction times following neutral expressions (M = 288.17 ms) than happy (M = 333.92 ms) and fearful (M = 342.38 ms) emotional expressions (see Table 2).

Table 2.

Means and Standard Deviations of infants’ reaction time and of number of correct trials (Experiment 2).

Expression	Condition	Mean (ms)
Expression	Condition	Mean (ms)	SD (ms)	Mean (N)	SD (N)
Neutral	Valid	265.86	102.36	4.2	1.66
Neutral	Invalid	310.47	111.42	4	1.6
Fearful	Valid	324.4	64.71	4.87	1.13
Fearful	Invalid	360.36	114.39	4.2	1.61
Happy	Valid	320.08	120.44	4.6	1.55
Happy	Invalid	347.77	111.19	4.93	1.28

These results confirm that infants at 4 months of age are already able to use others’ gaze direction to shift their attention. In addition, the facial expression affects attention-orienting in infants, as slower reaction times to the target following emotional expressions than neutral faces were recorded. Noticeably, not only the fearful expression, but also the expression of happiness captured the attention of the infant, thereby slowing down the orienting of attention toward the target.

General discussion

In the present study, the role of facial emotional expressions in attention-orienting by gaze cues in adults and 4-month-old infants was examined.

The results from adults showed faster reaction times in the valid condition, suggesting that the congruence between the direction of the gaze and the location of the peripheral target facilitated the shift of the participant’s attention. This effect was not modulated by the expression displayed by the face. Furthermore, no effect of emotion expression on attention-orienting was found. These results indicate that attentional orienting is independent of emotional expression processing, confirming the findings from Hietanen and Leppänen (2003). In contrast, the infants showed both an effect of attention-orienting by gaze and an effect of facial expression, with faster reaction times in the valid than invalid conditions, and faster attention shifting with neutral than emotional (both happy and fearful) expressions. This finding suggests that at least the emotional facial expressions used in the current study do not facilitate attention-orienting but rather tend to hold the infants’ attention.

The findings from the adults are in contrast with those studies that suggested that emotional facial expressions modulate the effect of attention-orienting (Putman et al., 2006; Tipples, 2006). Yet, such modulation has been inconsistently reported: Hori et al. (2005) found it to be selective of happy faces whereas Tipples et al. found it selective for fearful faces, and several studies did not find evidence for any emotional modulation of the gaze cueing effect (e.g., Hietanen and Leppänen, 2003). A possible source of data variability seems to be anxiety level: several studies reported the influence of emotion on gaze cueing to be dependent on the participants’ anxiety level (e.g., Fox et al., 2000; Mathews, Fox, Yiend, & Calder, 2003; Putman et al., 2006; but see also Bayless et al., 2011). Here, the anxiety level of participants was not measured. One can therefore only speculate about the contribution of this variable to our results. Another modulating variable may be the task at hand (Pecchinenda et al., 2008). Tipples (2006) used a discrimination task while we used a simple detection task. The latter task was necessary to run comparable studies in infants and adults but it may not be best suited to reveal an interaction between emotion- and gaze-processing in attention-cueing (Pecchinenda et al., 2008). The dynamics of gaze and emotion changes in the cue may constitute another important aspect (Bayless et al., 2011; Putman et al., 2006). Finally, another modulating variable may be the number of face models used. Four different models were used in the present study, while Tipples (2006) used only one facial identity. It is also notable that in the studies where the effect of emotional expression was not found (Mathews et al., 2003; Hietanen and Leppänen, 2003), more than one facial identity was used. The interdependence between identity and emotional expression-processing is sustained by several behavioural and neuroimaging studies that found that a change in one of these dimensions impairs the ability to detect changes in the other one (Ganel, Goshen-Gottstein, & Goodale, 2005; Winston, Henson, FineGoulden, & Dolan, 2004), and that perceivers can use facial identity as a reference to compute expression and vice versa (Ganel and Goshen-Gottstein, 2004). Ganel and colleagues (2004, 2005) proposed that “the way an individual expresses emotion is always constrained by his or her identity. Differences between individuals should lead to systematic differences in the way they express emotions” (Ganel et al., 2005, p. 1645). This implies complex interactions between emotional expression and identity processing, which may be further complicated by task requirements. Taken together with the previous studies on attention-orienting by emotional and neutral face gaze, the present study suggests that, in attentional orienting tasks, adults’ performance may be modulated by facial expression only under specific conditions that include displaying expressions by always the same face identity.

On the contrary, infants’ responses were affected by emotional expression. As in Flom and Pick (2005), the neutral expression was found to reduce the overall response time to targets as compared to other emotional expressions. Therefore, infants may be more attracted by emotional expressions that, in turn, may hold their attention more than a neutral face, thus impeding orienting. However, in order to better understand the results from infants, we should also take into account that the condition where neutral faces were displayed was methodologically different from the conditions in which fearful and happy expressions were presented. In fact, when the cue was a neutral face, the only feature that changed from the previous face displayed was gaze direction, from direct to averted. In contrast, when the cue stimulus was an emotional expression, both gaze direction and facial expression changed, from direct to averted gaze, and from a neutral expression to an emotional one. As a consequence, given the smaller number of perceptual changes, the neutral condition could have been processed more easily, or at least more quickly, than the emotional expressions. While this may be seen as a limitation of the paradigm used in this study, it is worth to note that the different number of changes in the neutral and the emotion conditions made no difference on the gaze shift and magnitude of cueing in the adults’ performances.

The current findings are inconsistent with those from De Groote et al. (2007) and Phillips et al. (2002), who found that infants gained benefit from happy expressions. Nonetheless, it is noteworthy that in the former study, a happy expression was compared only with the expression of disgust while a neutral condition was not included, whereas in the latter study, much older infants were tested (12-month-olds). It is therefore difficult to compare directly across these studies and ours.

In sum, emotional expressions modulated attention-orienting responses to gaze cues in infants by slowing down the orienting. This emphasizes the power of emotional expressions in grabbing the infants’ attention. This effect was additional to the gaze-orienting effect, suggesting parallel processing of facial expression and gaze direction cues. In contrast, attention-orienting was not affected by emotional expression in adults. This may be tentatively related to the intricate processing of face emotion and identity in adults. It is also possible that it reflects a greater task dependency of the interaction across facial components. Overall, our results are in support of the view of complex interactions between emotional expressions and gaze processing in attention-orienting, which seem to undergo changes in the course of development.

Footnotes

Acknowledgements

The authors thank the parents and infants who participated in the study, and Stefano Massaccesi for the help with the software and for creating the eye tracker system.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

This work was supported by the Ateneo degli studi di Padova to Teresa Farroni; the CNRS and an ANR grant (Project Impress) to Nathalie George; the Marie Curie Fellowship [MEST-CT-2005-020725 to Silvia Rigato]; the European Research Council under the European Community's Seventh Framework Programme (FP7/2007-2013) / ERC Grant agreement no. 241242.

References

Bayless

S. J.

Glover

Taylor

M. J.

Itier

R. J.

(2011). Is it in the eyes? Dissociating the role of emotion and perceptual features of emotionally expressive faces in modulating orienting to eye gaze. Visual Cognition, 19(4), 483–510.

Bayliss

A. P.

Frischen

Fenske

M. J.

Tipper

S. P.

(2007). Affective evaluations of objects are influenced by observed gaze direction and emotional expression. Cognition, 104(3), 644–653.

De Groote

Roeyers

Striano

(2007). Gaze following as a function of affective expression in 3-, 6- and 9-month-old infants. Infant Behavior & Development, 30(3), 492–498.

Farroni

Johnson

M. H.

Brockbank

Simion

(2000). Infants’ use of gaze direction to cue attention: The importance of perceived motion. Visual Cognition, 7(6), 705–718.

Farroni

Mansfield

E. M.

Lai

Johnson

M. H.

(2003). Infants perceiving and acting on the eyes: Tests of an evolutionary hypothesis. Journal of Experimental Child Psychology, 85(3), 199–212.

Flom

Pick

A. D.

(2005). Experimenter affective expression and gaze following in 7-month-olds. Infancy, 7(2), 207–218.

Fox

Lester

Russo

Bowles

R. J.

Pichler

Dutton

(2000). Facial expressions of emotion: Are angry faces detected more efficiently? Cognition and Emotion, 14(1), 61–92.

Fox

Mathews

Calder

A. J.

Yiend

(2007). Anxiety and sensitivity to gaze direction in emotionally expressive faces. Emotion, 7(3), 478–486. doi:10.1037/1528-3542.7.3.478.

Friesen

C. K.

Kingstone

(1998). The eyes have it! Reflexive orienting is triggered by nonpredictive gaze. Psychonomic Bulletin & Review, 5(3), 490–495.

10.

Ganel

Goshen-Gottstein

(2004). Effects of familiarity on the perceptual integrality of the identity and expression of faces: The parallel-route hypothesis revisited. Journal of Experimental Psychology: Human Perception and Performance, 30(3), 583–597.

11.

Ganel

Goshen-Gottstein

Goodale

M. A.

(2005). Interactions between the processing of gaze direction and facial expression. Vision Research, 45(9), 1191–1200.

12.

Hietanen

J. K.

Leppänen

J. M.

(2003). Does facial expression affect attention orienting by gaze direction cues? Journal of Experimental Psychology: Human Perception and Performance, 29(6), 1228–1243.

13.

Hori

Tazumi

Umeno

Kamachi

Kobayashi

Ono

Nishijo

(2005). Effects of facial expression on shared attention mechanisms. Physiology & Behavior, 84(3), 397–405.

14.

Kleinke

C. L.

(1986). Gaze and eye contact: A research review. Psychological Bulletin, 100(1), 78–100.

15.

Langton

S. R. H.

Bruce

(1999). Reflexive visual orienting in response to the social attention of others. Visual Cognition, 6(5), 541–567.

16.

Langton

S. R. H.

Watt

R. J.

Bruce

(2000). Do the eyes have it? Cues to the direction of social attention. Trends in Cognitive Sciences, 4(2), 50–59.

17.

Massaccesi

(2004). Metodo per la rilevazione dei movimenti oculari. Patent ITA 4518PTIT, 28 October 2004.

18.

Mathews

Fox

Yiend

Calder

(2003). The face of fear: Effects of eye gaze and emotion on visual attention. Visual Cognition, 10(7), 823–835.

19.

Pecchinenda

Pes

Ferlazzo

Zoccolotti

(2008). The combined effect of gaze direction and facial expression on cueing spatial attention. Emotion, 8(5), 628–634. doi:10.1037/a0013437.

20.

Phillips

A. T.

Wellman

H. M.

Spelke

E. S.

(2002). Infants’ ability to connect gaze and emotional expression to intentional action. Cognition, 85(1), 53–78.

21.

Posner

M. I.

(1980). Orienting of attention. The Quarterly Journal of Experimental Psychology, 32(1), 3–25.

22.

Putman

Hermans

van Honk

(2006). Anxiety meets fear in perception of dynamic expressive gaze. Emotion, 6(1), 94–102. doi:10.1037/1528-3542.6.1.94.

23.

Schuller

A-M.

Rossion

(2001). Spatial attention triggered by eye gaze increases and speeds up early visual activity. Neuroreport, 12(11), 2381–2386.

24.

Schuller

A-M.

Rossion

(2004). Perception of static eye gaze direction facilitates subsequent early visual processing. Clinical Neurophysiology, 115(5), 1161–1168.

25.

Tipples

(2006). Fear and fearfulness potentiate automatic orienting to eye gaze. Cognition and Emotion, 20(2), 309–320.

26.

Winston

J. S.

Henson

R. N. A.

FineGoulden

M. R.

Dolan

R. J.

(2004). fMRI-adaptation reveals dissociable neural representations of identity and expression in face perception. Journal of Neurophysiology, 92(3), 1830–1839.