Complex facial emotion recognition and atypical gaze patterns in autistic adults

Abstract

While altered gaze behaviour during facial emotion recognition has been observed in autistic individuals, there remains marked inconsistency in findings, with the majority of previous research focused towards the processing of basic emotional expressions. There is a need to examine whether atypical gaze during facial emotion recognition extends to more complex emotional expressions, which are experienced as part of everyday social functioning. The eye gaze of 20 autistic and 20 IQ-matched neurotypical adults was examined during a facial emotion recognition task of complex, dynamic emotion displays. Autistic adults fixated longer on the mouth region when viewing complex emotions compared to neurotypical adults, indicating that altered prioritization of visual information may contribute to facial emotion recognition impairment. Results confirm the need for more ecologically valid stimuli for the elucidation of the mechanisms underlying facial emotion recognition difficulty in autistic individuals.

Keywords

eye movement eye tracking facial expression social cognition

Individuals with a diagnosis of Autism Spectrum Disorder (ASD) commonly experience difficulty during facial emotion recognition (FER). During behavioural assessment, autistic¹ individuals demonstrate measurable impairment in accurately recognizing both basic (i.e. happy, angry, fear, disgust, surprise and sadness; Uljarevic & Hamilton, 2013) and complex (i.e. resentfulness and intimacy; Golan, Baron-Cohen, & Hill, 2006) emotions compared to their neurotypical (NT) counterparts.

This observed difficulty in FER may, in part, be driven by aberrant gaze behaviour (Black et al., 2017; Harms, Martin, & Wallace, 2010). Atypical gaze behaviour is commonly observed in autistic individuals, typically characterized by a reduced fixation time to the eyes compared to NT individuals. These differences however appear to be largely stimulus-dependent and remain relatively equivocal (Guillon, Hadjikhani, Baduel, & Rogé, 2014). Many studies have failed to confirm the purported reduced gaze to the eyes (Black et al., 2017; Harms et al., 2010). Others have also noted atypical gaze patterns for other facial regions, such as increased fixation on the mouth, though these findings are also inconsistent (Black et al., 2017; Harms et al., 2010).

The vast majority of research exploring the gaze behaviour of autistic adults has employed static representations of basic emotions, which alone may not fully represent the experience of everyday social functioning for autistic adults. Very few studies have employed complex emotions to explore the gaze-based correlates of FER in autistic adults, with these studies using only static representations of complex emotions, limiting their ecological validity (Black et al., 2017).

Given the stimulus-dependent nature of gaze behaviour and the fact that everyday social functioning is both dynamic and complex, the current study sought to examine the gaze behaviour of autistic adults during the recognition of emotions which are both complex and dynamic in order to increase the real-world applicability of findings. It was predicted that autistic adults, compared to NT adults, would exhibit poorer recognition performance, as well as reduced fixation time to eye regions.

Methods

Participants

Autistic (n = 20) and NT (n = 20) adults participated (Table 1) in this research. All autistic adults had a diagnosis of ASD according to the DSM V: Diagnostic and Statistical Manual of Mental Disorders (5th ed.; DSM-V; American Psychiatric Association, 2013) confirmed via self-report, with a formal diagnosis in Western Australia mandating consensus across a multi-disciplinary team (Glasson et al., 2008). Participants were matched on gender, Verbal Comprehension Index (VCI), Perceptual Reasoning Index (PRI) and Full-Scale IQ as measured by the Wechsler Abbreviated Scale of Intelligence (WASI-2; Wechsler, 2011), and Attention Switching as measured by the timed visual elevator scores of the Test of Everyday Attention (TEA; Robertson, Ward, Ridgeway, & Nimmo-Smith, 1994). Groups differed significantly on age, autistic trait severity, as measured by the Social Responsiveness Scale (SRS-2; Constantino & Gruber, 2012), and map search subtests of the TEA, indicating that autistic adults had greater autistic-like traits and lower visual selective attention (please refer to supplemental material for additional analysis accounting for these factors).

Table 1.

Participant demographics.

	NT(n = 20)	ASD(n = 20)	Test of significance
Age years (SD)	28.2 (9.5)	24.3 (7.9)	U = 120.500, Z = −2.151, p = .030
Gender (Male: Female)	16:4	17:3	χ² (1) = .173, p = .677
WASI-2
VCI	103.8 (12.0)	101.3 (12.6)	t(38) = .655, p = .516
PRI	113.5 (14.6)	109.5 (18.4)	t(38) = .761, p = .451
FSIQ	109.4 (11.4)	105.7 (12.9)	t(38) = .949, p = .349
TEA
Map search 1 min	11.1 (2.6)	7.6 (3.7)	t(38) = 3.479, p = .001
Map search 2 min	9.9 (2.7)	6.2 (4.2)	U = 81, Z = −3.245, p = .001
Visual elevator	12.4 (2.3)	10.6 (3.0)	U = 128.500, Z = −2.032, p = .052
Timed visual elevator	10.8 (3.6)	9.2 (4.8)	U = 158, Z = −1.142, p = .265
SRS-2	36.1 (16.2)	81.5 (24.5)	t(37) = −6.874, p < .001
T-score 59 and below (normal range)		5
T-score 60–65 (mild range)		5
T-score 66–75 (moderate range)		7
T-score 76 and above (severe range)		2

NT: Neurotypical; ASD: Autism Spectrum Disorder; SD: standard deviation; WASI-2: Wechsler Abbreviated Scale of Intelligence-2; TEA: Test of Everyday Attention; VCI: Verbal Comprehension Index (composite score) from WASI-2; PRI: Perceptual Reasoning Index (composite score) from WASI-2; FSIQ: Full-Scale Intelligence Quotient (composite score) from WASI-2; SRS-2: Social Responsiveness Scale-2.

Measures

SRS-2

The SRS-2 Adult Self Report Form (Constantino & Gruber, 2012) was used to measure autism trait severity. The SRS-2 is a standardized 65-item questionnaire pertaining to social communication, social motivation, social awareness, social cognition and rigid behaviours. Items are Likert-type-scaled ranging from 0 = not true to 3 = almost always true (max score = 195 and clinical cut-off of T-scores > 59).

WASI-2

The WASI-2 (Wechsler, 2011) provided an estimate of performance, verbal and full-scale IQ. This widely used short version of the Wechsler Adult Intelligence Scale and the Wechsler Intelligence Scale for Children includes the subtest for vocabulary, similarities, block design and matrix reasoning. The WASI-2 is standardized for individuals aged 6–89 years and has demonstrated good psychometric properties (Wechsler, 2011).

TEA

Four subtests of the TEA (Robertson et al., 1994) were administered. The map search (1 min and 2 min) subtests provided a measure of visual selective attention while visual elevator and timed visual elevator tasks provided a measure of attention switching (Robertson et al., 1994). These subtests were utilized to characterize the sample. The TEA is conceptualized for individuals 18–80 years of age. The four subtests applied in this study have adequate reliability (Robertson et al., 1994).

FER task

A sub-set of stimuli were selected from the Cambridge Mind Reading Face-Voice Battery (CAMs; Golan et al., 2006), which has previously been used to demonstrate FER difficulties in autistic adults in the normative IQ range (Golan et al., 2006). For this study, a set of 15 complex emotional stimuli was presented in a pseudo-randomized order. Emotions were presented once and included exonerated, intimate, empathic, vibrant, insincere, resentful, stern, grave, subservient, appalled, confronted, mortified, distaste, lured and appealing. Each trial consisted of an initial 1 s fixation cross, followed by a silent video clip (3–5 s) of a single actor expressing a complex emotion. Following each video clip, participants were asked to identify the displayed emotion from four options taken from previous work using the CAMs (Golan et al., 2006) using a keyboard.

Apparatus

Eye movements were recorded using a SensoMotoric Instruments Remote Eye Tracker Device (R.E.D; SensoMotoric Instruments, 2014). Stimulus presentation and behavioural data acquisition was controlled using E-Prime software (Psychology Software Tools, 2016) while eye movement data acquisition was controlled by SensoMotoric Instruments Eye View X software (SensoMotoric Instruments, 2014). The R.E.D was positioned in front of a 42-inch screen on which the stimuli were presented. Participants were seated 70 cm from the R.E.D on a height adjustable chair.

Procedure

Ethical approval (Curtin University HR52/2012) and written informed consent was obtained. This study was conducted as part of a larger experimental battery. Participants completed the WASI-2 and the TEA, before being familiarized with the eye tracker and FER task. A 9-point calibration procedure was then conducted. Upon adequate calibration, two practice items were shown, followed by the experimental trials of the FER task.

Data preparation and reduction

Fixation accuracy checks were conducted on all data with eye movement data excluded if the calibration accuracy was in excess of 1.5° of visual angle (VA). Fixations were defined as consecutive gaze samples held within 1° of VA for a minimum duration of 100 ms. Areas of Interest (AOIs) were dynamically defined over the actor’s eyes, nose and mouth region for each stimulus. The proportion of total fixation time to each of the eye, nose and mouth AOIs for each stimulus video was then calculated for correct and incorrect responses. Analysis of fixation time was subsequently conducted only for correct response trials.

Results

Emotion recognition accuracy

All participants received a total accuracy score (percentage). The proportions of correct responses were submitted to Mann-Whitney U tests as Kolmogorov Smirnov tests found the accuracy data to be not normally distributed. Analysis for FER accuracy revealed a near-significant trend indicating that autistic adults were less accurate in recognizing complex emotions compared to NT adults (U = 129, Z = −1.95, p = .056).

Fixation time

Fixation time was subjected to a mixed factorial analysis of variance (ANOVA), group (between subject factor; ASD and NT) × AOI (within subject factor; eyes, nose and mouth). The results of the multivariate analysis are reported (Philai’s trace) with partial eta square as the measure of effect size.

No main effect of group, F(1, 38) = 2.79, p = .10, ƞ²p = .07, was found. A main effect of AOI, F(2, 37) = 58.41, p < .001, ƞ²p = .76, was qualified by a group by AOI interaction, F(2, 37) = 4.42, p = .02, ƞ²p = .19. Bonferroni-corrected pairwise comparisons indicated that autistic adults fixated longer on the mouth than NT adults did (p = .04) (Figure 1). There was no significant difference between groups for the eye AOI (p = .14) or the nose AOI (p = .74). These comparisons also revealed that both NT and autistic adults fixated significantly longer on the eyes (NT: p < .01, ASD: p = .01) and nose (NT: p < .01, ASD: p = .01) than the mouth.

Figure 1.

Proportion of fixation time to areas of interest.

Discussion

It was hypothesized that autistic adults would demonstrate poorer FER accuracy and altered gaze behaviour. A marginal effect was observed for accuracy, indicating that autistic adults tended towards reduced accuracy during the FER task. Findings are consistent with previous reports of impairments in basic (Uljarevic & Hamilton, 2013) and complex emotions (Golan et al., 2006; Golan, Baron-Cohen, Hill, & Golan, 2006) in autistic individuals.

In addition to this marginal effect for poorer FER abilities, autistic adults demonstrated increased gaze to the mouth compared to NT adults. Contrary to predictions, however, no significant group difference in fixation time to the eye region was observed. In contrast to previous studies using static representations of emotions to examine FER, with atypical gaze most often observed to the eyes (Black et al., 2017; Harms et al., 2010), the present findings suggest that when complex, dynamic emotional concepts are considered, the primary difference observed in autistic individuals may pertain to the mouth region. Increased gaze to the mouth has been observed within the wider body of literature examining social attention in autistic individuals; however, similar to the FER literature, these findings are not consistently observed (Guillon et al., 2014). Taken together, the nature of the stimuli may likely play an integral role in the elicitation of gaze-based differences (Guillon et al., 2014).

Increased gaze to the mouth may indicate altered prioritization of visual information with this region arguably more physically salient and exhibiting a larger degree of movement than other facial features (Klin, Jones, Schultz, Volkmar, & Cohen, 2002). Previous research has found that rather than prioritizing information based on social salience (i.e. eyes), autistic individuals may prioritize features based on other criteria, such as physical salience (i.e. motion) (Klin et al., 2002). This gaze behaviour during FER may have been elicited in the current study due to the movement inherent in the stimuli, with these effects being unobservable in previous investigations using static stimuli.

While these findings may provide insight into the mechanisms underlying FER difficulty in autistic individuals, limitations must be considered. Analysis was based on a relatively small sample and a modest selection of stimuli with an unequal number of positive, negative and neutral stimuli. Future research should seek to understand how gaze-based differences to complex emotion may vary in accordance with emotional valence. Investigation of the functional role of core facial features during complex emotion processing via direct experimental manipulation or through combining eye tracking with neurophysiological measurement, such as electroencephalography, would also be of interest.

Through this study, it is shown that the use of complex, dynamic stimuli elicited gaze behaviour that is less commonly documented in the current body of literature, namely increased gaze to the mouth. Inconsistency observed in gaze-based findings during FER may be due to differences in the stimuli and experimental paradigms, with future research required to confirm how gaze behaviour may differ in autistic adults when more ecologically FER valid stimuli are used. Findings based on such stimuli may be useful in ensuring that the findings have a direct practical relevance to the difficulties faced by autistic individuals.

Supplemental Material

AUT856969_Lay_Abstract – Supplemental material for Complex facial emotion recognition and atypical gaze patterns in autistic adults

Supplemental material, AUT856969_Lay_Abstract for Complex facial emotion recognition and atypical gaze patterns in autistic adults by Melissa H Black, Nigel TM Chen, Ottmar V Lipp, Sven Bölte and Sonya Girdler in Autism

Supplemental Material

AUT856969_Supplemental_material – Supplemental material for Complex facial emotion recognition and atypical gaze patterns in autistic adults

Supplemental material, AUT856969_Supplemental_material for Complex facial emotion recognition and atypical gaze patterns in autistic adults by Melissa H Black, Nigel TM Chen, Ottmar V Lipp, Sven Bölte and Sonya Girdler in Autism

Footnotes

Acknowledgements

The authors would like to acknowledge the participants who took part in this research. The assistance of Dr Chiara Horlin is acknowledged for her support in the set-up of the experiment and Dr Wee Lee Lih for his assistance in eye tracking data preparation. The authors would also like to thank Dr Marita Falkmer and Professor Tele Tan for their support with this study.

Declaration of conflicting interests

The authors declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: Sven Bölte declares no direct conflict of interest related to this article. Bölte discloses that he has in the last 5 years acted as an author, consultant or lecturer for Shire, Medice, Roche, Eli Lilly, Prima Psychiatry, GLGroup, System Analytic, Kompetento, Expo Medica and Prophase. He receives royalties for text books and diagnostic tools from Huber/Hogrefe, Kohlhammer and UTB.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research is supported by the Cooperative Research Centre for Living with Autism (Autism CRC) (Project Number 3.032RS), established and supported under the Australian Government’s Cooperative Research Centres Program. This research is also supported by an Australian Research Training Program and top-up scholarship awarded by the Autism CRC. Sven Bölte was supported by the Swedish Research Council.

Supplemental material

Supplemental material for this article is available online.

Notes

ORCID iDs

Melissa H Black

Sven Bölte

References

American Psychiatric Association. (2013). Diagnostic and Statistical Manual of Mental Disorders (5th ed.). Arlington, VA: American Psychiatric Publishing.

Black

Chen

Iyer

Lipp

Bolte

Falkmer

. . . Girdler

(2017). Mechanisms of facial emotion recognition in autism spectrum disorders: Insights from eye tracking and electroencephalography. Neuroscience and Biobehavioural Reviews, 80, 488–515. doi:https://doi.org/10.1016/j.neubiorev.2017.06.016

Constantino

Gruber

(2012). Social Responsiveness Scale (2nd ed.). Los Angeles, CA: Western Psychological Services.

Glasson

MacDermott

Dixon

Cook

Chauvel

Maley-Berg

Wray

(2008). Management of assessments and diagnoses for children with autism spectrum disorders: The Western Australian model. The Medical Journal of Australia, 188, 288–291. doi:10.5694/j.1326-5377.2008.tb01623.x

Golan

Baron-Cohen

Hill

(2006). The Cambridge Mindreading (CAM) Face-Voice Battery: Testing complex emotion recognition in adults with and without Asperger syndrome. Journal of Autism and Developmental Disorders, 36, 169–183. doi:10.1007/s10803-005-0057-y

Golan

Baron-Cohen

Hill

Golan

(2006). The ‘Reading the Mind in Films’ task: Complex emotion recognition in adults with and without autism spectrum conditions. Social Neuroscience, 1, 111–123. doi:10.1080/17470910600980986

Guillon

Hadjikhani

Baduel

Rogé

(2014). Visual social attention in autism spectrum disorders: Insight from eye tracking studies. Neuroscience and Biobehavioural Reviews, 42, 279–297. doi:10.1016/j.neubiorev.2014.03.013

Harms

M. B.

Martin

Wallace

G. L.

(2010). Facial emotion recognition in autism spectrum disorders: A review of behavioral and neuroimaging studies. Neuropsychology Review, 20, 290–322. doi:10.1007/s11065-010-9138-6

Klin

Jones

Schultz

Volkmar

Cohen

(2002). Defining and quantifying the social phenotype in Autism. The American Journal of Psychiatry, 159, 895–908. doi:10.1176/appi.ajp.159.6.895

10.

Psychology Software Tools. (2016). E-Prime 3. Pittsburgh, PA: Author.

11.

Robertson

Ward

Ridgeway

Nimmo-Smith

(1994). The test of everyday attention. London, England: Pearson.

12.

SensoMotoric Instruments. (2014). SMI gaze and eye tracking systems. Available from www.smivision.com

13.

Uljarevic

Hamilton

(2013). Recognition of emotions in autism: A Formal meta-Analysis. Journal of Autism and Developmental Disorders, 43, 1517–1526. doi:10.1007/s10803-012-1695-5

14.

Wechsler

(2011). Wechsler Abbreviated Scale of Intelligence, Second Edition (WASI-II). San Antonio, TX: Pearson.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.50 MB

0.01 MB