Face the Uncanny: The Effects of Doppelganger Talking Head Avatars on Affect-Based Trust Toward Artificial Intelligence Technology are Mediated by Uncanny Valley Perceptions

Abstract

This experiment (N = 228) examined how exposure to a talking head doppelganger created by an artificial intelligence (AI) program influenced affect-based trust toward AIs. Using a 3 (talking head featuring the participant's or a stranger's face, audio-only condition) by 2 (pro-AI pitch and anti-AI pitch playback) design, we uncovered that exposure to a talking head featuring the participant's face instead of a stranger's face increased uncanny valley perceptions. Furthermore, uncanny valley perceptions mediated the link between exposure to a talking head with the participant's face on affect-based trust. Overall, exposure to a doppelganger talking head, who delivered a persuasive pitch, triggered discomfort on the participant whose features were sourced to craft a synthetic talking head, which in turn decreased affect-based trust attributed to AIs. This phenomenon is rooted in basic psychological mechanisms that underpin the uncanny valley hypothesis. Future studies may test for these findings across different platforms and also provide evidence regarding user mental processing.

Introduction

Artificial intelligence (AI) technology can effortlessly transform and produce photos, audio, video, and text in real time. Such transformations range from digital retouching and editing, to developing synthetic speech delivered by digitally created human-like avatars and agents. As the use of AI-driven technologies becomes fully integrated into everyday life through avatars, agents, chatbots, digital assistants, and robots, it is of critical importance to establish how interpersonal communication that is generated, transmitted, modified, or augmented by AIs may influence communicative processes and outcomes.¹

Though individuals' abilities to detect AI-generated messages receive much attention,² few studies investigate the psychological impact on individuals as a result of AI-generated messages. Consider that replacing the face of one person with another in a video,³ avatar, or digital body^4,5 is progressively becoming easier to accomplish. Using existing photos and recordings, AI technology can reconstruct a new version of ourselves as a talking head capable of producing speech and gesture. Exposure to talking head avatars can influence basic interpersonal effects. For instance, individuals stand closer and are more willing to commit embarrassing behaviors in front of virtual agents featuring the participant's face rather than the face of someone else.⁶ However, individuals are more likely to reject social media avatar partners that raise eerie or uncanny valley perceptions⁷ Considering this, this study examines how exposure to a talking head constructed by an AI can influence attributions of AI affect-based trust, which represents trustworthiness beliefs based on perceived care and concern.⁸ The study also explores how uncanny valley or eerie perceptions raised by talking head exposure mediates the link between AI-constructed talking head exposure and affect-based trust.

The Uncanny Valley Hypothesis

The uncanny valley hypothesis predicts that individuals display aversion toward synthetic beings with near-human features, unsettlingly realistic-looking features, and defective animated movement^9,10 as described by Masahiro Mori's “creepy valley phenomenon.”¹¹ In theory, conflicting information (e.g., avatar looks human, but is not and avatar looks like me, but is not me) may trigger perceptual tension, including cognitive dissonance,¹² uncertainty, eeriness, and discomfort.¹³ High-fidelity, human-like talking heads are perceived as more uncanny when their upper face region animation reduces emotional expressiveness.¹⁴ In addition, subtle changes in the facial expression of robots (i.e., eyebrow raise) decrease human judgments of robot trustworthiness.¹⁵ Similarly, participants attributed lower trust to human-like robots relative to mechanical robots.¹⁶ Though previous studies devote attention to how near-human robots, avatars, and talking heads can instigate the uncanny valley and decrease perceived trust, the effects of synthetic characters that resemble the user's facial traits deserve further attention. The concept of mirror self-experience refers to profoundly unsettling encounters with one's specular reflection.¹⁷ Exposure to one's image in a mirror can be upsetting, especially for individuals and members of cultures not used to it or if the reflected image is distorted.¹⁷ Mirrors not only can be uncanny at a physical and perceptual level as they seemingly reflect reality but also can be a source of illusory perceptions.¹⁷ This assumption harkens back to Freud's link between the uncanny and encountering a doppelganger or evil, aberrated, or repressed twin version of the self.¹⁸ Although virtual doppelgangers or digital characters who resemble their users, but operate independently are investigated in regard to their effects on health behaviors, persuasive marketing, and false memories,¹⁹ their influence has not been examined in regard to affect-based trust and uncanny valley perceptions. Based on the assumption that synthetic doppelganger characters are more uncanny and thus less trustworthy, we hypothesize that (H1) individuals will attribute lower affect-based trust toward AIs after exposure to an AI-generated talking head, which features the participant instead of a stranger's face, and relative to a control group exposed to audio delivery (i.e., no talking head).

The Link Between the Uncanny Valley and Attributions of Trustworthiness

Uncanny avatars and robots may influence trustworthiness based on the degree to which they trigger aversive and defensive cognitive systems that result in negative judgments and distancing from negative stimuli.²⁰ Feelings of eeriness evoked by avatars can suppress information processing and reduce accuracy when making partner judgments.²⁰ Inhibitory-devaluation accounts of the uncanny valley hypothesis show how stimuli that activate competing visual category representations during recognition can elicit negative affect.²¹ For example, interlinked areas of the brain may encode human-like artificial agents along a “human likeness” continuum, a “human like, but not actually human” detection activity threshold, and an amygdala signal that predicts rejection of uncanny human-like agents.²² Congruent with how the uncanny valley elicits aversive states, high-quality talking heads are perceived as less trustworthy relative to non-photorealistic talking heads, thus showing how uncanny avatar rendering style affects perceived trust.²³ Based on the link between uncanny valley perceptions and perceived trust,^15,16,23 we hypothesize that (H2) uncanny valley perceptions will mediate the effects of talking head exposure on affect-based trust toward AI-generated messages.

Can AI-Generated Messages Trigger Uncanny Valley Perceptions and Reduce Trustworthiness?

AIs may generate and deliver messages with minimal user input,¹ which poses new questions for uncanny valley research in regard to the effect of messages conveyed by AI-generated talking heads. Participants presented with medical advice by a computer-animated medic show increased persuasion as perspective-taking with the synthetic character increased regardless of whether the character experienced positive or negative subsequent professional consequences, whereas participants presented with video recordings of a medic show augmented persuasion as perspective-taking increased but only when the character experienced positive professional consequences.²⁴ However, participants who receive medical advice show increased persuasion regardless of whether advice is delivered by a computer-animated or digitally recorded character with smooth or jerky animations, implying that figures of authority may be persuasive even if delivered by uncanny-looking characters.²⁵ More specifically, participants experience increased uncanny perceptions and more negative affect after cooperating with an animated avatar relative to a text chatbot.²⁶ Considering how the language in media content suggests benchmarks and interpretative frames that can influence individuals' evaluations about opinions presented to them,²⁷ it can be expected that exposure to a pro-AI message delivered by an AI-generated talking head featuring the participant's face should be particularly eerie as it exposes participants to a conspicuously biased message delivered by a doppelganger. Thus, (H3) relative to the remaining conditions, uncanny valley perceptions will mediate the effects of exposure to a talking head with the participant's face, which delivers a recording of a pro-AI pitch on affect-based trust toward AI-generated messages.

Method

Participants

Participants (N = 228) were undergraduates at a large West Coast American university, who received extra credit for their participation in this study. The study was approved by the local institutional review board (IRB ID: 1497031-1). The participants were 13.8% male, 86.2% female, and less than 1% identified as nonbinary. The participants were 46.9% Asian/Pacific Islander, 44.3% Caucasian, including Hispanic, 2.6% African American, less than .05% Native American, and 5.7% declared other or multiple ethnic backgrounds. The average age of participants was 19.6 years (SD = 1.46 years).

Materials

Talking heads

Photos taken from the front and side viewing angle were converted into a talking head using Reallusion CrazyTalk8 (Fig. 1). Talking heads were built using the participants' faces in the self condition. The stranger condition had a yoked design so that participants in both talking head conditions would receive similar stimuli. Thus, stranger condition participants were exposed to the talking head constructed using the photos of the previous same-sex participant in the self condition. The control condition had no talking head and merely played back audio recorded by the participants.

FIG. 1.

Examples of Male and Female Talking Head Avatars.

Procedure

The study was a 3 × 2 factorial experiment. Participants were randomly assigned to one of six experimental conditions using a random numbers table. Participants were told a cover story that involved laboratory testing of a new AI program that would recombine photos and audio recordings to create a talking head for participants to evaluate. After obtaining consent, participants' photos were taken and then they read out loud both a pro- and anti-AI pitch. The pitches were rotated so as to prevent order effects. Participants answered a decoy survey for ∼5 minutes while the talking head was created. This was done out of sight of participants by an unseen research assistant. The decoy survey inquired about participants' lifestyle and political beliefs. Each participant was then exposed to a single experimental condition. Participants then answered a survey containing items measuring perceived uncanny valley and attributions of affect-based trust toward AI-generated messages.

Measures

Uncanny valley perceptions

This factor was evaluated with a scale²⁸ that measured the strength of discomfort and unease triggered by the experimental manipulations. It consisted of 20 semantic differential items on a 1 (“strongly disagree”) to 7 (“strongly agree”) Likert-type scale. The scale achieved good reliability α = 0.898. Sample items included “How would someone else rate you as presented in the audio/video AI-generated message? spine-tingling/numbing; not human-like/human-like; or eerie/normal”

Affect-based trust

This scale captured feelings of security and comfort related to relying on someone else. It consisted of three items on a 1 (“strongly disagree”) to 7 (“strongly agree”) Likert-type scale. The scale had acceptable reliability α = 0.723. Items included “I feel I can trust AI-generated messages to form my opinions on AIs,” “I would feel uncomfortable if I had to rely on AI-generated messages to form my opinion about AIs,” and “I would feel safe if I had to rely on AI-generated messages to form my opinions about AIs.”

Results

Group differences were explored with a 3 × 2 analysis of variance. The full results appear in https://osf.io/kngvz/ To test for the hypotheses, three separate mediation tests²⁹ examined the direct effects resultant from the manipulations on attributions of affect-based trust toward AIs, along with mediation effects of uncanny valley perceptions. Products were mean centered in all tests. Multicategorical independent variables were dummy coded²⁹ and the reference group was set at the hypothesized most uncanny condition (i.e., talking head with the participant face relative to the stranger's face and audio conditions for H1 and H2 and pro-AI pitch playback delivered by a talking head with the participant's face relative to all of the remaining conditions for H3). Descriptive statistics appear in Tables 1 and 2.

Table 1.

Descriptive Statistics Per Experimental Condition for Affect-Based Trustworthiness of Artificial Intelligence Generated Messages and Uncanny Valley Perceptions

Variable	n	AI affect-based trustworthiness		Uncanny valley perceptions
Variable	n	M	SD	M	SD
Talking head
Participant face	76	2.605	1.102	4.722	0.684
Stranger face	75	2.724	1.117	4.314	0.750
Audio playback	77	2.866	1.018	3.425	0.706
AI pitch stance
Pro-AI	115	2.672	1.064	4.177	0.890
Anti-AI	113	2.794	1.097	4.122	0.904

AI, artificial intelligence.

Table 2.

Descriptive Statistics for Multicategorical Experimental Conditions for Affect-Based Trustworthiness of Artificial Intelligence Generated Messages and Uncanny Valley Perceptions

Condition	n	AI trust		Uncanny valley perceptions
Condition	n	M	SD	M	SD
Participant face talking head – Pro AI pitch	39	2.579	0.985	4.707	0.611
Participant face talking head – Anti AI pitch	38	2.632	1.220	4.737	0.757
Stranger face talking head – Pro AI pitch	37	2.658	1.135	4.567	0.595
Stranger face talking head – Anti AI pitch	38	2.793	1.109	4.055	0.810
Audio playback – Pro AI pitch	39	2.778	1.085	3.279	0.640
Audio playback – Anti AI pitch	38	2.956	0.950	3.574	0.747

The talking head with the participant's face condition did not directly decrease attributions of affect-based trust toward the AI-generated message relative to the other two conditions, b = 0.005, SE = 0.168, t(225) = 0.032, p = 0.974, 95% CI: −0.325 to 0.336. H1 was disconfirmed. However, the talking head with the participant's face condition increased uncanny valley perceptions relative to the other two conditions, b = 0.858, SE = 0.112, t(226) = 7.636, p < 0.001, 95% CI: 0.637–1.080. In addition, uncanny valley perceptions were negatively related to affect-based trust toward the AI-generated message, b = −0.229, SE = 0.089, t(225) = −2.581, p = 0.010, 95% CI: −0.403 to −0.054. Uncanny valley perceptions yielded a significant indirect-only mediation effect on the relationship between exposure to a talking head featuring the participant's face on affect-based trust toward AIs relative to the remaining conditions, b = −0.196, SE = 0.080, 95% CI: −0.358 to −0.048. This significant indirect-only mediation effect between the participant exposure condition through uncanny valley perceptions onto participant affect-based AI trust confirmed H2.

In regard to H3, the talking head featuring the participant's face, which delivered a pro-AI pitch playback condition, did not directly decrease affect-based trust relative to the remaining conditions, b = −0.035, SE = 0.197, t(225) = −0.178, p = 0.895, 95% CI: −0.423 to 0.353. However, the talking head with the participant's face delivering a pro-AI pitch playback augmented uncanny valley perceptions relative to the other conditions, b = 0.668, SE = 0.153, t(226) = 4.364, p < 0.001, 95% CI: 0.366–0.970. Uncanny valley perceptions were negatively related to affect-based trust, b = −0.223, SE = 0.082, t(225) = −2.715, p = 0.007, 95% CI: −0.385 to −0.061. In addition, uncanny valley perceptions had significant indirect-only mediation effects on the relationship between the talking head featuring the participant's face delivering a pro-AI pitch on affect-based trust relative to the remaining five conditions, b = −0.149, SE = 0.061, 95% CI: −0.279 to −0.039. H3 was confirmed.

Although there was formal theoretical model or hypothesis informing the direct effect of pro- and anti-AI message manipulations, these effects were examined with procedures similar as above to fully account for the experimental manipulations. There was no significant direct effect of playback of the pro- or anti-AI pitch on affect-based trust, b = −0.109, SE = 0.141, t(225) = −0.772, p = 0.441, 95% CI: −0.387 to 0.169. Pro- or anti-AI pitch playback had no significant effect on uncanny valley perceptions, b = 0.054, SE = 0.119, t(226) = 0.455, p = 0.650, 95% CI: −0.180 to 0.288. Same as above, uncanny valley perceptions were linked to decreased affect-based trust, b = −0.225, SE = 0.079, t(225) = −2.858, p = 0.005, 95% CI: −0.381 to −0.070. Uncanny valley perceptions did not exert relative indirect effects on the relationship between either pro- or anti-AI pitch playback on affect-based trust, b = −0.012, SE = 0.029, 95% CI: −0.075 to 0.043.

Discussion

This study capitalized on the uncanny valley hypothesis to examine how exposure to a doppelganger talking head ostensibly created by an AI program influenced individuals' affect-based trust toward AI-generated messages. Presenting individuals with a talking head featuring their own face decreased affect-based trust toward AIs relative to talking heads featuring a stranger's face or relative to simple audio playback. This finding was consistent with how uncanny animated characters decrease trust,^15,16,23 and expanded previous research by showing how talking heads featuring individual's face can trigger uncanny valley perceptions. It is possible that exposure to a talking head with their face decreased participants' affect-based trust toward AIs by activating aversive cognitive systems that trigger rejection of uncanny synthetic characters.^20–22 This finding resonated with how uncanny valley perceptions dampened partner liking⁷ and involvement with narratives,²⁴ and also expanded previous research by showing how talking head doppelgangers can prompt uncanny valley perceptions in ways similar to seeing one's warped mirror image.¹⁷

The mediating role of uncanny valley perceptions further underscored the effects of doppelganger exposure. Participants expressed an increased spine-tingling and eerie feeling when presented with a talking head featuring their face instead of a stranger's face, and such uncanny valley perceptions were negatively related to affect-based trust toward AIs. Previous studies had focused on how facial expressions and synthetic speech,¹⁴ along with realism, human-likeness,³⁰ and mortality salience of robots³¹ trigger uncanny valley effects. This study further advanced the uncanny valley hypothesis so as to account for decreased affect-based AI trustworthiness. In addition to mediation effects, future studies should examine how talking heads featuring different faces can be made more or less uncanny by manipulating facial features, animations, speech, human-likeness, and mortality salience. These factors may influence the degree to which individuals accept and intend to use AIs capable of crafting doppelganger synthetic characters. Additionally, attributions of cognitive trust (i.e., technical expertise) toward AIs should also receive attention. It is possible that uncanny valley reactions toward talking heads exert stronger effects on affective process related to trustworthiness, but it may have weaker effects on cognitive trust or judgments of expertise that require more systematic appraisals. Future studies should also investigate how increased familiarity and experience with avatars, AIs, and robots can moderate the effects of exposure to talking head versions of the self by diminishing uncanny valley reactions. For instance, tech-savvy users may be more habituated to uncanny interactions with talking heads, robots, and digital assistants.

Limitations included the use of an imbalanced sample as the majority of the participants were female. No gender differences or interaction effects on the outcome variables were found. In addition, the talking heads were created and shown to participants within the same laboratory session. However, this was done out of sight and under a cover story that an AI mixed and matched photos and audio to create a talking head. The study provided initial findings looking at a single platform; thus, future research should replicate these results using virtual and augmented reality talking heads, along with robotic talking heads. Because our procedure produced eerie talking heads out of participants' photos, future studies are encouraged to decouple the effects of user-talking head similarity and uncanniness. This study was also limited by lack of evidence in regard to participants' cognitive processes. Thus, future studies should provide evidence regarding the cognitive systems involved in perceiving talking heads with different faces.

This study shows that we have the capacity to create doppelganger talking heads that can deliver social messages and opinions. Exposure to such advancement can trigger discomfort for people whose faces and speech were used to create a synthetic talking head, which may in turn decrease affect-based trust attributed to AIs. This phenomenon is rooted in the uncanny valley hypothesis.

Footnotes

Author Disclosure Statement

No competing financial interests exist.

Funding Information

No funding was received.

References

Hancock

, Naaman

, Levy

. AI-mediated communication: definition, research agenda, and ethical considerations. Journal of Computer-Mediated Communication, 2020; 25:89–100.

Güera

, Delp

. (2018) Deepfake video detection using recurrent neural networks. In: 2018 15th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS). Auckland, New Zealand: IEEE, pp. 1–6.

Korshunov

, Marcel

. Deepfakes: a new threat to face recognition? Assessment and detection. arXiv preprint arXiv:181208685. 2018.

Ahn

, Bailenson

. Self-endorsing versus other-endorsing in virtual environments. Journal of Advertising. 2011; 40:93–106.

Fox

, Bailenson

. Virtual self-modeling: the effects of vicarious reinforcement and identification on exercise behaviors. Media Psychology. 2009; 12:1–25.

Bailenson

, Beall

, Blascovich

, et al. (2001) Intelligent agents who wear your face: Users' reactions to the virtual self. In: de Antonio

, Aylett

, D. B, eds. Intelligent virtual agents IVA 2001 lecture notes in computer science. Berlin, Heidelberg: Springer Berlin Heidelberg, pp. 86–99.

Shin

, Song

, Chock

. Uncanny valley effects on friendship decisions in virtual social networking service. Cyberpsychology, Behavior, and Social Networking. 2019; 22:700–705.

McAllister

. Affect- and cognition-based trust as foundations for interpersonal cooperation in organizations. The Academy of Management Journal. 1995; 38:24–59.

MacDorman

, Ishiguro

. The uncanny advantage of using androids in cognitive and social science research. Interaction Studies. 2006; 7:297–337.

10.

MacDorman

, Green

, Ho

C-C

, et al. Too real for comfort? Uncanny responses to computer generated faces. Computers in Human Behavior. 2009; 25:695–710.

11.

Mori

, MacDorman

, Kageki

. The uncanny valley [from the field]. IEEE Robotics & Automation Magazine. 2012; 19:98–100.

12.

Festinger

. (1957) A theory of cognitive dissonance. Stanford, CA: Stanford University Press.

13.

Moore

. A Bayesian explanation of the ‘Uncanny Valley’ effect and related psychological phenomena. Scientific Reports. 2012; 2:864.

14.

Tinwell

, Grimshaw

, Nabi

, et al. Facial expression of emotion and perception of the Uncanny Valley in virtual characters. Computers in Human Behavior. 2011; 27:741–749.

15.

Mathur

, Reichling

. (2009) An uncanny game of trust: Social trustworthiness of robots inferred from subtle anthropomorphic facial cues. 2009 4th ACM/IEEE International Conference on Human-Robot Interaction (HRI). La Jolla, CA, USA, pp. 313–314.

16.

Mathur

, Reichling

. Navigating a social world with robot partners: a quantitative cartography of the Uncanny Valley. Cognition. 2016; 146:22–32.

17.

Rochat

, Zahavi

. The uncanny mirror: a re-framing of mirror self-experience. Consciousness and Cognition. 2011; 20:204–213.

18.

Vardoulakis

. The return of negation: the doppelganger in Freud's “The Uncanny.”. SubStance. 2006; 35:100–116.

19.

Bailenson

, Segovia

. (2010) Virtual doppelgangers: Psychological effects of avatars who ignore their owners. In: Bainbridge

, ed. Online Worlds: Convergence of the Real and the Virtual. London: Springer London, pp. 175–186.

20.

Shin

, Kim

, Biocca

. The uncanny valley: no need for any further judgments when an avatar looks eerie. Computers in Human Behavior. 2019; 94:100–109.

21.

Ferrey

, Burleigh

, Fenske

. Stimulus-category competition, inhibition, and affective devaluation: a novel account of the uncanny valley. Frontiers in Psychology. 2015; 6:249.

22.

Rosenthal-von der Pütten

, Krämer

, Maderwald

, et al. Neural mechanisms for accepting and rejecting artificial social partners in the uncanny valley. The Journal of Neuroscience. 2019; 39:6555–6570.

23.

McDonnell

, Breidt

. (2010) Face reality: Investigating the uncanny valley for virtual faces. ACM SIGGRAPH ASIA 2010 Sketches. Seoul, Republic of Korea: Association for Computing Machinery, p. Article 41.

24.

MacDorman

. In the uncanny valley, transportation predicts narrative enjoyment more than empathy, but only for the tragic hero. Computers in Human Behavior. 2019; 94:140–153.

25.

Patel

, MacDorman

. Sending an avatar to do a human's job: compliance with authority persists despite the uncanny valley. Presence. 2015; 24:1–23.

26.

Ciechanowski

, Przegalinska

, Magnuski

, et al. In the shades of the uncanny valley: an experimental study of human–chatbot interaction. Future Generation Computer Systems. 2019; 92:539–548.

27.

Scheufele

, Tewksbury

. Framing, agenda setting, and priming: the evolution of three media effects models. Journal of Communication. 2006; 57:9–20.

28.

Destephe

, Brandao

, Kishi

, et al. Walking in the uncanny valley: importance of the attractiveness on the acceptance of a robot as a working partner. Frontiers in Psychology. 2015; 6:204.

29.

Hayes

, Montoya

. A tutorial on testing, visualizing, and probing an Interaction involving a multicategorical variable in linear regression analysis. Communication Methods and Measures. 2017; 11:1–30.

30.

MacDorman

. (2006) Subjective ratings of robot video clips for human likeness, familiarity, and eeriness: An exploration of the uncanny valley. ICCS/CogSci-2006 long symposium: Toward social mechanisms of android science. Vancouver, Canada, pp. 26–29.

31.

MacDorman

. (2005) Mortality salience and the uncanny valley. 5th IEEE-RAS International Conference on Humanoid Robots. Seoul, South Korea: IEEE, pp. 399–405.