AI's empathy gap: The risks of conversational Artificial Intelligence for young children's well-being and key ethical considerations for early childhood education and care

The limits of emotion recognition

Emotion recognition in AI often relies on analysing voice tone, facial expressions or textual sentiment cues. Machine learning models, such as classifiers or regression networks, ¹ are trained on data sets that correlate these cues with labelled emotions. However, human emotional expression is multidimensional, involving physiological, linguistic and contextual cues. AI models lack the perceptual capabilities to comprehend the entire spectrum of emotional expression. When young children express nuanced emotions, the AI's response might fall short if it fails to fully grasp the depth of the feelings expressed.

Worrying precedents include the 2018 testing of two mental-health chatbots by the BBC, as discussed above, which showed that these chatbots failed to respond to children reporting abuse and dismissed their concerns, even though both applications had been considered suitable for children (White, 2018). A more recent example is Snapchat's My AI chatbot. When speaking with a user it believed to be 13 years of age, My AI went rapidly off course when it advised the supposed 13-year-old to use candles and music when having sex for the first time with a 31-year-old partner (Fowler, 2023). My AI also told a user it believed to be 15 years old how to conceal the smell of alcohol and drugs in a list of tips for a pleasant birthday party (Fowler, 2023). A UNICEF (2020: 2) briefing noted that ‘when not designed carefully, chatbots can compound rather than dispel distress’, which ‘is particularly risky in the case of young users who may not have the emotional resilience to cope with a negative or confusing chatbot response experience’.

Moreover, AI is typically designed to react to specific cues and behaviours rather than accurately discern subtle emotional nuances. For instance, if a child expresses self-doubt, a conversational agent trained to affirm user statements might respond generically, reinforcing the child's negative self-perception instead of offering constructive support. This could jeopardise child well-being, exacerbating mental-health challenges such as anxiety or depression. A stark example of AI generating deceptively ‘agreeable’ responses emerges in the recent case of a chatbot suggesting self-harm methods to please a suicidal user (Xiang, 2023). When AI is anthropomorphised, as is the case with conversational agents, which are often designed to appear friendly and empathetic, it might be harder to shield young children from the emotional impact of harmful interactions (Kurian, 2023b).

Even if AI is trained to be sensitive to positive and negative signs of children's well-being, it could still respond simplistically. For example, when confronted by a young child showing distress, a conversational agent may generate a preprogrammed phrase without fully recognising or addressing the child's emotional needs. Consequently, the child would feel frustrated or invalidated without receiving the nuanced support so crucial to well-being in the early years (Kurian, 2023a).

The limits of language processing

AI systems rely on predefined contexts from training data. Natural language processing, a vital component in conversational AI development, relies on statistical patterns to understand language. These models process vast amounts of text to learn associations between words and their contexts. Through natural language processing, conversational AI systems leverage these learned associations to engage in meaningful interactions with users. By deciphering grammatical structures, syntax and semantic meaning, AI models equipped with natural language processing can respond contextually to user queries and prompts, giving the illusion of understanding.

However, despite impressive pattern recognition, natural language processing models lack true comprehension of language in the way of humans. Their ‘empathy gap’ stems from the fact that their understanding is rooted in statistical probabilities rather than genuine insight into meaning. These models thus struggle when faced with novel situations or language expressions that fall outside their training scope.

This can mean that when young children introduce unconventional speech patterns, non-literal expressions or imaginative scenarios that have not been part of the AI's training data, it encounters a lack of reference. AI's tendency to interpret language literally can result in misinterpreting children's intentions or failing to decipher nuances such as idioms, playful language and sarcasm. Its inability to adapt to contexts beyond its training data set may become evident, with responses appearing illogical or disconnected. Child users may experience frustration or even stress when AI responses fail to align with their intentions. Moreover, misinterpreting children's creative or playful utterances could be particularly stifling for their cognitive and linguistic development in the early years – a period when children thrive with opportunities for sustained shared thinking with interlocutors who are sensitive to their verbal cues (Brodie, 2014; Kurian, 2023a).

Algorithmic bias

It seems crucial to recognise that gaps in technology are rarely detached from their sociopolitical context (Benjamin, 2019). Societal biases can seep into training data. Neural networks, for example, amplify bias because they learn the statistical associations embedded in their training samples. Conversational outputs can then exacerbate stereotypes or misinformation since AI does not possess ethical reasoning; it merely reflects the imbalances in the data it was trained on. The consequences of exclusionary design for user well-being can be profound. For example, Mengesha et al. (2021) explored the psychological consequences of racial disparities in automated speech recognition systems. They found that the failure of automated speech recognition to recognise African American speech was alienating and demoralising for African American users. These users were left feeling that the technology was not made for them. They even had to modify their own speech for the technology to understand them. Mengesha et al. (2021) call for inclusive AI that can capture the needs of speakers who are traditionally misheard by voice-activated AI systems. Such research underscores the need to think about demographically diverse young children growing up with AI systems that are not necessarily designed for inclusion.

It is also relevant to consider the role of adaptive learning mechanisms, including reinforcement learning, which allow AI systems to improve over time based on user interactions. These models update their internal parameters to maximise a predefined reward signal. However, without ethical guard rails, a danger emerges when these adaptive learning mechanisms encounter unfiltered or malicious user interactions. An example is the case of Microsoft's chatbot, Tay. Tay was unleashed on Twitter in 2016 to learn freely from human users and mimic their language. However, in less than 24 hours, it began to post discriminatory tweets, ranging from tirades against feminists to calls for genocide (Brandtzaeg and Følstad, 2018). This was because some users manipulated Tay to mimic harmful content (Neff and Nagy, 2016). Microsoft shut down Tay's account within 16 hours, acknowledging ethical violations. The Tay incident now serves as a cautionary tale for AI development, showing how users can manipulate AI behaviour with biased content and how AI can amplify existing biases in its training data. The potential for young children to encounter age-inappropriate and discriminatory content through unsupervised learning by conversational agents, often in unfiltered and unpredictable online environments, makes it all the more vital to prioritise robust pre-training and continuously monitor AI–child interactions.

Communication and understanding

Transparency and authenticity

Continuous monitoring and improvement

Child-centred design

While conversational AI models may wear the cloak of empathy, they can struggle to offer the genuine article. At this pivotal moment, to shape the ethical landscape of AI interactions for young children, child-centred design and use seems more crucial than ever.