DE-ENIGMA: Multimodal Human–Robot Interaction for Teaching and Expanding Social Imagination in Autistic Children

Zeno: A Humanoid Robot for Interacting with Autistic Children

A humanoid robot known as Zeno helps to teach school-aged autistic children, who have additional intellectual disabilities or limited spoken communication, to express emotions. It is able to process children's movements, vocalizations, and facial expressions in order to present activities linked to emotions adaptively, and engage in feedback, support, and play. The technology includes deep learning algorithms to detect and interpret the child's vocalizations, behaviors, and facial expressions; interactive robot behaviors that adapt to the child's actions and phase of learning; and a gaming engine to help the children progress through the learning steps.

In the first period of DE-ENIGMA, 128 autistic children were randomly assigned to participate in Zeno-assisted or adult-led teaching conditions of a six-step emotion-training program. More than 154 hours of raw audio-visual data were collected and have been annotated, and will serve for machine analysis of facial, bodily, vocal, and verbal behaviors.

Through two design iterations and subsequent usability studies, the project has explored several game activities aimed at teaching autistic children about facial features and emotional facial expressions. The partners have investigated appropriate methods for children to interact with the game and with the robot.

Complementing these efforts, the DE-ENIGMA team have been developing state-of-the-art technical systems, based on artificial intelligence, to detect automatically and reason about the emotional state of the children during the therapy and their level of interest in the activity at hand. The technical system incorporates vocal, facial, and body posture cues with deep learning architectures to create robust and generalizable models across the target population. Furthermore, the DE-ENIGMA team will utilize the same technologies to create a novel reporting tool for the therapist, which will log a wide range of nonverbal behavior observed during the therapy.

Sharing the Interaction Data Set with the Scientific Community

DE-ENIGMA is the first project of its kind that allows a wide scientific audience access to valuable behavioral data, extends social signal processing capability with this challenging user group, and provides a robust test of robot effectiveness in teaching socio-emotional skills to autistic children.

The DE-ENIGMA team have annotated the recordings of 50 (25 British and 25 Serbian) participants in terms of continuous emotion stated, measured in valence (positivity/negativity of the emotion) and arousal (intensity of the emotion). The annotation results were then aligned using dynamic time warping for development of the machine learning method.

Online distribution of the data set has been shown to be impractical due to the large size of the corpus (∼7 TB). Instead, data distribution is now facilitated by delivering of physical storage medium and is stored at more than one partner to help with data storage and distribution.

The DE-ENIGMA data set has been collected and is now being digitally tagged for features such as:

facial mapping coordinates (smiles and frowns and other facial expressions can be recognized);

This data set will allow the wider scientific community to research the behavior of children on the autism spectrum to improve current recognition software that will lead to better automatic recognition of physical features in a neurodiverse population. This should eventually lead to improved therapeutic and educational solutions for neurodiverse children.