Abstract
Autism Spectrum Disorder (ASD) is defined as a condition or disorder that begins in childhood and that causes problems in establishing relationships and communicating with other people. Aarya works as a personal well-being companion to children with Autism Spectrum Disorder while they interact with a virtual environment that is gesture based. By making an ASD affected child face real world situations, we try to improve his/her confidence in facing the world and being open to learning various skills. Social interaction and communication are the major challenges faced by children with ASD. In Aarya, we use gesture-based interface that is the Microsoft Kinect so that the child can find it easier to interact in the real world environment. Through the interactions made with the children and the results obtained, we understand that this tool can be a companion while giving chance for growth and improving their interacting ability. With further refinement and expert inputs, this tool can be built better.
Introduction
Autism Spectrum Disorder (ASD) is a neurodevelopmental disorder that causes impairment in social interaction and communication [12]. According to The Times of India, it is estimated that 1 in 250 children in India are diagnosed with ASD and currently 10 million Indians are autistic [10]. Current therapies in India include Homeopathy, Ayurveda and Music Therapy. In addition to this, there are wide varieties of Android and iOS applications that assist autistic children with communication. A good number of learning centers have faculties who are fully committed in helping these children.
The main idea behind this project is to provide autistic children with a constant companion named Aarya, a virtual friend, who would encourage and praise them for the tasks completed, like their teachers and parents.
The different modules of Aarya comprise of real life situations such as going to school, shopping from a grocery store, playing and cleaning a room. When children perform these activities in the virtual world, they can easily connect with them and adapt to them without much difficulty. Aarya works towards improving the confidence of Autistic children in an interactive systematic approach.
Most of the applications available for autistic children restrict the movement of the child to his/her fingertips. These applications help them improve their mental abilities such as memory and response to visual stimuli. However, their body movements are limited. With the positive results of these applications, there is evidence that virtual interfaces can help autistic children with their social skills. In Aarya, Kinect motion sensor is used to substitute this lack in body movement and to help in hand eye coordination [6]. Microsoft develops Kinect for the Xbox360 console; it is a natural user interface that takes in gestures and spoken commands as input. Scratch is a programming language that provides an interface that is user friendly and enables us to connect to the gesture based motion sensor. When a child interacts with Aarya, he/she is not only learning from a good visual application but also communicating with the interface with his entire body and not just fingertips.
Literature survey
Virtual Reality increases the feeling of presence and realism of the experience thus Miguel Bernardes et al. concluded that it is a potential learning tool for children with ASD. However, using a Head mount device such as Oculus can be overwhelming and cause motion sickness in individuals with ASD [4]. Commercially available Xbox Kinect videogame with its new infrared 3D technology promotes use of full body as compared to the previous single point sensors [2]. Considering this, Aarya is developed on Microsoft Kinect, which is a motion-sensing device for Xbox 360.
Aimilia Tzanavari et al. observed that children with ASD can learn to perform a task in a simulated environment which they can later apply in a real world situation [1]. Aarya thus uses real world situations which teach the children with ASD to be independent while learning in a fun and accommodating environment.
Research from Nkiruka Uzuegbunam et al. has shown that self-imaging has potential educational and therapeutic value for individuals with Autism Spectrum Disorder [5]. A virtual environment that is based on real world situations is created with Aarya. This will not only give the children an exciting environment to work in, but will also help them feel more involved. Zengguo Ge et al. also identified that this helps children with Autism Spectrum Disorder in development of communication skills, verbal skills, interests as well as peer relationships [9].
Mario Saiano et al. concluded that gesture-based interfaces are superior in terms of transfer of learned skills to real life environment for individuals with ASD [3]. In Aarya a gesture-based interface is used, that is the Microsoft Kinect, so that the child can find it easier to interact in a real world environment.
Tests by Xiaofeng Liu et al. have shown that an active teaching system of human-robot interaction that a child shows robot an action and then robot is asked to imitate it can effectively attract Autistic children’s attention and can assist children to develop social skill by interactive action imitation [8]. Aarya uses this interactive action imitation to get the attention of children with ASD in order to improve their social skills.
A robot-assisted therapeutic intervention system was used by Sang-Seok et al. to develop the social behavior of children with ASD [7]. The proposed treatments had a competitive advantage in improving social skills of children with ASD from the clinical results. Aarya is designed to be a companion to the children with ASD and to encourage them to have better social skills. Aarya works as a virtual guide and serves the same purpose that the robot served in this paper.
The paper is divided into four sections: Section 2 describes the various methods and modules employed in Aarya. Section 3 discusses the experiment carried out. Section 4 analyses the results obtained from this experiment and discusses the future scope of the project.
Methods
It is important that we make Aarya interesting with cartoons and an encouraging environment in order to retain the attention of Autistic children; Scratch enables us to do that. Children with Autism Spectrum Disorder find change stressful [11]. However, when it comes to real world situations change is inevitable; In Aarya as the level difficulty increases more change is introduced. This is done in order to encourage acceptance of change.
Hand eye coordination and physical activity is required to interact with the game. When the child follows the instructions he/she is praised and encouraged, thus making them interested to play more. When this model was introduced to teachers and parents of Autistic children, they were enthusiastic at the scope and potential of this project. They said that if virtual environment can be used to teach social skills to individuals with ASD, apart from helping them communicate better, it would be a great tool for their parents and teachers.
This project uses the Kinect motion sensor to receive input in the form of gestures. This input is then transferred to the computer and the software used is Kinect2Scratch. This software is used to configure and analyze the body movements and connect these inputs to Scratch. In Scratch, the positions of the joints in a human body are analyzed and once the required movement is recognized, Scratch processes this data and the flow of control is transferred to the next module. It then gives the next instruction. This architecture is illustrated in Fig. 1. For example, the child is required to raise his/her hand in order to start the game. Once this gesture is completed, he/she is given the next task.
Modules
The modules in Aarya aim at making autistic children comfortable with social interaction. Each of these modules are designed on Scratch and then connected to Kinect using the Kinect2Scratch software.
Shopping module level 1
In this module, autistic children are encouraged to interact with the shopkeeper and buy the items on the list. The list is shown continuously to allow them to refer to the list as and when required. Once all the items are in the basket, their companion Aarya congratulates them.
Autistic children often face problems with interaction on a day-to-day basis. Running for errands makes them more independent and helps them assist their parents. This module encourages them to come out of this dilemma and praises them on their achievement. The Kinect motion sensor detects their hand movements and identifies when all the items from the shopping list are placed in the basket. The list is available at all times to prevent confusion. However, it is essential that the children get used to change therefore a feature where the list is hidden is included in the next level. We use raising hand overhead gesture to start the game. This is used in all the modules to prepare them for the game.
Shopping module level 2
In this module, autistic children are encouraged to interact with the shopkeeper and buy the items on the list. The list is hidden after a couple of seconds. On moving their hand over Aarya, they can view the list again as shown in Fig. 2. There are more items in this level as compared to the previous level. Once all the items are in the basket, their companion Aarya congratulates them.
Children with Autism Spectrum Disorder often repel change but it is a part of everyday life. The changes in this level include increase in number of items and hiding the shopping list. The aim is to make them more comfortable with change however if they forget they can always ask Aarya, which encourages interaction.
School module level 1
In the school module, the Autistic children will be placed in a classroom environment. Here, the child will learn the basic class etiquette like raising their hands to answer a question. They will interact with the teacher in the game thus improving their involvement in a real class environment.
In this module, they are taught basics of numbering and alphabets. This virtual environment makes it easier for them to learn as well as open up to teachers. The child needs to raise his/her hand to answer a question; this is done in order to promote classroom interaction and communication with the teacher.
School module level 2
In this level, the children make use of what they learnt in the previous level. They are asked to find the missing numbers and place them in the right position. Since the numbers are removed from the actual position like in Fig. 3, the change might make the child uncomfortable. With the help of Aarya and the teacher, they are encouraged to accept the change. This also tests how much information they absorbed from the first level.
Bedroom module
Here the child is encouraged to keep his/her room clean. At the end of the module the child will be aware that a banana peel should go into the dustbin and that his/her bed has to be clean. Figure 4 shows the screenshot of the bedroom module before the child cleans it. This module in particular is very important and helpful to the parents because it teaches the basic chores to be done in one’s house. This is to make them more independent.
Experimental set up
The aim of this project is to enhance the performance of Autistic children over time as they interact with Aarya on a continuous basis. The performance of the children in the beginning and after continuous monitoring for a period of 1 month is noted.
The research conducted lead to the experimental set up being simple and interesting for children with Autism Spectrum Disorder. The Kinect console is attached to the Computer. Kinect2Scratch application connects Scratch and Kinect. Once this connection is established, the game is activated in Scratch. The computer screen and the console are placed in front of the child. The child is required to follow the instructions given by Aarya and the teacher in the module.
The participants under observation are five children with ASD (4 males, 1 female) in the age group of 7–12 years. The module under observation is the shopping module. A control group of the same age and with the same gender ratio, on an average took 12 seconds to raise their hand above the head, 90 seconds to complete level 1 and 120 seconds to complete level 2.
The results obtained with the autistic children for the shopping module are noted continuously for a period of one month. The time taken for the autistic children to raise hand and complete both the levels is noted simultaneously. Tables 1–4 illustrate the results obtained by this experiment on day 1, day 10, day 20 and day 30 simultaneously.
Results
After research and the subsequent experimentations, the results in Fig. 5 were obtained. From this, it can be concluded that with repetitive and systematic approach, children with Autism Spectrum Disorder can be taught social skills. The teachers and parents of these children were satisfied with the results and suggested we include more modules and levels for the future work. This research has opened the doors for new avenues for using other sensors for teaching and bringing up autistic kids. One such instance can be leap motion sensor, which accepts gesture inputs and it can be a great source of learning if it can be used.
The character in the application, Aarya who acts as a companion to the autistic child, provides instructions to the child throughout and ensures the completion of the task. During the experiment, no child reported a problem with the instructions provided, however a few other problems were faced in the initial days. During the first session of the experiment, few children reported confusion on whether they should stand away from the Kinect sensor or stand right in front of the laptop and few children expected the monitor to be a touch screen. Nevertheless, on attending further sessions, they have shown better results.
One of the main drawbacks faced during this experiment was that not all children with ASD are at the same level of understanding, further research lead to the conclusion that Aarya functions well only with those children that are vocal and understand instructions. The research also confirms that the results obtained with children who have decent vocal and understanding, learn faster and the same is acknowledged by parents of those autistic children.
Future scope
In the future Aarya could be linked to a database so that the teachers and trainers can have access to the child’s records and his/her progress. This database if given remote access can be reviewed by experts in the field. The application can also have an extension wherein the child can follow different and alternative action paths to encourage creativity.
Various other situations that involve social interaction can be included in the form of modules with multiple levels. Aarya could be made customizable in order to meet the needs of individual autistic children; a child might be left handed whereas the game is coded for a right-handed individual.
Footnotes
Acknowledgments
The experiment and research carried out in this project was made possible by the support of the teachers and staff at a learning center called Stepping Stones. We would like to thank and dedicate this work to the kids affected, who supported us for this research. We are extremely grateful to the parents of those kids to permit us to interact with the kids quiet often.