IoT-Based Serious Gaming Platform for Improving Cognitive Skills of Children with Special Needs

Abstract

The objective of this article is to design, develop, and evaluate an Internet of Things (IoT)-based serious gaming platform for children with autism spectrum disorder The platform aims to improve the children’s cognitive skills. To attain this research objective, first, a conceptual framework for developing the gaming platform was proposed. Then, the conceptual ideas were materialized by developing three games (a puzzle game, a function card game, and a road crossing game) to be included in the gaming platform, which also incorporated a mobile application and three hardware systems. The hardware platform was used to play the game, and the mobile application was used to select, control, manage, and store the gaming performance and visualize players’ progress. The proposed gaming platform was evaluated with 15 special needs children. We found that the gaming platform was usable, effective, and useful for the children on the spectrum and noticeably contributed to improving their cognitive skills.

Keywords

ASD serious games cognitive skill learning device IoT RFID

Introduction

Cognitive skills are important in the overall development process of a child and consist of some significant abilities such as attention and logical reasoning (Keen, 2011). This set of skills plays a vital role in processing a variety of sensory information from birth to every phase of life. However, some children may lack these skills, which can cause them to lag in comparison to other children with respect to the progressive mental development process (Charman et al., 2011). This limitation is formally known as cognitive disability or intellectual disability, which can occur for many reasons, such as genetic issues, problems during birth, environmental factors, various health problems, and so on (Harris, 2006). Intellectual disability is a broader term, whereas autism spectrum disorder (ASD) is a development-related disorder that is quite extensively spread throughout the world (Peñagarikano & Geschwind, 2012) and usually refers to some specific conditions that mainly involve repetitive and restricted behaviors and difficulty regarding social interaction and communication (Lord et al., 2000). Approximately 1% of the general population is considered to have an intellectual disability, whereas around 10% of the population is thought to have ASD (Intellectual Disability and ASD | CAR, n. d.).

As cognitive skills are strongly connected with information processing, they have an impact on ability and performance in learning (Kerckhoff et al., 2001). Children with ASD often face several challenges throughout their entire learning process. Designated instructors provide assistive support during their learning process. With the advancement of computerized and automated technology, a huge number of assistive technologies are now being introduced to overcome this challenge and ease the learning process for children with special needs. Some notable assistive technologies involve different mobile apps such as AutEdu, Autism Read WritePro, Otsimo, etc., and some websites such as Do2Learn, IXL Worldwide, FunBrain, etc. Apart from mobile and web applications, different tactile-based tools such as interactive boards, puppet sets, pencil fidgets, puzzles, and building block sand tiles also help to enhance the learning process. Also, developments of interactive digital tabletop learning tools (N. Hasan & Islam, 2020) to provide educational assistance are on the rise. Children are more prone to use games than adults, and it is evident that children with ASD gain significant satisfaction from the gaming experience (Hughes et al., 2016).

Game-based simulations can enhance various learning aspects in driving education (Gounaridou et al., 2021). Due to the special needs of children on the spectrum, education can be provided through gaming rather than conventional approaches. Such games are referred to as serious games, which focus on rendering learning content through a gaming environment (Susi et al., 2007). Some existing serious games for ASD are cMotion (teaches emotion recognition and programming logic) (Finkelstein et al., 2009) and Auction Game (teaches emotion regulation training for financial decision making) (Jerčić et al., 2012).

While providing necessary educational services through assistive technologies, it is also important that the children’s progress is being recorded and analyzed. Such progress tracking applications are also available (e.g., Autism TrackerPro: Track and Analyze ASD). However, most of these applications aim to track behavioral data such as food consumption. Limited works exist on tactile-based serious games that provide learning content as well as corresponding analysis of one’s mental progress.

Therefore, the objective of this research is to develop an Internet of Things (IoT)-based gaming platform for children on the spectrum to improve their cognitive skills. To attain this research objective, first, a conceptual framework to develop IoT-based serious games is proposed based on the literature survey. Then, three serious games were developed and incorporated into an IoT-based gaming platform. The games were controlled through a mobile application that can track the children’s performance in playing games and present the results graphically to help in observing their cognitive development and making recommendations for future improvement. Finally, the serious games were evaluated through an empirical study replicated with real end users (children on the spectrum). It is worth mentioning here that the earlier version of this article was published in a conference proceeding (Hasan et al., 2020).

The relevant literature review is discussed in Section 2. Next, the conceptual framework depicting the interaction between the users and the device is presented in Section 3. Section 4 highlights the development of the system in detail. Then, the evaluation of the system is described in Section 5, followed by a brief conclusion in Section 6.

Related Work

A significant number of studies were conducted focusing on the impact of assistive technology on children with autism. Next, we briefly present the studies related to our work. To understand facial expression and teach basic emotions like anger, joy, love, and surprise, Castillo et al. (2016) developed an assistive tool for children with ASD. Daud et al. (2018) reviewed the impact of assistive learning among students diagnosed with ASD. A study conducted by Hwang and Hughes (2000) showed the relation between early social interaction skills of children with ASD and consecutive development of these children’s communication skills. 16 studies were conducted to show the impact of social interactive training and how it could be a promising technique for stimulating advanced preverbal and verbal communication of children with ASD in daily classroom activities. This research was done to enhance children’s social communication skills within the school environment. Parsons et al. (Parsons et al., 2015) created an innovative digital storytelling technology named Shape. The digital stories were in the form of short films or narrated sequences of slides and images. The objective of this project was to enhance children’s social communication skills within the school environment with the help of school practitioners.

Knight et al. (2013) conducted an in-depth review of articles published between 1993 and 2012 to determine the effectiveness of technology-based interventions to teach academic skills to individuals with ASD. The results suggested that practitioners should take precautions while using technology-based interventions. Additionally, Goldsmith and LeBlanc (2004) evaluated five technology-based interventions, which include (a) tactile and auditory prompting devices, (b) video-based instruction and feedback, (c) computer-aided instruction, (d) virtual reality, and (e) robotics. In another research (Bartoli et al., 2013), a field study was described regarding the potential benefits of touchless and motion-based gaming for autistic children. They conducted the experiment with five children with autism and analyzed their behavior toward motion-based touchless play. Sarah Parsons (Parsons, 2015) applied a conceptual model of learner-centered design to illustrate a novel collaborative virtual environment for children with autism.

Children on the spectrum also face difficulties in remembering things and tend to learn more slowly than children not on the spectrum. In this regard, Tanaka et al. (2010) investigated the Let’s Face It! program and pre-screened 79 children diagnosed with ASD to examine their face and object processing abilities. The results showed that the Let’s Face It! computer-based intervention was adaptable to the specific learning needs of the individual child and suitable for both home and school. Another cost-effective, portable, and user-friendly interactive learning tool was introduced in Zaki et al. (2017), where a pressure sensing keypad was used to provide an easy and flexible means of interaction for children with autism. The effectiveness and usability of this tool were also evaluated. Islam et al. (2018) demonstrated and reviewed the existing online applications and developed an online support system, namely, Autism Sohayika. This online application mainly focused on children with autism and their parents in Bangladesh.

Recently, a considerable amount of research studies have also been conducted on innovative gaming applications for educational purposes. The use of serious games is one of the most significant (Luigini & Basso, 2021). Based on serious games, Fridenson-Hayo et al. (2017) performed a cross-cultural evaluation of a game, namely, Emotiplay. Also, Tsikinas and Xinogalos (2019) examined 54 studies on the effect of serious games on children with ASD. The results showed that learning through serious games is considered promising in this field. Hussaan et al. (2011) proposed an intelligent module whose purpose was to develop a serious game with adaptive pedagogical scenarios for the diagnosis and training of children with cognitive disabilities.

A list of existing technologies is presented in Table 1. According to this table, most of the technologies are focused on improving communication skills and basic academic learning. A few of them emphasized emotion/facial recognition and cognitive skills. Similarly, only a limited number of studies have focused on the design, development, and evaluation of serious games for children with ASD. However, there was no such research explicitly focused on improving cognitive skills through an IoT-based serious gaming platform. IoT-based gaming devices usually consist of both hardware and software parts that provide a more realistic gaming experience. To improve teaching and learning, IoT in combination with game-based learning offers a new dimension and a wide range of possibilities (Petrović et al., 2017). Therefore, this research is focused on developing an IoT-based serious gaming platform for children with autism that will expedite the improvement of their cognitive skills.

Table 1.

Summary of the Existing Technologies.

Reference	Technology	Objective
(Castillo et al., 2016)	Web environment	Teaching and identification of basic emotions
(Parsons et al., 2015)	Collaborative virtual-reality environment (CVE)	To support communicative perspective-taking skills (teaching how to relate to other people)
(Tanaka et al., 2010)	Let’s Face It	Improve facial recognition skills
(Zaki et al., 2017)	Portable learning tool	Teach basic academics (English alphabets) using a pressure sensing keypad
(Islam et al., 2018)	Autism	Develop an online support system for autistic children and their parents
(Islam et al., 2018)	Sohayika
(Fridenson-Hayo et al., 2017)	Serious game-based online Emotiplay app	Cross-cultural effectiveness evaluation of a serious game to teach recognition emotion (using face, voice, body, and their integration) to the children with ASD
(Hussaan et al., 2011)	Computer-based serious games	Generate adaptive pedagogical serious game scenarios for training of children with cognitive disabilities
(Aresti-Bartolome & Garcia-Zapirain, 2015)	A touch screen and tactile pointer-based serious game app	Assess the effectiveness (using eye-tracking parameters) of such serious game to incorporate it as a cognitive rehabilitation tool
(Omelina et al., 2012)	Specialized configuration interface computer-based serious game	Proposed architecture for developing computer-based serious games aiming to neuromuscular rehabilitation
(Torrente et al., 2012)	eAdventure game authoring tool	Develop of two serious games to evaluate improvement of professional education
(Luigini & Basso, 2021)	Web-based serious game	A web-based application of an immersive serious game was presented as a suitable solution for addressing the problem of how VR could be applied to distance learning

Conceptual Framework

While the IoT-based gaming experience adds a completely new dimension to the concept of gaming, serious games are designed to serve a broader purpose than just mere entertainment. As per (De Gloria et al., 2014), the interest in serious games for the purpose of education and training is ever increasing, and they can create a challenging and realistic gaming environment supporting situated cognition.

The core idea of this research is to develop an IoT-based gaming platform that contains serious games to teach basic cognitive skills or behavior to a child on the spectrum through hands-on learning and keep a record of his/her gradual improvement. The conceptual framework for the development of such an interactive learning and progress tracking gaming platform is shown in Figure 1. The framework depicts how the different components of the device are connected and transmit data among each other. Also, how a child selects and plays a game, how the progress is recorded, stored, and displayed, and the way users (children, therapists, and parents) interact with the device are shown. The gaming platform has two main parts: a hardware game box and a mobile application. The connected mobile application is used to control the hardware games. The platform includes three games that have been designed based on the concepts of serious games for the purpose of teaching children with special needs. The purpose of these games is to improve basic cognitive behaviors of the child, such as remembering, reasoning, cautionary, learning, and attention skills.

Figure 1.

Conceptual design of the proposed gaming tool.

The details of the three games in the platform are as follows:

(a) Function card game: This game consists of some function cards to teach behaviors that are right or wrong. There are four options in the function card, among which only one is correct corresponding to the question asked in the card. These cards can be used to teach precautionary behaviors to a child or to teach actions that are correct or incorrect. The child must press the correct option on top of a card. These cards can serve the purpose of teaching basic objects like animals, alphabets, fruits, etc. and teaching the alphabet to the child. When the child presses an option on the card, the animation of the corresponding pressed option appears on the mobile application. This game aims to test the long-term memory or remembering skills of the child and develop precautionary habits by identifying the correct actions to take.

(b) Road crossing game: This game is an Android game where the player is to cross a road safely, avoiding obstacles like cars, objects, etc. on the road. A joystick is used to control the movement of the character for crossing the road. A player wins once he/she can successfully move the game character from one side of the road to the other, avoiding the hurdles on the way. It teaches a child basic precautionary measures to follow while crossing a busy road and also teaches basic movement directions like left and right to develop his/her attention skills.

(c) Puzzle solving game: This is a basic puzzle game with six parts. The aim is to complete a sequence of doing a particular task. The tasks can be any day-to-day activities like washing hands, brushing teeth, getting ready for school, etc. Six images of a complete task are given in each of the six puzzle cards. A child needs to place each of the cards step by step to complete the entire task and thereby win the game. The logical reasoning of why it is important to do one activity after the other and also the learning capabilities of the child is developed through this game.

To summarize, the main features of our cognitive skill-developing gaming platform are as follows:

1) Playing cognitive games: Three basic types of cognitive skill-based games can be played by a child. These games are fun to play, and their hands-on nature will impact the long-term learning and improvement of a child.

2) Individual account creation: A user account will be created for an individual child to provide authorized access.

3) On-spot result display: The success/failure of playing a game will be immediately shown through multimodal feedback like audio, animation, and text or a combination of them.

4) Individual’s progress record: Data related to playing date, number of trials to accomplish the game, gaming results, or scores will be stored for each player, which will be accessible only by the authorized person.

5) Generating progress charts: The system will generate progress charts to show the individual’s progress within a specific period.

6) Viewing progress and supporting decision making: An individual’s progress can be observed by any authorized person to take necessary actions for future improvement of a child.

Developing the Platform

The development of the gaming platform is broadly discussed in this section. Figure 2 depicts the external outlook of the gaming device. The entire development was carried out at the Software Engineering Laboratory of the authors’ institute. As mentioned earlier, the gaming platform consists of a hardware device and an associated mobile application. The mobile application is wirelessly connected with the device and the server to store data. Players’ interactions and actions with the device are evaluated and stored to the server. Previous data can be retrieved, viewed, and analyzed from the server using this application.

Figure 2.

The developed prototype.

The hardware is activated from a power source. There are three switches corresponding to the three games on the device. When the respective switch is on, the connected devices get power. There are four piezo sensors fitted on the face of the game box on the right for the object-finding game. It also has a slot for keeping the function cards. On the left side of the device, there are six RFID readers fitted on top, which is basically the box for a puzzle game. RFID is one kind of automatic identification and data capturing device that automatically identifies an object, collects data about the identified object, and then transfers that data directly to the computer system through radio waves with little or no human intervention. A joystick module is placed on the middle box, which is for the road crossing game. It has an arrangement to keep a mobile phone standing while playing the game.

The main hardware component is the Arduino Mega, which connects the other components or modules. Data from the piezo sensors, RFID, and joystick modules are received in the Arduino. Then, this data is passed to the Android application through the Bluetooth module connected to Arduino, where the results or animation can be viewed.

For children to learn or adopt new behaviors, playing games can be an effective approach (Garris et al., 2002). Our first game (Figure 3) is a basic function card game. Any function card can be used to teach some basic objects or activities to do in a specific situation. All the function cards consist of a RFID tag and multiple-choice question with four options in the picture format. An RFID tag reader and four piezo sensors are placed under the function card to identify any card uniquely and to detect the answer pressed by the player. For example, in Figure 3, the function cards displayed teach a child what actions are right and what are wrong or to be avoided to stay safe from the COVID-19. The child is either asked to identify which action out of the four options shown on the card is correct (Figure 3(a)–(b)) or which one is incorrect (Figure 3(c)–(d)). These cards have been designed in line with the precautionary actions guided by WHO (Advice for the public on COVID-19 – World Health Organization, n.d.) to avoid the risk or spread of COVID-19. Some include maintaining at least 1 m (3 feet) distance from people, avoiding touching eyes, nose, and mouth, covering mouth and nose with bent elbow or tissue when coughing or sneezing, washing hands with soap or alcohol-based hand rub, wearing safety masks, staying home or self-isolation, etc. In the case of a card asked to find the correct action, visuals of three wrong actions and one correct action are shown on the card. When a card is placed on the corresponding position of the box, the RFID tag attached to the function card is read by the RFID reader on the box, and this card is then shown on the mobile app.

Figure 3.

(a), (c) Function card game box and (b), (d) respective app interface.

There is a piezo sensor under each of the objects or characters on the card. When an object is pressed on the card, an animation of the correct or wrong answer is shown on the app, and the result is recorded in the database.

The second game is a puzzle game designed to teach some sequence of actions to be done to complete an entire task. The game window is divided into six equal parts. Each part has an independent RFID reader. There are six different puzzle cards consisting of a unique part of a picture and a unique RFID tag. Every tag can be identified using the RFID readers and the respective tag number is sent to the mobile application. The puzzle cards are to be placed in the appropriate order to win the game. Learning the steps of a task by playing puzzles is effective for remembering. One of the puzzle games displayed in Figure 4 has been designed to teach a child the steps provided by UNICEF (Coronavirus Disease (COVID-19), n. d.) to wash hands properly. The steps are as follows:

Figure 4.

Mobile app interface of puzzle game (steps to wash hands).

Step 1

Wet hands with running water.

Step 2

Next, apply enough soap to cover the wet hands.

Step 3

Then, scrub the entire surface of the hands (for at least 20 s).

Step 4

After that, rinse thoroughly with running water.

Step 5

Finally, dry hands with a clean cloth or one time-use towel.

The puzzle game also has a hardware part connected to the mobile application. To play the puzzle game, the corresponding switch on the hardware device is to be pressed. The RFID readers on the game box on the left side read the tags on the back of the puzzle cards and then send the data to the mobile app where the puzzle parts are shown (in a similar way, the players placed the cards on the game box), which is demonstrated in Figure 4.

The third game (Figure 5(a)) is a game that teaches a child how to cross a road safely, avoiding obstacles. There is a joystick controller placed in the middle of the gaming device. If the player starts playing this game, joystick movements are sent to the mobile application. When the switch for the road crossing game is selected, a person can be seen on the game in the Android app. The person in the game can be controlled by moving the joystick on the game box. The child needs to make that person cross the road through a zebra crossing. Then, the success/failure of crossing the road is stored on the database connected to the mobile app. The game is won when the road is successfully crossed, avoiding any kind of obstacles. A flow diagram to show the working procedure of the proposed gaming application is shown in Figure 6.

For the mobile application of the gaming system, firebase authentication and database were used for account creation and result storage. Primary users of the app are children on the spectrum who will play the games, and their parents/teachers/doctors will monitor or observe the progress. Guardians/teachers of a child should open an account for a child to initiate game playing and to keep his/her progress through this application. Parents or authorized users can log in to the system with the mobile app, select the game that the child wants to play, and modify any kind of game settings from the student/child view. The parents/doctors can regularly check the progress of the child through a graphical view on the mobile app (Figure 5(b)) and get a general overview of a child’s gradual progress by selecting the parents/doctor view on the app. This progress is calculated from the results or performance of playing the games.

Figure 5.

Mobile app interface of (a) road crossing (crosswalk) game, and (b) a child’s progress of playing the games in three sessions (a gap of 4 days).

Figure 6.

Flow diagram of the developed games.

Figure 7.

A child is playing with the device.

Figure 8.

Effectiveness of function card game.

Figure 9.

Effectiveness of puzzle game.

Figure 10.

Effectiveness of the road crossing game.

Evaluating the Platform

We conducted a study to evaluate the gaming platform. This section presents the evaluation study objectives, participants’ profile, study procedure and the data analysis and findings of the evaluation study.

Study Objectives

The goal of the evaluation was to find out how effective the platform is for children on the spectrum in terms of their cognitive development. Another objective was to determine to what extent it serves the learning purpose and its usefulness in terms of real-time usage. Moreover, we also seek to identify UI problems.

Participants Profile

A total of 15 children were recruited from a local specialized school for children on the spectrum to conduct the evaluation study. Among them, eight were boys and seven were girls. Their average age was approximately 5 years and ranged between four and 9years. During the selection process, a screening process was conducted to ensure that none were familiar with the topics/concepts (e.g., COVID-19 safety, safety measures during road crossing, etc.). According to their respective teachers, the children learned slowly and had different levels of mental proficiency. None of the participants had prior experience of any integrated hardware- and mobile-based game for learning purposes.

Study Procedure

To gain proper insight from the evaluation data, the device was evaluated among the same participants in three sequential sessions that occurred over a span of 4 days each. In each of the three sessions, the performance data were recorded.

During the first day of the evaluation study, a brief presentation about the objective of this study was given to the participants, including the children and their instructors. After that, the usage procedure of the proposed system was demonstrated, and they were given the chance to explore and use the device for around 15 minutes. Finally, the participants were asked to start playing the games. This was the first session conducted on day 1 of playing the games. Data about the number of attempts, success rate, game completion time in seconds, number of times help was requested, and overall satisfaction were recorded while they played the games. On the consecutive sessions on day 5 and day 9, the same data were collected from the players. For data collection, test sessions were audio-video recorded, and later the recorded videos were meticulously manually observed to code or record the data in a tabular format for performance analysis. In this case, quantitative (objective) data were collected and analyzed through descriptive statistics.

Finally, on the last session day, the instructors were asked to provide their opinion about how they think the device can impact the learning of children and the usability and effectiveness of the platform. They were also asked to give any other related recommendations for its improvement. Their opinions and feedback were recorded with permission from the instructors; then, the recorded data were transcribed for qualitative data analysis.

Data Analysis and Results

The device was evaluated to determine its performance among the end users. For performance measurement, two parameters were studied: effectiveness and efficiency of the device.

Effectiveness

As per (Frøkjær et al., 2000), the accuracy and completion of goals is effectiveness. A measure of effectiveness may include the successful completion of given tasks with a minimum possible number of attempts. In this study, two variables were used to measure the device’s effectiveness: (1) success rate (number of successes out of a certain number of trials) and (2) number of attempts (total how many times each game played).

The success rate for the games have been calculated separately. For the function card and road crossing game, the success rate is defined as the number of times the child has successfully played or won the game out of five times of playing each of the two games. While in the case of the puzzle game, this is the number of successes out of three times of playing. Each of these data were taken on the consecutive sessions with a gap of 4 days in each as mentioned earlier. From the data recorded, it is observed that on an average, the success rate of a function card game was 3.00, 3.23, and 3.53 in sessions 1, 2, and 3, respectively. That means there has been an improvement in the number of successfully played games in each of the consecutive sessions. In case of the road crossing game, the success rate was also found to increase gradually in each of the sessions. From the first session to the second session, the success rate increased by 0.43, whereas from the second to the third session, the success rate increased by 0.09. Also, a notable increase in success rate is observed in the case of the puzzle game. A brief summary of the recorded data is given in Table 2 and Table 3.

Table 2.

Survey Response of the Participants in Terms of Effectiveness.

Data Type	Game	Session-1 Sec Day 1) [Mean ± SD]	Session-2 Sec (Day 5) [Mean ± SD]	Session-3 Sec (Day 9) [Mean ± SD]
Success rate	Function	3.00 ± 0.70	3.23 ± 0.74	3.53 ± 0.68
	Card game	3.00 ± 0.70	3.23 ± 0.74	3.53 ± 0.68
	Puzzle game	0.42 ± 0.49	0.73 ± 0.37	0.88 ± 0.45
	Road crossing game	2.8 ±.50	3.23 ± 0.46	3.34 ± 0.58
Number of attempts	Function	7.42 ± 1.03	6.83 ± 0.98	6.78 ± 1.01
	Card game	7.42 ± 1.03	6.83 ± 0.98	6.78 ± 1.01
	Puzzle game	4.66 ± 0.84	4.42 ± 0.75	4.3 ± 0.49
	Road crossing game	8.32 ± 1.33	8.43 ± 0.98	8.02 ± 1.11

Table 3.

Survey Response of the Participants in Terms of Efficiency.

Data Type	Game	Session-1 Sec (Day 1) [Mean ± SD]	Session-2 Sec (Day 5) [Mean ± SD]	Session-3 Sec (Day 9) [Mean ± SD]
Game completion time	Function	57.83 ±7.20	50.25 ±7.74	48.7 ± 6.59
	Card game	57.83 ±7.20	50.25 ±7.74	48.7 ± 6.59
	Puzzle game	214 ± 33.7	194 ± 26.7	180 ± 27.1
	Road crossing game	81.4 ± 9.20	85.25 ±7.74	77.89 ± 1.89
Number of times asking help	Function	2.83 ± 0.8	2.33 ± 0.74	2.12 ± 0.83
	Card game	2.83 ± 0.8	2.33 ± 0.74	2.12 ± 0.83
	Puzzle game	7.63 ± 1.4	7.72 ± 1.74	7.28 ± 1.32
	Road Crossing game	4.83 ± 0.85	4.53 ± 0.74	4.31 ± 1.13

On average, the participants took 2.42, 1.83, and 1.78; and 3.32, 3.43, and 3.02 more attempts than the required number of attempts (i.e., five attempts) for the function card game and road crossing game, respectively, in each session (see Table 2), which in turn implies that the children needed gradually less number of attempts (greater efficiency in playing) by using the device in subsequent days. Also, in the case of the puzzle game, the number of attempts needed gradually decreased in the subsequent sessions, indicating gradual progress in learning performance.

Efficiency

Efficiency is defined as the resources expended to complete a task (Ahmed et al., 2019). For this evaluation, two variables were used to measure the game’s efficiency: (1) game completion time and (2) number of times help was asked from the instructor while game playing.

In the case of game completion time, the results showed that participants took on average 57.83, 50.25, and 48.7 seconds to play the function card game five times in the three sessions, respectively. For the road crossing game, an average of 81.4, 85.25, and 77.89 seconds were taken by the players in the same respective sessions. The average time needed to play the puzzle game was 214, 194, and 180 s in the three sessions, respectively. Regarding the number of times asking for help, due to the complexity of the puzzle game, the participants asked for help more often compared to the other two games.

Subjective Feedback

The subjective feedback (qualitative data) received from the instructors at the end of test sessions was analyzed through a thematic analysis approach (Braun & Clarke, 2006; Ibrahim, 2012). The findings from the thematic analysis are discussed below.

Easy to view and maintain progress: The developed prototype was admired by the instructors. Some of them opined that the graphical results generated each time by the platform is an innovative and hassle-free way for them to view and maintain progress results of the children on the spectrum in the schools and in home tutoring.

Track progress and provide recommendations: As the device stores data and tracks the progress, it could be a useful tool for the doctors/therapists to monitor the cognitive progress of a child on the spectrum and to provide useful recommendations for their further development. One instructor stated, “This tool could be used for both purposes like teaching them and for their treatment purpose as well!”

Learning by doing: Instructors found the gaming tools to be an interesting and useful device for improving the learning skills of children on the spectrum. One of the instructors stated that “Such an interactive educational device can draw their attention more toward learning and help in further retention of information due to hands-on practicing”. Another instructor opined that “Gaming has always been a better way to educate children on the spectrum and this device will add a new dimension due to its easy interface and interesting games.”

Discussion and Conclusion

Principal Results

The main outcome of this research is an IoT-based gaming platform that includes serious games for developing the cognitive skills of children on the spectrum and supporting their education. This platform offers an interactive form to learn basic cognitive skills and can store players’ gaming results to show (graphically) their progress within a specific time range. As a combination of both hardware and software games, it will add a new dimension to the learning methods available and ease the process of making recommendations for therapists. An evaluation study was conducted with real end users, which showed that the children gained skills using the gaming platform. The participants found the games interesting and useful for learning in an interactive way and improving their cognitive skills.

Comparison with Prior Works

In contrast to the previous research, the contributions of this research could be defined as follows. First, although the potential benefits of gaming for children with autism are explored in (Bartoli et al., 2013), most of the earlier studies focused on teaching basic education, programming logic, social interaction, and communication skills through pictures, audio-video, voice annotation, and digital storytelling (Daud et al., 2018; Finkelstein et al., 2009; Hwang & Hughes, 2000; Parsons et al., 2015). Again, although a few studies have developed gaming applications for children with ASD, most of those are for basic education and financial decisions (or counting) (Jerčić et al., 2012). This research focused on developing cognitive skills through gaming to stimulate the learning process of children with ASD. As such, the proposed system seems more interactive, engaging, and interesting.

Second, most of the assistive technologies developed for the children on the spectrum are either a mobile app, desktop application, or a hardware system, while the proposed IoT-based gaming system includes both hardware and software games and aims to help improve the cognitive skills of children with special needs and ease the process of monitoring.

Third, applications discussed in the earlier studies were adopted or used mainly by the parents and teachers in addition to the children with ASD, while the proposed system would be adopted not only by the teachers, parents, and children but also by the therapists to track or monitor their cognition process/state for learning basic and social communication skills.

Fourth, limited applications have been developed considering the features of keeping and tracking the progress of children with ASD (Daud et al., 2018). Some used either application or hardware tools where tracking application works independently according to the manually entered data. For example, in (Daud et al., 2018) Autism TrackerPro can track and analyze ASD, but the user must put data manually. No study was conducted on combining the hardware and software approaches where data is generated and recorded automatically, whereas our developed system combined both approaches, where data was generated periodically over time.

Limitations

Our work has a few limitations. First, the proposed platform was developed targeting children aged 4 years or above. We evaluated the platform with a limited number of participants. Therefore, in future work, we plan to test with more end users (children/instructors) through field study. Second, we included only three games in the proposed platform. Therefore, the proposed gaming platform can be upgraded by including more games.

Conclusions

Cognitive skills are critical to the overall development process of a child. The proposed IoT-based gaming platform includes serious games for children on the spectrum to improve their cognitive skills. The study found the proposed system interesting, engaging, and useful for them to learn in an interactive way, keep their progress over time, and compare, which in turn will contribute to improving their cognitive skills and learning.

Supplemental Material

Footnotes

Acknowledgments

The authors would like to thank all the participants whose participation made the study possible. Their efforts are gratefully acknowledged.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Muhammad Nazrul Islam

Supplemental material

Supplemental material for this article is available online.

Authors Biographies

Muhammad Nazrul Islam is an Associate Professor at the department of computer science and engineering at the Military Institute of Science and Technology (MIST), Mirpur Cantonment, Dhaka, Bangladesh. He was awarded a PhD in Information Systems from ÅboAkademi University (Finland) in 2014, a M.Sc. in computer engineering from Politecnico di Milano (Italy) in 2007, and a BSc in computer science and information technology from Islamic University of Technology (Bangladesh) in 2002. His research areas include but not limited to human-computer interaction (HCI), humanitarian technology, health informatics, military information systems, information systems usability, and computer semiotics. He is the author of more than 100 peer-reviewed publications in International journals and conferences.

Uzma Hasan completed her BSc in Computer Science and Engineering from the Military Institute of Science and Technology (MIST). Currently, she’s working as a Lecturer in MIST for more than a year. Her primary research interest lies in the areas of Artificial Intelligence, Machine Learning and HCI. Her research works have been published in International Springer conferences.

Fourkanul Islam completed his BSc in Computer Science and Engineering from the Military Institute of Science and Technology (MIST. His primary research interest lies in the areas of Serious Game, Simulation, Machine Learning and HCI. His research works have been published in International Springer conferences

ShailaTajmimAnuva completed her graduation in Computer Science and Engineering from the Military Institute of Science and Technology, Bangladesh, in 2020. Two of her research papers based on human-computer interaction (HCI) have been published in international journals. Her research interest includes health informatics, human-computer interaction (HCI).

TarannumZaki has completed her undergraduate degree in Computer Science and Engineering (CSE)in 2016 from Military Institute of Science and Technology (MIST), Mirpur Cantonment, Dhaka, Bangladesh. She has also completed her Master’s degree in CSE from the same institute in February 2021. Her research interests cover Human-Computer Interaction, Data Analytics, Internet of Things, and Cognitive Science and Technology. Currently, she is appointed as a lecturer in the Department of CSE at MIST from July 2018.

A.K.M. Najmul Islam is an Adjunct Professor at Tampere University, Finland. He is a Scientist at LUT University, Finland. He also works as University Research Fellow at the Department of Future Technologies, University of Turku, Finland. Dr Islam holds a PhD (Information Systems) from the University of Turku, Finland and an M.Sc. (Eng.) from Tampere University of Technology, Finland. He has 90+ publications. His research focuses on Human Centered Computing.

References

Advice for the public on COVID-19 – World Health Organization (n.d.). https://www.who.int/emergencies/diseases/novel-coronavirus-2019/advice-for-public

Ahmed

Md. A.

Islam

M. N.

Jannat

Sultana

(2019). Towards developing a mobile application for illiterate people to reduce digital divide. In: 2019 International Conference on Computer Communication and Informatics (ICCCI), Phuket, Thailand, 23-26 March 2020 (pp. 1–5). https://doi.org/10.1109/ICCCI.2019.8822036

Aresti-Bartolome

Garcia-Zapirain

(2015). Cognitive rehabilitation system for children with autism spectrum disorder using serious games: A pilot study. Bio-Medical Materials and Engineering, 26(s1), S811–S824. https://doi.org/10.3233/BME-151373

Bartoli

Corradi

Garzotto

Valoriani

(2013). Exploring motion-based touchless games for autistic children’s learning. In: Proceedings of the 12th international conference on interaction design and children. ACM, 102–111. https://doi.org/10.1145/2485760.2485774

Braun

Clarke

(2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101. https://doi.org/10.1191/1478088706qp063oa

Castillo

T. A.

de Celis

C. P.

Lara

Somodevilla

M. J.

Pineda

I. H.

de Alba

K. F.

Romero

(2016). Authic: Computational tool for children with autistic spectrum disorder. In: 2016 International Symposium on Computers in Education (SIIE), Castilla y León, Spain, 13–15 September. 2016, 1–6. https://doi.org/10.1109/SIIE.2016.7751838

Charman

Pickles

Simonoff

Chandler

Loucas

Baird

(2011). IQ in children with autism spectrum disorders: Data from the special needs and autism project (SNAP). Psychological Medicine, 41(3), 619–627. https://doi.org/10.1017/S0033291710000991

Coronavirus disease (COVID-19): What parents should know (n.d.). https://www.unicef.org/easterncaribbean/coronavirus-disease-covid-19-what-parents-should-know

Daud

Maria

Shahbodin

Ahmad

(2018, October 3). Assistive technology for autism spectrum disorder: A review of literature. Proceedings of International MEDLIT Conference 2018, Kuala Lumpur, Malaysia, 5-6 March 2018, pp 1–7.

10.

De Gloria

Bellotti

Berta

(2014). Serious games for education and training. International Journal of Serious Games, 1(1). https://doi.org/10.17083/ijsg.v1i1.11

11.

Finkelstein

S. L.

Nickel

Harrison

Suma

E. A.

Barnes

(2009). cMotion: A new game design to teach emotion recognition and programming logic to children using virtual humans. In: IEEE Virtual Reality Conference, Lafayette, LA, USA, 14–18 March 2009, 249–250. https://doi.org/10.1109/VR.2009.4811039

12.

Fridenson-Hayo

Berggren

Lassalle

Tal

Pigat

Meir-Goren

O'Reilly

Ben-Zur

Bölte

Baron-Cohen

Golan

(2017). ‘Emotiplay’: A serious game for learning about emotions in children with autism: results of a cross-cultural evaluation. European Child & Adolescent Psychiatry, 26(8), 979–992. https://doi.org/10.1007/s00787-017-0968-0

13.

Frøkjær

Hertzum

Hornbæk

(2000). Measuring usability: Are effectiveness, efficiency, and satisfaction really correlated? In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems - CHI ’00, New York, 1–6 April 2000, 345–352. https://doi.org/10.1145/332040.332455

14.

Garris

Ahlers

Driskell

J. E.

(2002). Games, motivation, and learning: A research and practice model. Simulation & Gaming, 33(4), 441–467. https://doi.org/10.1177/1046878102238607

15.

Goldsmith

T. R.

LeBlanc

L. A.

(2004). Use of technology in interventions for children with autism. Journal of Early and Intensive Behavior Intervention, 1(2), 166–178. https://doi.org/10.1037/h0100287

16.

Gounaridou

Siamtanidou

Dimoulas

(2021). A serious game for mediated education on traffic behavior and safety awareness. Education Sciences, 11(3), 127. https://doi.org/10.3390/educsci11030127

17.

Harris

J. C.

(2006). Intellectual disability: Understanding its development, causes, classification, evaluation, and treatment. Oxford University Press.

18.

Hasan

Islam

M. N.

(2020). Exploring the design considerations for developing an interactive tabletop learning tool for children with autism spectrum disorder. In Pandian

A. P.

Palanisamy

Ntalianis

(Eds), Proceeding of the international conference on computer networks, big data and IoT (ICCBI - 2019). Springer International Publishing. (Vol. 49, pp. 834–844). https://doi.org/10.1007/978-3-030-43192-1_91

19.

Hasan

Islam

M. F.

Islam

M. N.

Bin Zaman

Tajmim Anuva

Islam Emu

Zaki

(2020). Towards developing an IoT based gaming application for improving cognitive skills of autistic kids. In Sitek

Pietranik

Krótkiewicz

Srinilta

(Eds), Intelligent information and database systems. Springer Singapore. (1178, pp. 411–423). https://doi.org/10.1007/978-981-15-3380-8_36

20.

Hughes

D. E.

Vasquez

Nicsinger

(2016). Improving perspective taking and empathy in children with autism spectrum disorder. In: IEEE International Conference on Serious Games and Applications for Health (SeGAH), Dubai, UAE, 4–6 August. 2021, 1–5. https://doi.org/10.1109/SeGAH.2016.7586232

21.

Hussaan

A. M.

Sehaba

Mille

(2011). Tailoring serious games with adaptive pedagogical scenarios: A serious game for persons with cognitive disabilities (pp. 486-490). IEEE 11th International Conference on Advanced Learning Technologies. https://doi.org/10.1109/ICALT.2011.152.

22.

Hwang

Hughes

(2000). The effects of social interactive training on early social communicative skills of children with autism. Journal of Autism and Developmental Disorders, 30(4), 331–343. https://doi.org/10.1023/A:1005579317085

23.

Ibrahim

(2012). Thematic analysis: A critical review of its process and evaluation. West East Journal of Social Sciences. 1(1), 9.

24.

Intellectual Disability and ASD | Center for Autism Research (n.d.). https://www.carautismroadmap.org/intellectual-disability-and-asd/

25.

Islam

M. N.

Kabir

Sultana

Ferdous

C. N.

Zaman

Bristy

U. H.

Moumi

P. K.

Tamanna

(2018). Autism sohayika: A web portal to provide services to autistic children. In Younas

Awan

Ghinea

Catalan Cid

(Eds), Mobile web and intelligent information systems. Springer International Publishing (Vol. 10995, pp. 181–192). https://doi.org/10.1007/978-3-319-97163-6_15.

26.

Jerčić

Astor

Adam

Hilborn

Schaff

Lindley

Sennersten

Eriksson

(2012, June 1). A serious game using physiological interfaces for emotion regulation training in the context of financial decision-making. ECIS 2012 - Proceedings of the 20th European Conference on Information Systems, 11–13 June, 2012, Barcelona, SPAIN.

27.

Keen

(2011). The development of problem solving in young children: A critical cognitive skill. Annual Review of Psychology, 62(1), 1–21. https://doi.org/10.1146/annurev.psych.031809.130730

28.

Kerckhoff

A. C.

Raudenbush

S. W.

Glennie

(2001). Education, cognitive skill, and labor force outcomes. Sociology of Education, 74(1), 1. https://doi.org/10.2307/2673142

29.

Knight

McKissick

B. R.

Saunders

(2013). A review of technology-based interventions to teach academic skills to students with autism spectrum disorder. Journal of Autism and Developmental Disorders, 43(11), 2628–2648. https://doi.org/10.1007/s10803-013-1814-y

30.

Lord

Cook

E. H.

Leventhal

B. L.

Amaral

D. G.

(2000). Autism spectrum disorders. Neuron, 28(2), 355–363. https://doi.org/10.1016/S0896-6273(00)00115-X

31.

Luigini

Basso

(2021). Heritage education for primary age through an immersive serious game. In Bolognesi

Villa

(Eds), From building information modelling to mixed reality. Springer International Publishing (pp. 157–174). https://doi.org/10.1007/978-3-030-49278-6_10

32.

Omelina

Jansen

Bonnechère

Van Sint Jan

Cornelis

(2012, September 1). Serious games for physical rehabilitation: Designing highly configurable and adaptable games. In Proceedings of the 9th International Conference on Disability, Virtual Reality & Associated Technologies, 10-12 September, 2012, Laval, France, pp. 195–201).

33.

Parsons

(2015). Learning to work together: Designing a multi-user virtual reality game for social collaboration and perspective-taking for children with autism. International Journal of Child-Computer Interaction, 6, 28–38. https://doi.org/10.1016/j.ijcci.2015.12.002

34.

Parsons

Guldberg

Porayska-Pomsta

Lee

(2015). Digital stories as a method for evidence-based practice and knowledge co-creation in technology-enhanced learning for children with autism. International Journal of Research & Method in Education, 38(3), 247–271. https://doi.org/10.1080/1743727X.2015.1019852

35.

Peñagarikano

Geschwind

D. H.

(2012). What does CNTNAP2 reveal about autism spectrum disorder? Trends in Molecular Medicine, 18(3), 156–163. https://doi.org/10.1016/j.molmed.2012.01.003

36.

Petrović

Jezdović

Stojanović

Bogdanović

Despotović-Zrakić

(2017). Development of an educational game based on IoT. IJEEC - International Journal of Electrical Engineering and Computing, 1(1), pp. 36–45. https://doi.org/10.7251/IJEEC1701036P

37.

Susi

Johannesson

Backlund

(2007). Serious games: An overview . Institutionen för kommunikation och information. http://urn.kb.se/resolve?urn=urn:nbn:se:his:diva-1279

38.

Tanaka

Wolf

Klaiman

Koenig

Cockburn

Herlihy

Brown

Stahl

Kaiser

Schultz

(2010). Using computerized games to teach face recognition skills to children with autism spectrum disorder: The Let’s face it! Program: face training. Journal of Child Psychology and Psychiatry, 51(8), 944–952. https://doi.org/10.1111/j.1469-7610.2010.02258.x

39.

Torrente

del Blanco

Á.

Moreno-Ger

Fernández-Manjón

(2012). Designing serious games for adult students with cognitive disabilities. In Huang

Zeng

Leung

C. S.

(Eds), Neural information processing. Springer Berlin Heidelberg (7666, pp. 603–610). https://doi.org/10.1007/978-3-642-34478-7_73

40.

Tsikinas

Xinogalos

(2019). Studying the effects of computer serious games on people with intellectual disabilities or autism spectrum disorder: A systematic literature review. Journal of Computer Assisted Learning, 35(1), 1–73. https://doi.org/10.1111/jcal.12311

41.

Zaki

Islam

M. N.

Uddin

Md. S.

Tumpa

S. N.

Hossain

MD. J.

MD Anti

M. R.

Hasan

MD. M.

(2017). Towards developing a learning tool for children with autism. 2017. In: 6th International Conference on Informatics, Electronics and Vision & 2017 7th International Symposium in Computational Medical and Health Technology (ICIEV-ISCMHT), Hyogo, Japan, 1–3 September. 2017, 1–6. https://doi.org/10.1109/ICIEV.2017.8338597

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB