Waking up in the morning (WUIM): A transmedia project for daily living skills training

Abstract

BACKGROUND:

One of the most serious concerns of parents, caregivers, teachers and therapists is children’s independent living, particularly of those with special educational needs (SEN). Purpose-built programs for the acquisition of independent living skills are considered a priority in special education settings. The main problem is the inefficacy of detached interventions to meet the needs of as many students as possible.

OBJECTIVE:

Our response is to create transmedia applications for inclusive learning environments. To this end, we have taken a participatory design approach to develop a project for Daily Living Skills Training by combining special education pedagogies, filmic methods, game design and innovative technologies. In this paper, we present the design and development of Waking up In the Morning (WUIM), and its improvement through user-based and expert-based evaluations by students, therapists and developers. The main research purpose is to confirm if: (1) the final products of the WUIM project could be educational resources for students with SEN and (2) the common gaming experience could promote collaborative learning, regardless of students’ cognitive profile.

METHODS:

During the alpha phase, we developed and improved WUIM. In July 2020, we implemented and evaluated WUIM in special education settings (beta-phase). More specifically, a quantitative and qualitative formative evaluation was conducted with children who have developmental disabilities ( $N=$ 11), their therapists ( $N=$ 7) and developers ( $N=$ 2). Methods of data collection included questionnaires filled in by therapists and developers, participant observation by researchers and interviews with children.

RESULTS:

The results of the formative evaluation were generally positive regarding four-factor groups that shape the learning experience: Content, Technical characteristics, User state of mind, Characteristics that allow learning. After the design team reviewed the potential users and experts’ comments that were mainly related to the user interface, the application was improved.

CONCLUSIONS:

The two hypotheses have been largely confirmed. Overall, we propose a simplified development process that showcases the importance of arts-based methods and aesthetics which deliver representational fidelity. The study reveals the necessity of developing transmedia learning materials to meet each individual’s needs.

Keywords

Augmented reality game design game development gamification developmental disabilities independent living transmedia learning virtual reality 360-degree interactive videos

1. Introduction

Independent living is a key element and a fundamental indicator of the quality of human life [1]. As life skills are a prerequisite for an independent life, achieving them is vital for both children with special educational needs (SEN) and their typically developing peers [2]. Consequently, child independence is a matter of great concern for parents, caregivers, teachers and therapists [3]. However, children with SEN often have difficulties while performing basic activities of daily living (ADLs) (e.g., dressing, cooking, cleaning and personal hygiene). ADLs enhance independence; therefore, systematic training improves the transfer of functional skills. The most commonly used approaches to special education for ADLs training are in-vivo instruction [4, 5], play-based interventions [6], picture-based systems, such as PECS [7, 8, 9, 10, 11], which was developed in the field of Applied Behaviour Analysis (ABA) [12, 13], Social Stories ${}^{\text{TM}}$ [14], video-based instruction (VBI) [7, 15, 16, 17], computer-based interventions (CBI) [4], computer-based video instruction (CBVI) [18, 19] and teaching through cutting-edge technologies such as virtual reality (VR) and augmented reality (AR) [20, 21]. During ADLs training and practising, children with developmental disabilities face difficulties due to their limited abilities to identify with others, reduced comprehension of symbolic play and pretence, low imagination level, resistance to environmental changes or daily routines and trouble applying what they have learned to real-life [2]. When skills are taught to children with SEN solely through one intervention, it becomes difficult to generalise in real life due to children’s significant limitations in intellectual functioning and adaptive behaviour [3, 22]. This difficulty underscores the value of a differentiated teaching approach that is more likely to meet the learning needs of most students than the traditional “one-size-fits-all” method [23].

Individual differences are not overlooked since researchers prove that, for example, students with attention difficulties seem to perform better by manipulating static pictures than by VBI [24, 25]. On the other hand, students who struggle to pretend may overcome the obstacles resulting from lack or dysfunction in Theory of Mind (ToM) (i.e., cognitive ability to understand others’ mental state, beliefs, emotions, intentions, desires and perspectives) [26] and to be motivated and improve generalization of their skills to everyday settings, using “naturalistic” approaches such as in-vivo instruction [27]. Conversely, identifying a particular activity with a particular place can cause generalization difficulties. These difficulties are related to the difficulty of adopting learning procedures from other real-world environments in real-life situations, due to the differences between real house-place and public space [8]. Compared to the in-vivo instruction, VBI (video modelling and prompting) students can observe target-skills over and over again, and receive instant feedback in a cost-effective way [28]. Besides, in terms of classroom diversity, different types of VBI could be applied where appropriate. Video prompting has the advantage of presenting content in smaller parts, which supports students with SEN to be concentrated on one part rather than an entire task or activity. Video modelling, on the other hand, can decrease the negative effects of stimulus overselectivity which is widespread in many types of disability, especially of individuals with Autism Spectrum Disorders (ASD). Going back to ToM, creating videos with both types of perspective (first and the third point of view), using models (actors/avatars) or self-modelling, could assist individuals who struggle to identify with others. Especially in children with ASD, VBI proves to be a more effective method of intervention than in-vivo instruction for functional skills training [29, 30].

Teaching in a digital age, computer-based interventions (CBIs) deliver instruction through auditory and visual stimuli, and unlike VBI, encourage students to interact with the software using peripherals. Contemporary computers can support all of the above-mentioned forms of training. An intervention that leverages the capabilities of both computers and VBI is called computer-based video instruction (CBVI). CBVI includes methods for ADLs training employing multimedia material such as audio, video, photos, and animation in a real-life environment where students could practise these skills [31, 32]. The latest progress in emerging technologies, which refers to VR, AR, and 360-degree videos, move the virtual worlds from two dimensions (2D) to three (3D), contributing to deeper immersion and opening new avenues in education for SEN students, through multi-sensory experiences, combining vision, hearing and movement [33]. In this context, VR-based and AR-based interventions utilise cutting-edge technology that is applied in many fields such as healthcare and education with remarkably encouraging learning outcomes [34]. Thanks to the unique features of 3D virtual environments, VR systems can adapt pace and sequence to student cognitive needs and competencies for better assimilation of the targeted skill, and therefore are suitable for individuals with learning difficulties and different learning styles [35, 36]. VR is increasingly used in clinical psychology as a therapeutic strategy because of its ability to create real-world experiences and maintain stimulus control [37]. Comparing VBI with VR systems to train functional skills and behaviours in students with SEN, although VBI is proving to be an invaluable instruction method, VR systems seem to be much more effective because they provide representational fidelity and interaction in a safe environment [38, 39]. Representational fidelity refers to the quality of the devices to reproduce the most realistic content possible [40]. Interaction refers mainly to the dynamic embodiment experience through an avatar used to represent the user. Through avatars, users can communicate and control the environment as if they were actually inside the 3D environment simulation [40, 41]. Therefore, representational fidelity combined with interaction creates a specific psychological experience described as presence. AR-based interventions are equally promising in the education of individuals with SEN but unlike VR, which is an artificial environment, AR interfaces allow students to observe the physical and virtual worlds simultaneously in real-time. The added value of AR interventions relies on collaborative learning between classmates and educators by enhancing face-to-face communication, interacting with 3D virtual objects from different viewpoints, sharing insights and preventing isolation [42, 43].

An interesting emerging technology is 360 ${}^{\circ}$ degree videos which provide users with a panoramic view, thus overcoming one of the main limitations of standard videos, that of the point of view [44]. Those videos are characterized by representational fidelity and are preferred over other graphics-based applications. Such videos provide greater immersion, presence and interaction. In recent years, their impact on learning has been studied. The findings are very promising as positive results are reported regarding students engagement, motivation enhancement and enjoyment [45, 46].

This paper presents the participatory design for the development of the transmedia project “Waking up In the Morning” (WUIM), which combines special education pedagogies, filmic methods, game design and innovative technologies, and its formative assessment through user-based and expert-based evaluations by students, therapists and developers.

2. Theoretical and conceptual framework

2.1 Transmedia reality, pedagogy and disability

Cutting-edge technologies have a positive impact on physical, cognitive and socio-emotional children development [47, 48]. At the same time, assistive technology supports people with diverse type of disability, such as sensory disorders, special communication needs, learning difficulties, etc., to improve their daily life and learn in a more personalised way as this technology can be adapted to their capabilities, preferences and needs [49, 50]. Combined VR and AR systems can be introduced into the “education for all” programs. For these emerging technologies, authors propose the term “Transmedia Reality” based on research into the concept of “transmediality” and interactive multimedia technology as a narrative medium. In the age of convergence, individuals gather information from many sources to form a new synthesis. Transmediality allows individuals to create their own interpretation for a story and to have a deeper connection and engagement with characters and plot [51, 52, 53]. In an educational environment, transmediality refers to the use of both digital and non-digital media as well as students’ interaction with narration allowing them to continue, expand and/or change the stories [54, 55, 56, 57, 58]. Student engagement with physical and virtual worlds can be accomplished by reading a book, playing a digital game and engaging in many other digital activities based on the same story [59]. Transmedia learning (TL) is the new paradigm for effective education which allows teachers and students to unfold a basic knowledge or experience across different technology platforms and multiple instruments and emotionally engage by involving them personally in the story [3, 60, 61].

TL encompasses different learning models, pedagogies and transmedia storytelling media such as games and videos, promoting the synergy between Universal Design for Learning (UDL) [62], Differentiated Instruction (DI) [23] and Learning Theories [60, 61, 63]. As each student, with or without SEN, is unique and perceives information in a personalised way, there is no optimal means of presentation/representation, expression, engagement, and assessment that suits all students [64]. According to DI, a variety of means can be utilised to meet students’ needs, desires and preferences. DI is an ideal approach for inclusive environments, providing the guidelines for suitable flexibility in designing content, process, product and learning environment so that training corresponds to students’ level of readiness, interests and learning profile [23]. The 21st-century learning principles which are based on the synthesis of the prominent learning theories (i.e., behaviourism, cognitivism, constructivism and their branches), UDL and DI approach could lead to deeper learning in inclusive educational settings and improve critical thinking, communication, collaboration, and creativity [65]. Roughly speaking, behaviourism points out that learning is the result of instruction while constructivism argues that learning is the result of construction [66]. For behaviourism, learning is achieved through a systematic series of trial-and-error action. The most commonly used applications of behavioural approach in educational technology are the drill-and-practice and tutorial instructional software. The contribution of constructivism is indicated by the dynamic ecosystem of TL, and principles arising from the research field of human-computer interaction [61, 67, 68, 69, 70]. A key concept of Vygotsky’s social constructivist theory is the Zone of Proximal Development (ZPD) [71]. ZPD is determined by the cognitive tools that individuals have at a particular time in their lives, as well as by what they can learn with the minor assistance of significant others. Children move forward through play which creates children’s ZPD [71]. During shared play, when peers help each other they achieve learning goals more effectively [72]. Similarly, Bruner introduced the concept of scaffolding alongside the idea of discovering learning [73]. Recent research concludes that more directed forms of discovery learning are preferable to pure unstructured discovery learning, especially when combined with playful activities [74].

Contemporary educational technology acknowledges the dynamic of digital games as innovative educational tools. Game design incorporates the basic traditional laws and fundamental principles of effective learning [75].

2.2 Learning through games, gamification and simulation

Games attract people’s attention thanks to twelve fundamental elements: fun, play, rules, goals and objectives, adaptivity, outcome and feedback, conflict and competition, problem-solving, interactivity, interaction in a social context, story, and win states [76]. Fun and play engage people in a pleasure activity, while rules define games as organised activities that have specific goals and objectives. Goals and objectives are related to people’s motivations and push them to get involved and play. Adaptivity is associated with the difficulty level and flow (i.e., the state that gives a holistic sense of presence when we are fully involved in any activity) [77]. A game could be boring if it is easy or frustrating if it is difficult. In both cases, players quit the game. Well-designed games keep players in the flow and immediate feedback allows direct information about their performance and progression [78]. Conflicts do not necessarily have to be related to an opponent, but they can be problems that players try to solve, which for example are included in a puzzle [76]. Interactivity refers either to human-computer interaction or social interaction during collaborative learning. The story is the narration that consists of theme, characters, environment and plot, and can derive both from real and virtual worlds. Important features of games are also the “gameplay”, that is, the way a game is played, and the “game mechanics” which are referred to as achievements, avoidance, countdown, rewards program, variable rewards programs, virtual objects, disincentives, and up-to-date information [79]. These features are essential to maintain and enhance players’ motivation. The synergy of technology and pedagogy that utilises digital games into the learning process is called digital game-based learning (DGBL) [80, 81]. Acquisition of knowledge through DGBL is achieved when students follow a predetermined course that leads to the right choice, through a process of trial and error. In this process, the reflection of behaviourism is evident [82].

A less structured playful activity that utilises some of the game elements and features such as points, badges, and leaderboards, is gamification [83]. Gamification is an integral part of any game that does not necessarily involve digital technology [84]. Gamification design is a complex process and results of a transdisciplinary collaboration from psychology, game design, and programming [85]. When game elements are added to learning materials, social interaction is enhanced, performance is improved and students are motivated to engage in a learning activity, that they would not otherwise participate, as it would probably be tedious, challenging or boring [85, 86]. If the goal is to teach a valuable skill that is practised in real-life (e.g., road crossing) rewards based on gamification can be effective. However, rewards should be quickly replaced by more essential elements, such as storytelling, plot, freedom to explore other routes, and opportunities for reflection. This process is known as “meaningful gamification” [87]. Educational activities that incorporate techniques of meaningful gamification and are designed with students in mind, allow teachers and students to connect real-world problems or situations to the school environment. This process gives convincing answers to the “why” of learning [88]. Applications that are directly related to meaningful gamification are simulations.

Table 1
The 5W2H framework of WUIM

Simulations are virtual representations of real-life situations, allowing users to act and experiment virtually in a safe environment without having to risk [89]. During the gaming experience provided by simulations, students are assumed to acquire knowledge, develop skills, or receive treatment [90]. The ultimate goal of simulations is user interaction, whereas the goal of games is to win [91].

While DGBL focuses on structural elements such as rules and goals [82], gamification and simulation are based on the unstructured act of playing. Play is related to the player’s exploration, experiment, and actions within an imaginary environment. This virtual gaming environment is a learning environment since it offers conditions free of any pressure and negative consequences [92]. Mitgutsch [82] proposed the concept of digital play-based learning (DPBL) to describe the learning provided by a circular and non-linear learning and relearning process, combined with the unprompted and unstructured dimension of playing experience.

Overall, the most dynamic element of playful activities, whether it is an entire game, a gamified application, or a simulation is the playing experience per se [86].

2.3 Educational game and simulation design

When teachers plan their lessons, create their own digital educational games and simulations and evaluate ready-made digital materials, they take into account four dimensions: (1) learners (who), (2) content (what), (3) pedagogy (how), and (4) context (when and where) [93, 94]. The content design should ensure players’ engagement. To this end, games should provide attention (curiosity and interest), relevance to student goals, confidence, satisfaction, self-regulation, competence (problem-solving), autonomy (player’s control), relatedness (interaction/connection to character/avatar and/or social interaction), immersion and presence [95, 96]. The necessity of ADLs training is considered an integral part of children’s daily life and represents the “why” of teaching. Ellington [97] was the first to indicate that “why” is the starting point for designing any educational game and simulation. The interplay of all these dimensions forms a compass for user experience design. The widest framework that drives the design, development and evaluation of user-centered applications is the 5W2H method. The 5W2H provides the guidelines for designing educational materials in a form of questions: who, what, why, when, where, how, and how much of learning. Answers to these questions conduct designers to include all the information needed for the final educational interactive product. The 5W2H method is applied to the WUIM project as follows (Table 1).

2.4 User experience design

Designing products, objects or services, even games for humans requires a detailed understanding of their feelings and needs like autonomy, competency, stimulation, relatedness, and popularity during and after their engagement, as well as the context it is designed for [95, 98]. While humans are getting more connected and dependent on technology, user experience (UX) has rapidly become a major concern for both academics and practitioners in Human-Computer Interaction (HCI) field. UX determines the overall quality of user interactions (pragmatic quality) and interactive systems (hedonistic quality) and relates not only to user actions but also to the impact caused by interaction through interfaces from physical, sensual, cognitive, emotional and aesthetic perspectives [78, 98, 99, 100]. Thus, UX and user interface (UI) have a dynamic relationship and are key parameters in any interaction design (IxD) process (Fig. 1) [101].

Figure 1.

User experience design, interaction design and user interface design. Source: adapted from UIUXTrend.com.

Figure 2.

Factors of user experience.

According to Soegaard [102], seven factors influence UX: usefulness, usability, findability, creditability, desirability, accessibility and value (Fig. 2). Usefulness is related to the quality, importance and value of a system or product to serve users’ purposes. Usability and utility are the two components of usefulness. A usable product provides effectiveness, efficiency, engagement, error tolerance and ease of learning. However, even though a product is easy and pleasant to use, it might not be useful to someone if it does not meet his/her needs. An equally important factor shaping UX is the ease of finding a product or system, i.e., findability. Creditability, on the other hand, is related to the characteristics of a product that lead users to trust it, not only because it does the job it promised to do, but also because it can last for a reasonable amount of time, with accurate and appropriate information. Desirability is based on branding, image, identity, aesthetics and emotional design, which are also elements of interface design. Finally, two other aspects of UX are crucial: accessibility and value. Accessibility refers to the experience provided to users, even if they have a disability, such as hearing, vision, movement or learning difficulties. After all, the ultimate goal of a product is the added value it provides to users.

In a user-centred design model, designers focus on users’ needs and their interaction with the product. Thus, IxD as an important component of UX design is all about the interface between users and products encompassing a large range of elements, such as aesthetics, motion, and sound. Five dimensions of IxD are essential: (1) words, e.g., button labels which should be simple and understandable and give as much information as need, (2) visual representations and graphical elements such as images, typography and icons that users interact with, (3) physical objects with which users interact, e.g., laptop with a mouse or touchpad, smartphones and tablets with the user’s fingers, head-mounted devices, etc., (4) time, that mostly refers to media that changes with time e.g., animation, videos, sounds, and (5) behaviour, that is all about how the previous dimensions define the interactions between user and product [102]. UI design focuses on the graphics interface creation aspect for devices [101]. The goal of UI design is to create user interaction as simple and user-friendly as possible, to effectively achieve the objectives of the specific tasks while providing the necessary level of functionality [103].

Especially when products are games, the principal intention of game designers is to provide pleasant experiences to players [78]. UX might be adversely affected by UI design when too much information is presented, resulting in cognitive overload, interruption of player’s flow state, disengagement and frustration if further combined with high difficulty [104]. Even more complicated is digital educational games and simulations design because, in addition to the pleasure offered by game elements and UX design, the instructional design aims at teaching specific content in a balanced system between edutainment and learning. Various materials and platforms such as cards, boards, desktops, laptops, smartphones, tablets, and any other multimedia-enabling systems can deliver gamified content.

3. Aim of the study

In this context, we have designed and developed an educational project called WUIM (by the acronym “Waking Up In the Morning”), taking into account the contribution of new technologies and games to inclusive education, the principles of learning theories widely used in game design, and learning approaches such as TL, UDL and DI. WUIM refers to the activities that students must complete from the time they wake up in the morning with the alarm clock until they open the front door to go to school [2].

WUIM applies TL philosophy through traditional educational materials, such as flashcards and wooden block puzzles, substantial educational interventions, such as in-vivo instruction, play-based interventions, picture-based systems, social stories, video-based instruction, computer-based intervention, and innovative approaches which comprise computer-based video instruction, AR and VR, smart devices (smartphones and tablets) and 360-degree videos. Recognizing the dynamics of games in education, the whole project was elaborated within the context of designing and developing content based on the gaming experience, utilising the principles of game design and gamification techniques.

WUIM extends to three types of applications:

1.
“WUIM-Puzzles” encompasses traditional board games with flashcards and wooden block puzzles which constitute the toys of the whole project.
2.
“WUIM-AR” is an AR game-like application that is combined with WUIM-Puzzles.
3.
“WUIM-VR” is a VR simulation using gamification techniques.

All WUIM applications were designed based on social constructivism and especially the ZPD provided either by the computer or by peers, scaffolding, and discovering-based learning. This work describes the overall design and development of the WUIM project that includes the seven factors of UX, the five dimensions of IxD and UI for each type of game.

Given that gaming experience can enhance the learning experience, each phase of the applications’ design was guided by two key hypotheses of the research project, if: (1) the final products could be educational resources for ADLs training, both for students with SEN and their typically developing peers (mental age), and (2) the common gaming experience could promote collaborative learning, regardless of students’ cognitive profile. Throughout the design and development phases, successive internal testing was performed, from the prototyping stage to beta-version. As participatory design and evaluation are considered critical game development methods [105, 106], to confirm or reject research hypotheses, we proceeded with user-based and expert-based evaluations, both in terms of content and technical functions. WUIM project was evaluated by potential users (children with SEN), content experts (social worker, occupational therapist, speech therapist, health visitor, nurse, special auxiliary staff and physiotherapist assistant), and developers.
4. WUIM design and development

Depending on the type of application, we used the corresponding technology: traditional board games, video modelling (film production) and cutting-edge technologies, i.e., 360-degree interactive videos, AR and VR. Designing and developing the gaming experience for educational purposes is a multifaceted process that combines UX design, IxD, UI design, and learning theories [94, 107]. For this reason, a transdisciplinary team was organized. The team consisted of ten developers and content experts (pedagogical, technological and aesthetic/audiovisual). Content creation was based on the same idea, concept, script, storyboard, and symbols, but followed different paths due to different technologies used, non-digital and digital, according to transmedia storytelling theory [54]. Recognizing that unifying a mixed process of video and simulation game production using 360-degree video shooting, AR and VR technologies could lead to a complicated model, it is proposed a simplified development process that showcases the importance of arts-based methods and aesthetics (Fig. 3).

Figure 3.

Overview development.

The final output of the film production provides the educational content for digital platforms. The AR platform is combined with non-digital objects which constitute the traditional games (WUIM-Puzzles). WUIM-Puzzles activate the AR application (flashcards and wooden blocks operate as the triggers of WUIM-AR). The same pictures/symbols used in WUIM-Puzzles and WUIM-AR are included in the VR version as game buttons.

WUIM-Puzzles and WUIM-AR are based on DGBL. Each application adopts most of the structural elements of game design: rules, objectives, conflicts, narration, outcome and feedback, conflict, problem-solving, interactivity, interaction in a social context, story, and win states. WUIM-VR is based on DPBL of free interaction, non-linear routes, and experimentation. For WUIM-AR and WUIM-VR, the penalty (loss of time) because it could stress students, is an option that depends on the choice of students or teachers. Reinforcement is afforded by a human pedagogical agent, who performs a dual role, as a tutor providing instructions (tutorial) and verbal encouragement to players to continue (reward).

In many phases of WUIM development, work-progressing was parallel. First, it was decided which of the daily morning activities would suit the project. The overall goal of WUIM applications is for students to comprehend, acquire, generalise, and maintain the basic morning activities performed at home. To this end, the scenario is structured in rooms. Each room has specific activities to complete based on the escape room game pedagogy [108]. When designing instruction, it is essential to consider the human cognitive architecture and especially the cognitive load theory [109] and limitations on human’s capacity for processing information [110]. Based on the patterns of the Picture Arrangement subtest of the Wechsler Intelligence Scale for Children-WISC III [111] that is focused on nonverbal temporal sequencing skills, we selected six activities. The selected activities were decided to be depicted by symbols from the Boardmaker collection with permission from Tobii Dynavox Picture Communication Symbols (PCS) ${}^{\@setsize{\scriptsize}{8pt}{\viipt}{\@viipt}{\textregistered}}$ : toilet, handwashing, breakfast, toothbrushing, dressing up, parent hugs. To develop the gameplay, two more scenes were added, namely the alarm clock and walking to school (Fig. 4).

Figure 4.

Activities – symbols from the Boardmaker collection.

PCS provide a multi-level approach to create attractive prints and interactive materials with ready-made items, activities schedules, and game-like applications that can engage students through the interactivity provided by smart boards, computers, tablets, and communication devices. To avoid reproducing gender stereotypes, symbols do not specify gender.

The choice of these concrete ready-made symbols was based on the factors that figure out an effective UX design:

Creditability and desirability: PCS are very popular in the field of special education and recognised by most children with SEN.

Usability: PCS are easily printed and no additional training is needed.

Findability: PCS are easy to find, as the availability of triggers also affects the functionality of AR application.

Accessibility: PCS display simplicity in depicting people and objects. Studies on the effect of the amount of detail on pictorial recognition memory have concluded that the simpler a picture, the lower cognitive load and therefore avoiding problems with concentration and distraction [112]. Besides, according to the instructions of the AR application development engine, pictures simplicity serves their rapid response as AR triggers.

Regarding UI design of WUIM-AR and WUIM-VR, particular attention was paid to visual aspects such as subtitle font size, contrasts, and colours of both letters and background, as well as audio aspects such as sounds and music. Due to individual differences and preferences, a more personalised environment and different options are provided. Players/students choose whether they desire subtitles to appear or sounds to be heard.

4.1 Flashcards and wooden block puzzles

Flashcards and wooden blocks are common symbol-based communication means for both children with or without disabilities, especially for those with ASD [10], and preschoolers as well. Among others, flashcards and wooden geometric shape sorter puzzle boards support hand-eye coordination, fine motor skills, critical thinking, judgment, problem-solving memory, shape recognition, and visual-spatial perception development [113].

Since students with disabilities often experience sensory processing disorders, such as tactile defensiveness [114], morning activities are presented both on flashcards and/or wooden blocks, according to their preference. Flashcards were printed on hard cardboard and then were laminated with a matte texture. Wooden material was designed with CorelDRAW ${}^{\@setsize{\scriptsize}{8pt}{\viipt}{\@viipt}\textregistered}$ software and then cut with CNC (computer numerical control) wood route. After cutting the wooden blocks, the symbols were printed on vinyl stickers and glued to the wooden blocks and boards. Wooden puzzle design was based on differentiated instruction. We created two different gameplay and difficulty levels, an easy one, and a little more difficult. For the easy version, the wooden board and puzzle blocks were made up of different geometrical shapes. Geometric shape sorter puzzle board consists of six removable blocks: pentagon, triangle, square, circle, hexagon, and rhombus. The most difficult version includes only square shapes (Fig. 5). The 1st and 8th symbols are already printed on the boards. Also, for children who experience difficulty with fine motor skills, pushpins were placed as handles, both for the easy and the difficult version, so that wooden blocks are easier to handle.

Figure 5.

Flashcards, wooden blocks and boards.

For an inclusive gaming experience, teachers give each student or student group a wooden shape sorter puzzle board. The difficulty level corresponds to their learning profile. Then students are asked to match each of the six wooden shape blocks with the board’s slot. The wooden blocks are given mixed, but always in the same predefined order. The easy version is a self-guided learning process in which children solve the puzzle also assisted by other features such as the different shapes. When children practice at the most difficult level, they are guided only by the content. Teachers have the opportunity to choose different learning conditions. For example, they can initially give students the easy version, and continue with the difficult one, to monitor how students perform by problem-solving or by chance (i.e., trial and error process). Then students can play again, either with flashcards or wooden blocks, but out of the box (without sorter boards) [115]. This process will help their creative thinking.

4.2 Film production process

Pre-production, production, and post-production constitute the three phases of film production: [116]. The pre-production phase is an early stage of the project before filming begins. In this phase, four tasks were undertaken: (1) script, (developed with Fade In Professional Screenwriting Software ${}^{\copyright}$ https://www.fadeinpro.com), (2) storyboard (created by photo shooting during the actors’ rehearsal, so along with the wording of the technical information, such as objects, lighting, costumes, characters, and sounds, the actors performed their “role”), (3) shoot list, and (4) breakdown sheet. Then, we proceeded with film production, where the actors, a girl and her mother, performed their roles in a real house according to the script with simultaneous 360-degree video-recording (Fig. 6).

Figure 6.

During video-shooting with 360 ${}^{\circ}$ camera.

The concrete actors were chosen for convenience and not for gender stereotypes reproduction. The mother belongs to the transdisciplinary team and the girl is her daughter, so there was no need for consent documents. The mother’s role is practical and helps make the simulation feel more inclusive, interactive, and life-like, as opposed to a completely isolated sandbox. Regarding emotional design, the mother is an ever-present and special figure in a child’s consciousness, symbolizing stability, and security. In future work, more secondary actors may be included pending further analysis of impressions in children players. After video-recordings, we proceed with video and audio editing (i.e., the post-production phase). During the post-production, tasks followed the criteria of UI design. For VR development, it was decided to take advantage of the capabilities of 360-degree videos as they provide a natural representation of the space, greater immersion and interaction than standard videos. However, for AR development, we ought to convert 360-degree videos to standard videos. Using 360-degree video on a 2D screen can confuse and discomposure users spoiling their immersion. Standard videos are pop-up information (overlays). Also, the human pedagogical agent was filmed, and subtitles/text, colour/size and background colour were added using the software Veed Studio ${}^{\copyright}$ (Fig. 7).

Figure 7.

Subtitling (pedagogical agent).

Figure 8.

Vuforia engine developer portal.

In addition to being a valuable aid to people with hearing loss, subtitles are considered a successful vocabulary learning tool (verbal and nonverbal/auditory-visual encodings) [68, 117, 118]. Finally, voiceovers and sound effects were created, and extra sounds were voice-recorded (i.e., pedagogical agent’s reinforcements such as “well-done”, “congratulations”, or assistance to continue the effort, and clock alarm sound effect).

4.3 AR game development process

WUIM-AR is a target-based AR app. AR targets are visual cues that trigger a specific action determined by the program displaying information through a camera application on a smartphone/tablet. To have an overall AR experience the program must detect and respond to targets in real-time. WUIM-AR uses predefined targets (Boardmaker symbols) to trigger the display of overlays (videos with morning activities). For WUIM-AR development, we used the Vuforia AR Engine and Unity 3D Game Engine. Vuforia Engine Developer Portal processed the images and created a database containing the targets. The instructions provided by Vuforia Engine declare that targets must have specific features, such as sharp edges, optimal image dimensions, aspect ratio, high image contrast, no repetitive patterns, format, and distributed textured areas. The software evaluates the quality of the targets. The best quality gets a 5 star rating, but also 3 or 4 star quality is acceptable. Below 3 stars the application may have a hard time recognising a target and at 1 star or 0, the target should be replaced. As already mentioned, WUIM-AR depends on the eight flashcards or wooden blocks of WUIM-Puzzles. Vuforia Engine Developer Portal created a database with all targets (triggers) to be used in WUIM-AR including images for both the game per se and tutorials (Fig. 8).

Figure 9.

Target list.

Figure 10.

Target.

Figure 11.

Unity inspector.

Figure 12.

Code snippets in visual studio.

Figure 13.

Editor of main menu.

Figure 14.

Flowchart of AR game.

After creating the database, we proceeded to develop the gaming experience with the Unity Game Engine. At first, we created a list (Fig. 9) which is a chain of targets. Each target recognizes which one comes before it.

The program also holds, videos or sounds of the pedagogical agent that should be displayed when the student has difficulty finding the next target (according to the gameplay), as well as the target that should have been previously located (e.g., a unique combination is brushing the teeth after eating) (Fig. 10).

Next, we imported data in the inspector window (Fig. 11). When a specific target and/or a unique combination of targets are detected, the corresponding video is displayed.

During the game, the previously scanned target of the chain is stored in memory. When the target is recognised, the game engine checks if the previous target from the one that has just been recognised is the same as the stored one. If so, the recognised target becomes the stored one and the corresponding video and audio are displayed. If not, a short vibration, alone or in combination with a “beep” sound, depending on students’ audio settings and according to how many times the student did not find the appropriate target (Fig. 12).

Game settings such as difficulty level, subtitles, language and graphics are stored in a file to remain the same even after the game is closed (Fig. 13).

The AR flow chart outlines the gameplay, interfaces, and options provided to students (Fig. 14).

For a comprehensive sense of the gaming experience, students should follow a specific sequence by matching the wooden blocks (targets) with the appropriate slots of the wooden boards. After placing the block, they scan using their smartphone or tablet. In case the students choose a different flashcard/wooden block than expected and consequently the application does not operate, the pedagogical agent drives them to try again. Students have the opportunity to try a few times depending on the selected difficulty. If the students are not able to understand the sequence and the appropriate combinations to progress the story, then the pedagogical agent provides “help”. When the sequence is completed a pleasant sound of success is heard and an animated “congratulations” message appears. Then, all the morning routines are displayed with a unified video and at the same time, after congratulating the student, summarizes the whole process.

4.4 VR game development process

As mentioned, WUIM-VR is based on DPBL given that is a gamified simulation. WUIM-VR is played with smartphone Head-Mounted VR Glasses. Students/players are allowed to navigate freely at home, interact and experiment by choosing any route they wish. During this non-linear tour, students learn effortlessly from the results of their choice, even from their deliberate mistakes. This process is a problem-solving process.

A basic aim of the VR implementation is to provide the ability for teachers to set up VR experiences for their students without needing the actual VR equipment to do so. This encourages collaboration, better planning and preparation of the source material and removes a lot of the stress of having to familiarize everyone in a project with VR technology. The VR experience comprises 3 parts: the source material, the configuration and the game engine. It is implemented in a way that separates the roles of directors/filmmakers, video/audio editors, developers and writers (game designers).

The source materials: the source materials are videos, audio clips and a model of the states, transitions between them and variables to be tracked as the game unfolds. The storyboard facilitated filming and recording so that the agent audio clips, audio cues, transition videos and static scene photos were filmed/captured in such a way that no re-shooting needs to be done.

All visual material was created with a 360-degree camera so that it can be imported directly into the game engine, with varying levels of compression to allow for hardware limitations on the output devices.

The configuration (game flow): the complicated part of a VR scenario, especially interactive ones with an open-world slant, is predicting all the possible transitions between states in the game. This is deliberately separated from the design and source material creation and handled by the game engine.

All media and an XML file representing the model of the experience were created first using a standardized dictionary (an XSD is available). This data was fed into the game engine, which supports transition videos in MP4 format (or any encoder available on the target system), static images and stereo audio.

Each scene contains an image or video (“static”) where the user interacts with hotspots, which are graphic icons superimposed on the skybox. This involves looking at a hotspot in VR, which then is activated after a short wait and triggers an optional 3D video transition displayed around the player (Fig. 15).

Figure 15.

A hotspot being activated via a rotating indicator (in this game the actor is visible).

Figure 16.

A sample XML file showing avatar, agent and game scene flow rules and attributes.

Figure 17.

Scene graph of VR game.

The movement inside the VR space is not supported, so scenes are mostly about looking around and finding hotspots rather than interacting with the scene objects themselves. Scenes can also contain conditional narrator audio or feedback for activating hotspots. If a child has difficulty stabilizing the head (e.g., spastic cerebral palsy) then he/she is given the opportunity to use controllers.

The engine also tracks free variables such as attributes of the player, e.g., score, time, clothing on or off, etc. This system is limited to unstructured values so could be “yes”/”no”, a number or a string.

In the configuration file (Fig. 16), the game designer can specify conditions to define and allow transitions based on boolean expressions based on free variables or built-in flags, such as if a scene has been played or which was the previous scene. Besides, the transitions can change the value of free variables, e.g., a player can have different transitions shown if they enter a room (scene) from one entrance to another. Or visiting one scene first may set a flag that changes the scene in a later, unrelated transition.

Example of a game, without free variables shown – there are three scenes, conditions are displayed on connecting arrows (Fig. 17).

The game engine: the game engine is a state transition engine imported into Unity. It is written in C# and loads an XML file of scenes with attributes and transitions between those scenes. It then creates a VR environment based on the properties of each scene, e.g., initial camera position, background audio, narrator cues and hotspots to re-create the dynamic system modelled in the XML file. The engine’s built-in functionality is handling of scenes, transitions and playback of media based on the pseudo-script language contained in the configuration. It is as simple as possible to allow open-ended games to be visualized and experienced in VR. Finally, the game engine integrates with the VR drivers necessary for output (Fig. 18).

Figure 18.

Overview of VR game.

5. Formative evaluation process

Educational games can construct real-life learning activities and improve meaningful learning based on interactions between students and games. The more realistic an educational activity, the easier it is for students to generalise skills to real-life [90]. The methodology for evaluating digital games is directly related to the stage of their design and development. According to Novak [119], the stages refer to the concept, pre-production, prototyping, production, alpha-phase (internal evaluations by the design team), beta-phase (external evaluations by users and experts), and post-production. Depending on the design and development stage of any application, qualitative evaluation methods through observation, as well as quantitative methods through self-reference measurements, interviews (structured or semi-structured), usability tests, and questionnaires aimed at focused groups, are performed [120, 121, 122].

At each stage of application development, the design team conducted repeated internal tests and evaluations based on the System Usability Scale (SUS) [123] and the Serious Game Evaluation Scale (SGES). SGES, which simultaneously evaluates many game elements that shape users’ perceptions when playing games, has been tested for validity and reliability through multiple statistical analysis [122, 124, 125, 126, 127].

During the beta-phase, evaluations were performed by focus groups of children with SEN and/or disability (potential users) and experts. Focus groups of both children and experts are convenience samples. The choice of this type of sampling was based on practical and ethical reasons because there was the possibility of direct access to institutions and children’s parents (Center for Physical Medicine and Rehabilitation of the General Hospital of Florina, Greece, and Center for Creative Employment of Children with Disabilities of the Municipality of Florina). For WUIM implementation and evaluation by children, research approval was obtained by the Research Ethics and Conduct Committee of Ionian University, Greek Ministry of Education, and Scientific Council of the Centers. Alongside approvals, parents signed consent for their children’s participation. Implementation and evaluation of the WUIM project took place in July 2020.

Figure 19.

Individual evaluation (child with cerebral palsy).

Figure 20.

Peer mentoring during collaborative AR gaming.

Figure 21.

Group evaluation during AR gaming.

Figure 22.

Group evaluation during VR gaming.

Children focus group consisted of eleven children ( $N=$ 11): 2 children with moderate intellectual disability (IQ 35–49, mental age from 6 to 9 years) [128], 3 children with severe intellectual disability (IQ 20–34, mental age from 3 to 6 years), 3 children with cerebral palsy (1 with severe intellectual disability), 1 child with Down syndrome with severe intellectual disability, 1 child with ASD with severe intellectual disability, tactile defensiveness and hyperactivity, and 1 child with ASD without intellectual disability. The applications were evaluated both by each child individually (Fig. 19) and during collaborative gaming (Figs 20–22).

Table 2

Result of AR user-based evaluation

Table 3

Result of VR user-based evaluation

Focus group with experts on educational content consisted of 7 ( $N=$ 7) specialised therapists for children: social worker, occupational therapist, speech therapist, health visitor, nurse, special auxiliary staff, and physiotherapist assistant. Focus group with informatics experts consisted of 2 developers specialising in game design and development. Children’s evaluations were based on observations and structured interviews using the SGES questionnaire. Therapists filled in the SGES, while programmers filled in the SUS.

5.1 Evaluation results

After the exploratory and confirmatory factor analysis of SGES, eleven factors were formed, which are divided into four groups [124]:

1.
Content: subjective adequacy of feedback, subjective adequacy of educational material, subjective clarity of learning objectives, subjective quality of the narration.
2.
Technical characteristics: subjective usability/playability (functionality), subjective audiovisual experience/aesthetics, subjective realism.
3.
User state of mind: presence/immersion, pleasure.
4.
Characteristics that allow learning: subjective relevance to personal interests, motivation.

Given that samples of both children and therapists were small, we proceeded with trend analysis through a five-point Likert scale “Strongly Disagree” (1) to “Strongly Agree” (5). Out of the 11 children, only 5 children, 3 without intellectual disability and 2 with moderate intellectual disability, were able to evaluate the applications during structured interviews. The results of the quantitative analysis for the user-based evaluation of WUIM-AR are shown in Table 2 and of WUIM-VR in Table 3.

Table 4
Result of AR expert-based evaluation

Table 5
Result of VR expert-based evaluation

Regarding WUIM-Puzzles, the results are integrated into WUIM-AR, as they were part of the AR application. Besides, the feedbacks of both children and their therapists while playing with WUIM-Puzzles were very positive without providing specific recommendations that would lead to their improvement.

During interviews, researchers also recorded the following indicative children’s comments:

Irini (moderate intellectual disability and mental age of 7 years) made the following statements:

•
“I tried to concentrate and not listen to others”.
•
“I did not catch a real toothbrush but I caught the wooden pieces”.
•
“It was assistive that I caught the objects with my hands”.
•
“I liked the tablet; I hold it for the first time in my life and I really liked it”.
•
“I wanted to hear the instructions from Evaggelia” (Evaggelia is the pedagogical agent).
•
“The mask was exciting”.

Vasso (moderate intellectual disability and mental age of 9 years) said:

“I liked the tablet, I played for the first time and it was easy”. “At first, I was afraid of glasses, but it was OK”.

Panayiotis with spastic cerebral palsy, difficulty with fine motor skills, and without intellectual disability stated that due to many people in the room, he felt anxious and therefore he could not hold the tablet well. However, he didn’t feel anxious with Head-Mounted VR Glasses. Finally, Alexis, also with cerebral palsy, difficulty with severe fine motor skills, and without intellectual disability, stated that he really likes technology but he would like some help to hold the tablet. Both Panayiotis and Alexis were fascinated with the VR app.

The following results were obtained after processing the therapists’ questionnaires Table 4 shows therapists’ scores for the AR app and Table 5 for VR app.

An open-ended question for general comment was added to the therapists’ questionnaire. Below their comments are listed:

•
“A very good process for teaching and getting acquainted with technology” (Social Worker).
•
“I liked the structure, the materials and the way the message is passed to the child” (Speech Therapist).
•
“On positives: the detailed audiovisual presentation was excellent" and “On negatives: inaccessible equipment to fine motor skills and vision problems” (Health Visitor).
•
“It is an interesting way for children to learn daily activities, as it combines audiovisual media” (Nurse).
•
“Child’s participation as evidenced by the result” (Special Auxiliary Staff, assistant of a child with ASD, severe intellectual disability, tactile defensiveness, and hyperactivity).

Both user-based (children) and expert-based (therapists) evaluations produced positive results (above 4, Likert scale) with very slight variation. Regarding the factor groups, WUIM-VR seemed to present a greater effect on the state of mind (presence/immersion, pleasure) than WUIM-R. In general, both applications seem to have an equal effect on the learning dimension.

Children’s and therapists’ comments, developers’ notes about technical issues, and researchers’ naturalistic observations led to application improvement. Concerning WUIM-AR, it was immediately recognised by both researchers and children with difficulty related to fine motor skills that children could not stabilise the tablets over the triggers. For this reason, it was decided to provide special tablet cases (Fig. 23).

Figure 23.
Special tablet cases.

Also, a therapist noted that specific care should be to colours and contrasts in subtitles for users with vision difficulties. Initially, subtitles were drawn in white letters on black background. After this recommendation, the option of black letters on a yellow background was added (black and/or yellow according to visual acuity for hard of hearing students).
6. Limitations and future research

The initial research design regarded implementation in inclusive classrooms of formal education. However, due to the SARS-CoV-2 pandemic, access to schools was prohibited. Thus, the conditions for inclusive education were formed in settings supervised by the Ministry of Health. The inclusiveness was ensured by the participation of children of different types of disability and cognitive profile. Another limitation was the relatively small number of children who participated. Future research focuses on evaluating implementation in inclusive classrooms, to confirm other aspects of the hypotheses: (1) if the applications could be an educational material for typically developing students and (2) if the acquired knowledge and skills can be maintained and generalised in the real-world. Another direction of future work is related to the basic aspect of transmediality in which a transmedia story is never-ending, it is continuously reshaped expands or changes [129]. For example, a complex activity such as the process of waking up (Fig. 24) or getting dressed (Fig. 25), can be divided into smaller and simpler parts.

Figure 24.

Waking-up.

Figure 25.

Getting dressed.

This aspect of transmedia learning is consistent with the task analysis method and chaining, which are basic training strategies for daily living activities within the ABA approach [130].

7. Conclusions

Training students with SEN presents a particular interest as although students share characteristics according to their diagnosis, they need personalised learning approaches. Personalised learning is a state-of-the-art approach to an inclusive, student-centered learning environment, that prevents isolation and promotes collaborative learning while requiring the adaptation of teaching methods. These methods are designed in the framework of transmedia learning. The synthesis of learning theories, differentiated instruction, and universal design for learning combined with cutting-edge technologies contribute to the creation of smart learning environments. An excellent dimension of smart learning environments is playful learning. Playful learning provides teachers and students with a more realistic interactive environment and promises a better learning experience, using either entire games or gamified applications. Designing effective educational games and simulations takes into account the balance between education and game. Compared to complex and expensive games, gamification design enables teachers without special technical knowledge to offer students a gaming experience and an opportunity for learning by doing. Feedback is one of the most significant elements of student interaction with games, while immediate emotional feedback by a pedagogical agent has a significant enhancement in learning motivation, problem-solving, and collaborative learning.

In this paper, the authors presented the design and development of the educational application “Waking Up In the Morning”. Conceding that integrating a mixed game and simulation development process can drive to a complex design model that complicates rather than explains the process, we proposed a simplified model indicating the significance of arts-based methods, aesthetics, and cutting-edge technology. Three types of games have been developed. A traditional game that uses flashcards and wooden puzzles (WUIM-Puzzles), a game that uses AR (WUIM-AR), and a gamified VR simulation (WUIM-VR). The 5W2H framework was the compass for this design. From the formative evaluation, we concluded that children understood the content, i.e., the activities performed at home in the morning before a student arrived at school, and the playful process through technology. The two hypotheses of the research, i.e., (1) whether the final products could be an educational material for students with SEN in the context of ADLS training and (2) whether the common gaming experience could promote collaborative learning, regardless of students’ cognitive profile, are largely confirmed. At the same time, potential users’ and experts’ comments in conjunction with researchers’ observations were incorporated in the next phase to improve the applications.

Author contributions

CONCEPTION: Polyxeni Kaimara and Ioannis Deliyannis

SYSTEM DESIGN AND DEVELOPMENT: Polyxeni Kaimara, Ioannis Deliyannis and George Miliotis

LITERATURE RESEARCH: Polyxeni Kaimara, Andreas Oikonomou and Ioannis Deliyannis

MULTIMEDIA CONTENT DESIGN AND DELIVERY: Andreas Oikonomou, Ioannis Deliyannis, Emmanuel Fokides, Agnes Papadopoulou and Andreas Floros

SUPERVISION: Ioannis Deliyannis

Ethical considerations

According to the Declaration of Helsinki (World Medical Association Declaration of Helsinki, 2013), the project was approved by the Research Ethics and Conduct Committee of Ionian University (Ref. No. 4862/08-02-19) and the Greek Ministry of Education (Ref. No $\Phi$ 15/51675/58994/ $\Delta$ 1/15-04-2019), Scientific Council of the General Hospital of Florina (Ref. No 15/04-06-2020 ), Center for Creative Employment of Children with Disabilities of the Municipality of Florina (Ref. No 19/2019) and Association of Parents and Guardians of Children with Disabilities, “Sundberg” (Ref. No 20/22-06-2020). All parents signed informed consent for their children’s participation.

Footnotes

Acknowledgments

The following students of the Department of Audio and Visual Arts, Ionian University have contributed substantially to the WUIM project:

Stavros Karakoutis: AR developer, user experience design and programming

Evaggelia Koumantsioti: storyboard, photography and pedagogical agent

Aris Melachrinos: video and image editing

Evaggelos Pandis: recordings, sound and video editing

Marinos Pavlidis: director, video shooting and editing

and little Paraskevi Rizou: actress/avatar

We would also like to thank the children, their parents and therapists from the Center of Physical Medicine and Rehabilitation (General Hospital of Florina-Greece), Center for Creative Employment of Children with Disabilities (Municipality of Florina) and Association of Parents and Guardians of Children with Disabilities, “Sundberg” (Regional Unit of Florina).

Conflict of interest

The authors have no conflict of interest to report.

References

Kartasidou

Dimitriadou

Pavlidou

, et al. Independent living and interpersonal relations of individuals with intellectual disability: the perspective of support staff in greece. Int J Learn Divers Identities 2013; 19: 59-73.

Kaimara

Miliotis

Deliyannis

, et al. Waking-up in the morning: a gamified simulation in the context of learning activities of daily living. Technol Disabil 2019; 31: 195-198.

Kaimara

Deliyannis

Oikonomou

, et al. An innovative transmedia-based game development method for inclusive education. Digit Cult Educ 2021; (in press).

Ramdoss

Lang

Fragale

, et al. Use of computer-based interventions to promote daily living skills in individuals with intellectual disabilities: a systematic review. Journal of Developmental and Physical Disabilities. Epub ahead of print 2012. doi: 10.1007/s10882-011-9259-8.

Morrow

Bates

. The effectiveness of three sets of school-based instructional materials and community training on the acquisition and generalization of community laundry skills by students with severe handicaps. Res Dev Disabil 1987; 8: 113-136.

Charlop

Lang

Rispoli

. More Than Just Fun and Games: Definition, Development, and Intervention for Children’s Play and Social Skills. 2018. Epub ahead of print 2018. doi: 10.1007/978-3-319-72500-0_1.

Van Laarhoven

Kraus

Karpman

, et al. A comparison of picture and video prompts to teach daily living skills to individuals with autism. Focus Autism Other Dev Disabl 2010; 25: 195-208.

Kuo

Chang

Lyu

, et al. A pilot study of the situated game for autistic children learning activities of daily living. Res Pract Technol Enhanc Learn 2013; 8: 291-315.

Pierce

Schreibman

. Teaching daily living skills to children with autism in unsupervised settings through pictorial self-management. J Appl Behav Anal 1994; 27: 471-481.

10.

Bondy

Frost

. The picture exchange communication system. Focus Autistic Behav 1994; 9: 1-19.

11.

Mechling

. High tech cooking: a literature review of evolving technologies for teaching a functional skill. Educ Train Dev Disabil 2008; 43: 474-485.

12.

Matson

. Applied Behavior Analysis for Children with Autism Spectrum Disorders. New York, NY: Springer New York. Epub ahead of print 2009. doi: 10.1007/978-1-4419-0088-3.

13.

Bondy

. The unusual suspects: myths and misconceptions associated with PECS. Psychol Rec 2012; 62: 789-816.

14.

Gray

Garand

. Social stories: improving responses of students with autism with accurate social information. Focus Autistic Behav 1993; 8: 1-10.

15.

Charlop

Lang

Rispoli

. Lights, Camera, Action! Teaching Play and Social Skills to Children with Autism Spectrum Disorder Through Video Modeling. 2018; 71-94.

16.

Park

Bouck

Duenas

. The Effect of Video Modeling and Video Prompting Interventions on Individuals With Intellectual Disability: A Systematic Literature Review. J Spec Educ Technol. Epub ahead of print 2019. doi: 10.1177/0162643418780464.

17.

Shepley

Spriggs

Samudre

, et al. Increasing Daily Living Independence Using Video Activity Schedules in Middle Increasing Daily Living Independence Using Video Activity Schedules in Middle School Students With Intellectual Disability. Epub ahead of print 2017. doi: 10.1177/0162643417732294.

18.

Ayres

Cihak

. Computer- and video-based instruction of food-preparation skills: Acquisition, generalization, and maintenance. Intellect Dev Disabil. Epub ahead of print 2010. doi: 10.1352/1944-7558-48.3.195.

19.

Goo

Maurer

Wehmeyer

. Systematic review of using portable smart devices to teach functional skills to students with intellectual disability. Educ Train Autism Dev Disabil 2019; 54: 57-68.

20.

Newbutt

Sung

Kuo

, et al. The acceptance, challenges, and future applications of wearable technology and virtual reality to support people with autism spectrum disorders. In: Brooks

Brahnam

Kapralos

, et al. (eds) Recent Advances in Technologies for Inclusive Well-Being, Intelligent Systems Reference Library 119. Springer International Publishing, 2017, pp. 221-241.

21.

Kidd

Crompton

. Augmented learning with augmented reality. In: Churchill

, et al. (ed) Mobile Learning Design, Lecture Notes in Educational Technology. Singapore: Springer, 2016, pp. 97-108.

22.

American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders (DSM-5) – Fifth Edition. Arlington, VA: American Psychiatric Association. Epub ahead of print 22; May 2013. doi: 10.1176/appi.books.9780890425596.

23.

Tomlinson

Brighton

Hertberg

, et al. Differentiating instruction in response to student readiness, interest, and learning profile in academically diverse classrooms: a review of literature. J Educ Gift 2003; 27: 119-145.

24.

Alberto

Cihak

Gama

. Use of static picture prompts versus video modeling during simulation instruction. Res Dev Disabil 2005; 26: 327-339.

25.

Cihak

Alberto

Taber-Doughty

, et al. A comparison of static picture prompting and video prompting simulation strategies using group instructional procedures. Focus Autism Other Dev Disabl 2006; 21: 89-99.

26.

Wellman

. Developing a Theory of Mind. In: Goswami

, (ed) The Wiley-Blackwell handbook of childhood cognitive development. Wiley-Blackwell, 2011, pp. 258-284.

27.

Owens

Granader

Humphrey

, et al. LEGO

{}^{\@setsize{\tiny}{6pt}{\vpt}{\@vpt}\textregistered}

therapy and the social use of language programme: An evaluation of two social skills interventions for children with high functioning autism and Asperger syndrome. J Autism Dev Disord. Epub ahead of print 2008. doi: 10.1007/s10803-008-0590-6.

28.

Corbett

Abdullah

. Video modeling: why does it work for children with autism? J Early Intensive Behav Interv 2005; 2: 2-8.

29.

Charlop-Christy

Freeman

. A comparison of video modeling with in vivo modeling for teaching children with autism. J Autism Dev Disord 2000; 30: 537-552.

30.

Rehfeldt

Dahman

Young

, et al. Teaching a simple meal preparation skill to adults with moderate and severe mental retardation using video modeling. Behav Interv 2003; 18: 209-218.

31.

Goo

Therrien

Hua

. Effects of computer-based video instruction on the acquisition and generalization of grocery purchasing skills for students with intellectual disability. Educ Train Autism Dev Disabil 2016; 51: 150-161.

32.

Papadopoulou

. Art, technology, education: synergy of modes, means, tools of communication. Educ Sci 2019; 9: 237.

33.

Bates

. Teaching in a Digital Age – Second Edition. Q Rev Distance Educ, https//pressbooks.bccampus.ca/teachinginadigitalagev2/ (2019).

34.

Trondsen

. Augmented/Virtual Reality in Nordic/Baltic Education, Learning and Training. Silicon Vikings, https//www.innokas.fi/wp-content/uploads/2019/04/Nordic-Baltic-XR-Edu-Report_3-2019.pdf (2019).

35.

Lee

Wong

Fung

. How does desktop virtual reality enhance learning outcomes? A structural equation modeling approach. Comput Educ 2010; 55: 1424-1442.

36.

Lee

Wong

Fung

. Learning with Virtual Reality: Its Effects on Students with Different Learning Styles. In: Pan

Cheok

Müller

, et al. (eds) Transactions on Edutainment IV. Lecture Notes in Computer Science. Springer, Berlin, Heidelberg, pp. 79-90.

37.

Rizzo

Koenig

. Is clinical virtual reality ready for primetime? Neuropsychology 2017; 31: 877-899.

38.

Fokides

Chronopoulou

M-I

Kaimara

. Comparing videos and a 3D virtual environment for teaching school-related functional skills and behaviors to students with ADHD or developmental dyslexia, displaying challenging behaviors: a case study. Res Pract Technol Enhanc Learn 2019; 14: 22.

39.

Dalgarno

Lee

MJW

. What are the learning affordances of 3-D virtual environments? Br J Educ Technol 2010; 41: 10-32.

40.

Fowler

. Virtual reality and learning: Where is the pedagogy? Br J Educ Technol. Epub ahead of print 2015. doi: 10.1111/bjet.12135.

41.

Gee

. Learning and games. In: Salen

, (ed) The Ecology of Games: Connecting Youth, Games, and Learning. The John D. and Catherine T. MacArthur Foundation Series on Digital Media and Learning. Cambridge, MA: The MIT Press, 2008, pp. 21-40.

42.

Hsu

. Mobile augmented-reality artifact creation as a component of mobile computer-supported collaborative learning. Internet High Educ. Epub ahead of print 2015. doi: 10.1016/j.iheduc.2015.04.003.

43.

Henrysson

Ollila

Billinghurst

. Mobile phone based augmented reality. In: Haller

Billinghurst

Thomas

(eds) Emerging technologies of augmented reality: Interfaces and design. IGI Global, 2007, pp. 90-109.

44.

Fokides

Kefallinou

. Examining the impact of spherical videos in teaching endangered species/environmental education to primary school students. J Inf Technol Educ Res 2020; 19: 427-450.

45.

Argyriou

Economou

Bouki

. Design methodology for 360∘ immersive video applications: the case study of a cultural heritage virtual tour. Pers Ubiquitous Comput 2020; 24: 843-859.

46.

Baumgartner

. The Impact of Virtual Reality and 360-Degree Video on Spatial Reasoning Skills in Elementary Students. OhioLINK Electronic Theses and Dissertations Center, 2020.

47.

Sobel

. Immersive Media and Child Development: Synthesis of a Cross-Sectoral Meeting on Virtual, Augmented, and Mixed Reality and Young Children. New York, NY: Joan Ganz Cooney Center at Sesame Workshop, 2019.

48.

Kaimara

Oikonomou

Deliyannis

. A review of ethical issues and concerns regarding virtual reality technologies and gaming impact on children. 2021; Manuscript submitted for publication.

49.

Lancioni

Singh

(eds). Assistive Technologies for People with Diverse Abilities. New York, NY: Springer New York. Epub ahead of print 2014. doi: 10.1007/978-1-4899-8029-8.

50.

Mechling

. Assistive technology as a self-management tool for prompting students with intellectual disabilities to initiate and complete daily tasks: a literature review. Educ Train Dev Disabil 2007; 42: 252-269.

51.

Gambarato

Dabagian

. Transmedia dynamics in education: the case of robot heart stories. EMI Educ Media Int 2016; 53: 229-243.

52.

Freeman

Gambarato

. Introduction: Transmedia Studies – Where Now? In: Freeman

Gambarato

, (eds) The Routledge Companion to Transmedia Studies. Routledge, pp. 1-12.

53.

Jenkins

.An occasional paper on digital media and learning. Confronting the Challenges of Participatory Culture: Media Education for the 21st; Century. Chicago, IL: The John D. and Catherine T. MacArthur Foundation, 2006.

54.

Pratten

. Getting started with transmedia storytelling. 2011. Epub ahead of print 2011. doi: 10.4135/9781452226552.n11.

55.

Pence

. Teaching with transmedia. J Educ Technol Syst 2011; 40: 131-140.

56.

Alper

Herr-Stephenson

. Transmedia play: literacy across media. J Media Lit Educ 2013; 5: 366-369.

57.

Kaimara

Deliyannis

Oikonomou

, et al. Transmedia storytelling meets Special Educational Needs students: a case of Daily Living Skills Training. In: 4th International Conference on Creative Writing Conference. 2020.

58.

Fiorelli

. Transmedia storytelling: building worlds for and with fans, https://mozcom/blog/transmedia-storytelling-building-worlds-for-and-with-fans (2013).

59.

Sun

. Transmedia and Education: How Transmedia Is Changing the Way We Learn, http://wwwthedigitalshift.com/2014/06/featured/many-ways-tell-story-transmedia-transforming-education-classrooms/ (2014).

60.

Raybourn

. A new paradigm for serious games: transmedia learning for more effective training and education. J Comput Sci 2014; 5: 471-481.

61.

Fleming

. Expanding learning opportunities with transmedia practices: inanimate alice as an exemplar. J Media Lit Educ 2013; 5: 370-377.

62.

CAST. Universal Design for Learning Guidelines version 2.2, http://udlguidelines.cast.org (2018).

63.

Herr-Stephenson

Alper

Reilly

, et al. T is for transmedia: Learning through transmedia play. Los Angeles and New York: USC Annenberg Innovation Lab and The Joan Ganz Cooney Center at Sesame Workshop. https//joanganzcooneycenter.org/wp-content/uploads/2013/03/t_is_for_transmedia.pdf (2013).

64.

Glass

Meyer

Rose

. Universal design for learning and the arts. Harv Educ Rev 2013; 83: 98-119.

65.

Kaimara

Poulimenou

S-M

Deliyannis

. Epistemological Approaches to Digital Learning in Educational Contexts. Routledge. Epub ahead of print 17; April 2020. doi: 10.4324/9780429319501.

66.

Reeves

. A research agenda for interactive learning in the new millennium. In: Proceedings of ED-MEDIA. 1999, pp. 15-20.

67.

Dalton

. Universal design for learning: guiding principles to reduce barriers to digital & media literacy competence. J Media Lit Educ 2017; 9: 17-29.

68.

Moreno

Mayer

. Interactive multimodal learning environments. Educ Psychol Rev 2007; 19: 309-326.

69.

Oviatt

. Advances in robust multimodal interface design. IEEE Comput Graph Appl 2003; 23: 62-68.

70.

Jonassen

. Technology as cognitive tools: Learners as designers. ITForum Pap.

71.

Vygotsky

. Mind in Society: The Development of Higher Psychological Processes. 2nd ed. Cambridge: Harvard University Press, 1978.

72.

van Oers

. Learning resources in the context of play. Promoting effective learning in early childhood. Eur Early Child Educ Res J 2003; 11: 7-26.

73.

Bruner

Watson

. Child’s talk: Learning to use language. New York: Oxford University Press, 1983.

74.

Lillard

. Playful learning and Montessori education. Am J Play 2013; 5: 157-186.

75.

Murphy

. Why Games Work and the Science of Learning. In: MODSIM World 2011 Conference and Expo ‘Overcoming Critical Global Challenges with Modeling & Simulation’. Virginia Beach, VA. Novak: NASA Conference Publication, pp. 383-392.

76.

Prensky

. Fun, Play and Games? What Makes Games Engaging. In: Digital Game-Based Learning. St Paul, Minnesota: St Paul, Minnesota: Paragon House, 2007, pp. 105-144.

77.

Csikszentmihalyi

. Flow and the Foundations of Positive Psychology. Dordrecht: Springer Netherlands. Epub ahead of print 2014. doi: 10.1007/978-94-017-9088-8.

78.

Kiili

de Freitas

Arnab

, et al. The design principles for flow experience in educational games. Procedia Comput Sci 2012; 15: 78-91.

79.

Fabricatore

. Gameplay and Game Mechanics: A Key to Quality in Videogames. OECD, http//www.oecd.org/education/ceri/39414829.pdf (1990).

80.

Prensky

. Digital game-based learning. St Paul, Minnesota: Paragon House, 2007.

81.

Van Eck

. Digital game-based learning: still restless after all these years. Educ Rev 2015; 50: 13-28.

82.

Mitgutsch

. Digital play-based learning: a philosophical-pedagogical perspective on learning and playing in computer games. Hum IT J Inf Technol Stud as a Hum Sci 2008; 9: 18-36.

83.

Deterding

Dixon

Khaled

, et al. From game design elements to gamefulness. In: Proceedings of the 15th International Academic MindTrek Conference: Envisioning Future Media Environments. ACM Press, pp. 9-15.

84.

Deliyannis

Kaimara

. Developing Smart Learning Environments Using Gamification Techniques and Video Game Technologies. In: Daniela

, (ed) Didactics of Smart Pedagogy. Cham: Springer International Publishing, pp. 285-307.

85.

Morschheuser

Hassan

Werder

, et al. How to design gamification? A method for engineering gamified software. Inf Softw Technol 2018; 95: 219-237.

86.

Plass

Homer

Kinzer

. Foundations of game-based learning. Educ Psychol 2015; 50: 258-283.

87.

Nicholson

. A User-Centered Theoretical Framework for Meaningful Gamification. In: Games+Learning+Society 8.0. Madison, WI, http//scottnicholson.com/pubs/meaningfulframework.pdf (2012).

88.

Annetta

. The “I’s” have it: a framework for serious educational game design. Rev Gen Psychol 2010; 14: 105-113.

89.

Marczewski

. Game Thinking. Even Ninja Monkeys Like to Play: Gamification, Game Thinking and Motivational Design. In: Gamified UK. CreateSpace Independent Publishing Platform, 2015.

90.

Zeng

Parks

Shang

. To learn scientifically, effectively, and enjoyably: a review of educational games. Hum Behav Emerg Technol 2020; 2: 186-195.

91.

Gredler

. Games and simulations and their relationships to learning. In: Jonassen

, (ed) Handbook of research on educational communications and technology. Mahwah, NJ: Lawrence Erlbaum Associates, pp. 571-581

92.

Fabricatore

. Learning and Videogames: an Unexploited Synergy. In: 2000 Annual Convention of the Association for Educational Communications and Technology (AECT). Workshop: In Search of the Meaning of Learning. http//eprints.hud.ac.uk/id/eprint/28000/ (2000).

93.

Sims

. An interactive conundrum: constructs of interactivity and learning theory. Australas J Educ Technol 2000; 16: 45-57.

94.

de Freitas

Oliver

. How can exploratory learning with games and simulations within the curriculum be most effectively evaluated? Comput Educ 2006; 46: 249-264.

95.

Ryan

Rigby

Przybylski

. The motivational pull of video games: a self-determination theory approach. Motiv Emot 2006; 30: 344-360.

96.

Keller

. Motivation, learning, and technology: applying the ARCS-V motivation model. Particip Educ Res 2016; 3: 1-15.

97.

Ellington

. How To Design Educational Games and Simulations.Teaching. and Learning in Higher Education. Aberdeen, https//files.eric.ed.gov/fulltext/ED289502.pdf (1987).

98.

Hassenzahl

. User experience (UX): Towards an experiential perspective on product quality. In: Proceedings of the 20th International Conference of the Association Francophone d’ Interaction Homme-Machine on – IHM ’08. New York, New York, USA: ACM Press, p. 11.

99.

Lallemand

Koenig

Gronier

. How relevant is an expert evaluation of user experience based on a psychological needs-driven approach? In: Proc Nord 2014 8th Nord Conf Human-Computer Interact Fun, Fast, Found, 2014, pp. 11-20.

100.

Pereira de Aguiar

Winn

Cezarotto

, et al. Educational Digital Games: A Theoretical Framework About Design Models, Learning Theories and User Experience. 165-184.

101.

Will

. UI Design vs UX Design vs Interaction Design, https://uiuxtrendcom/ui-design-vs-ux-design-vs-interaction-design2/.

102.

Soegaard

. The Basic of User Experience Design. Interaction Design Foundation, 2018.

103.

Lei

. Design Entropy: A New Approach for Evaluating User Experience in User Interface Design. In: Rebelo

Soares

, (eds) AHFE 2016 – International Conference on Ergonomics in Design. Springer International Publishing Switzerland, pp. 583-593.

104.

Thorpe

Nesbitt

Eidels

. Assessing Game Interface Workload and Usability. In: Proceedings of the Australasian Computer Science Week Multiconference on – ACSW 2019. New York, New York, USA: ACM Press, pp. 1-8.

105.

Ramadan

Widyani

. Game development life cycle guidelines. In: 2013 International Conference on Advanced Computer Science and Information Systems, ICACSIS 2013. 2013. Epub ahead of print 2013. doi: 10.1109/ICACSIS.2013.6761558.

106.

Steinböck

Luckner

Kayali

, et al. Cologon. In: Proceedings of the 14th International Conference on the Foundations of Digital Games – FDG ’19. New York, New York, USA: ACM Press, pp. 1-5.

107.

Tobias

Fletcher

Wind

. Game-based learning. In: Handbook of Research on Educational Communications and Technology: Fourth Edition. 2014. Epub ahead of print 2014. doi: 10.1007/978-1-4614-3185-5_38.

108.

Fotaris

Mastoras

. Escape Rooms for Learning: A Systematic Review. In: Proceedings of the 12th European Conference on Game Based Learning. ACPI, p. 30.

109.

Sweller

. Cognitive Load Theory. In: Mestre

, Ross BHBT-P of L and M, (eds) Psychology of Learning and Motivation. Academic Press, pp. 37-76.

110.

Miller

. The magical number seven, plus or minus two: some limits on our capacity for processing information. Psychol Rev 1994; 101: 343-352.

111.

Georgas

Paraskevopoulos

Besevegis

, et al. The Hellenic WISC-III. Athens: Psychometric Laboratory, University of Athens & Ellinika Grammata, 1997.

112.

Pezdek

Chen

. Developmental differences in the role of detail in picture recognition memory. J Exp Child Psychol. Epub ahead of print 1982. doi: 10.1016/0022-0965(82)90016-9.

113.

Bhatia

Davis

Shamas-Brandt

. Educational gymnastics: the effectiveness of montessori practical life activities in developing fine motor skills in kindergartners. Early Educ Dev 2015; 26: 594-607.

114.

Ahn

Miller

Milberger

, et al. Prevalence of parents’ perceptions of sensory processing disorders among kindergarten children. Am J Occup Ther 2004; 58: 287-293.

115.

Adair

. The art of creative thinking: how to develop your powers of innovation and creativity. London and Philadelphia: Kogan Page, 2007.

116.

Panagopoulos

. Reshaping Contemporary Greek Cinema Through a Re-evaluation of the Historical and Political Perspective of Theo Angelopoulos’s Work. University of Central Lancashire, 2019.

117.

Kanellopoulou, Kermanidis, Giannakoulopoulos. The dual-coding and multimedia learning theories: film subtitles as a vocabulary teaching tool. Educ Sci 2019; 9: 210.

118.

Sadoski

. A dual coding view of vocabulary learning. Read Writ Q 2005; 21: 221-238.

119.

Bernhaupt

. User Experience Evaluation in Entertainment. In: Evaluating User Experience in Games: Concepts and Methods. Human-Computer Interaction Series. Springer, London, pp. 3-7.

120.

Almeida

Abreu

Silva

, et al. Applications and Usability of Interactive Television. 2018; 813: 44-57.

121.

Bernhaupt

, Mueller F ‘Floyd’. Game User Experience Evaluation. In: Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems – CHI EA ’16. New York, New York, USA: ACM Press, pp. 940-943.

122.

Kaimara

Fokides

Oikonomou

, et al. Evaluating 2D and 3D serious games: the significance of student-player characteristics. Dialogoi! Theory Prax Educ 2019; 5: 36-56.

123.

Brooke

. SUS: A ‘quick and dirty’ usability scale. In: Jordan

Thomas

Weerdmeester

, et al. (eds) Usability evaluation in industry. London: Taylor & Francis, 1996, pp. 189-194.

124.

Fokides

Atsikpasi

Kaimara

, et al. Let players evaluate serious games. Design and validation of the Serious Games Evaluation Scale. Int Comput Games Assoc – ICGA. Epub ahead of print 2019. doi: 10.3233/ICG-190111.

125.

Fokides

Atsikpasi

Kaimara

, et al. Factors influencing the subjective learning effectiveness of serious games. J Inf Technol Educ Res 2019; 18: 437-466.

126.

Kaimara

Fokides

Plerou

, et al. Serious games effect analysis on player’s characteristics. Int J Smart Educ Urban Soc 2020; 11: 75-91.

127.

Fokides

Kaimara

Deliyannis

, et al. Development of a scale for measuring the learning experience in serious games. In: Panagopoulos

Papadopoulou

Giannakoulopoulos

, (eds) 1st International Conference Digital Culture and AudioVisual Challenges, Interdisciplinary Creativity in Arts and Technology, Corfu, Greece, June 1–2, 2018. Corfu: CEUR-WS, pp. 181-186.

128.

World Health Organization. International statistical classification of diseases and related health problems, 10th revision (ICD-10), Fifth version. World Health Organization.

129.

Warren

Wakefield

Mills

. Learning and Teaching as Communicative Actions: Transmedia Storytelling. In: Wankel

Blessinger

, (eds) Increasing student engagement and retention using multimedia technologies video annotation, multimedia applications, videoconferencing and transmedia storytelling. Bradford: Emerald Group Publishing Limited, pp. 67-94.

130.

Pratt

Steward

. Applied behavior analysis: the role of task analysis and chaining, https://wwwiidc.indiana.edu/pages/Applied-Behavior-Analysis (2020).

131.

World Medical Association Declaration of Helsinki. JAMA 2013; 310: 2191.

Waking up in the morning (WUIM): A transmedia project for daily living skills training

Abstract

BACKGROUND:

OBJECTIVE:

METHODS:

RESULTS:

CONCLUSIONS:

Keywords

1. Introduction

2. Theoretical and conceptual framework

2.1 Transmedia reality, pedagogy and disability

2.2 Learning through games, gamification and simulation

Table 1 The 5W2H framework of WUIM

2.4 User experience design

Author contributions

Ethical considerations

Footnotes

Acknowledgments

Conflict of interest

References

Table 1
The 5W2H framework of WUIM