Assistive apps for activities of daily living supporting persons with Down’s Syndrome

3.1. Money-Handling Training App with interactive table

The interactive table is a prototype of a new interactive device. It can be compared to a 3D touchpad. It senses the hand when touched, but also when the hand is hold above the table up until 20 cm.

To use the advantages of the interactive table to support the learning experience we developed an interaction strategy of combining screen visualization and real world interaction, as shown in Fig. 1. The advantages are that gestures can be performed on the surface of the interactive table as well as in above the interactive table. Due to the big size of the table there is more space to interact directly.

OPEN IN VIEWER

Fig. 1.

Mockup of initial MHT App idea (a) and first software mockup (b).

We designed the application slightly game like. The initial idea of the application was that the person with Down’s Syndrome sits down in front of a screen and has the interactive table in front of her. We start the learning application. On the table an overlay is placed with the same basic layout as on the screen, as well as the used coins in form of real money. In the first scenario the application asks the user to sort the money in the “boxes”, the designed areas on the overlay. The user looks in the pile of coins in front of him for the indicated coin on the screen. Once found, the user puts it in the corresponding box on the table, like indicated on the screen. The overlay layout with the boxes could be fix or configurable, in order to adapt to the set of coins used for learning. Once this is done, the screen shows the sorted money and the corresponding sorting on the table is done as well.

In the second scenario a product appears with a price tag displayed at the cashier, see Fig. 1(b). The application asks the user to select the right amount of money in order to buy the displayed product. By performing a gesture on the table at the desired box filled with a coin or bill the user selects this coin or bill. Then the user executes another gesture in the orange area on the table, acknowledging the money selection. Consequently, the selected money appears on the screen. Before acknowledging, the user can change his selection. The user chooses the coins and bills necessary to pay the correct amount for buying the product. Subsequently, when having selected enough money, the user presses a button at the top of the table and thus pays for the product getting feedback and moving on to the next product.

For the first version of the Money-Handling Training App we implemented the second scenario of paying a product at the cashier. A screenshot and the overlay of the table are shown in Fig. 2.

OPEN IN VIEWER

Fig. 2.

(a) Screenshot and (b) overlay of Money-Handling Training App with interactive table, version 1.

The rationale behind this learning application follows the learning phase’s children with learning disabilities need to acquire to be able to handle money in real-life. With our application we are addressing following learning development steps, listed from the school curriculum for special education in Germany [20].

These learning phases have as a prerequisite the lower level steps of getting to know the coins and bills, know their value, understand the comma and the zero and know how to change money.

In the category of reading and giving the right amount of money only one difficulty level is addressed, adapted to the most real-life products. Users need to be able to read and give full and cent amounts of money. This first version does not give the choice of influencing the price tags.

OPEN IN VIEWER

Fig. 3.

Money-Handling Training App version 2, (a) screenshot and (b) overlay.

The second version of the Money-Handling Training App, see Fig. 3, empowers the users or carers to pay for their desired products. An image of an egg pack, which is taken by the user is displayed. The price tag is also selected by the user. In this manner users are able to reproduce their shopping lists, as it is used in collaboration with the POSEIDON system, or configure products in such a way, that different learning levels are addressed. The learning development steps addressed by the second version of the MHT App are:

By configuring the products, two additional learning sub goals are addressed in the category of reading and giving the right amount of money. By improving the feedback in the second version the user learns when he has payed too much and to await change. Figures 4(a), 4(b), and 4(c) show the three feedback symbols used in the first version of the app. Figure 4(d) shows the symbol for expecting change.

OPEN IN VIEWER

Fig. 4.

Feedback screens of MHT App version 1: (a) positive feedback – payed exactly right; (b) negative feedback – not payed enough (c) payed to much – too much money payed; and (d) improved get change indication from MHT App version 2.

Throughout Figs 1, 2 and 3 the changes in the overlay from the initial idea to the two versions of the MHT App are shown.

The need of moving money on the screen in boxes has generated the idea to have an overlay placed on the interactive table. The result is, that the money has moved in front of the user and the table recognizes the field where the hand is choosing a coin or bill. The second version of the overlay was designed in collaboration with teachers of special education. They have proposed to group the coins and bills separately. Furthermore, the coins are grouped into ones and cents. This visual differentiation helps in attributing the coins to the positions in front and after the comma. Furthermore, following the same rule in all groups the money is arranges in rising order of value from top to bottom and from left to right.

We developed the MHT App using the game engine Unity. This allows us to make the App portable and execute it on PCs and android devices. Figure 5 shows a screenshot of version 1 of the MHT App on a PC screen. Here the application screen and the previous overlay from the table are merged. The App holds the same functionality. Selection of coins and bills occurs by using the mouse.

OPEN IN VIEWER

Fig. 5.

MHT App version 1 for PC.

OPEN IN VIEWER

Fig. 6.

MHT App for tablet and smartphone (a) version 1 and (b) version 2.

Having developed the MHT App using the game engine Unity it is portable to android devices such as tablets and smartphones. Figure 6 shows a screenshot of MHT App version 2 on a smartphone. Due to the touch ability of the touchscreen of the phone, the lower part, where money can be selected has been improved by removing the now futile orange selection field. The money now directly gets selected by touching it. Furthermore, due to space restrictions of the new screen size, the overlay design of version 2 has been replaced by increasing the size of coins and bills.

As presented in Table 1 the interactive table has been evaluated in two different situations. The first evaluation, Pilot 1, was carried out by 5 persons with Down’s Syndrome (4 male, 1 female) at least 16 years old, living with their parents. These were three participants in Germany and two in Norway. The participants in Germany had access to their own setup of the interactive table at their home PC. The participants in Norway trained during the regular visits of the pilot supervisors. All participants were required to try out the application at least 10 times during a period of two months. Quantitative and qualitative results from questionnaires and observations of Pilot 1 are presented in Section 6.1.

Extended Pilot 1 was carried out in form of three one-day events. The workshops have been carried out in three different countries with differing number of persons with Down’s Syndrome. The persons with Down’s Syndrome (primary users, PUs) were accompanied by at least one of their carers (secondary users, SUs). Two of the workshops aimed at trying out the MHT App in combination with the interactive table and gathering feedback. The aim of the third workshop was to compare the different input modalities: mouse, touchscreen and interactive table.

OPEN IN VIEWER

Fig. 8.

Evaluation of MHT App version 1 with different interaction devices. Short usage explanation of interactive table and task (top left), using the interactive table (top right); using the PC (bottom left) and using the tablet (bottom right).

In the Extended Pilot 1 in UK, 9 PUs and 7 SUs participated. Three of the PUs were female, six were male. In Norway 5 PUs and 4 SUs participated. Two of the PUs were female, three were male. Primary users were between 18 and 38 years old. The PUs tried out the MHT App with the interactive table. After trying out the application, PUs were asked if they could imagine to use the applications again in the future and how they liked the applications. SUs were asked what they thought could be done to make the application better for the person with Down’s Syndrome. Further questions referred to the functionalities of the different devices. They also filled in questionnaires about design, usability, helpfulness and personalization.

At the Extended Pilot 1 in Germany 14 PUs and 11 SUs participated. Eight of the PUs were female, six were male. PUs were between 10 and 39 years old. During this workshop the MHT App was tried out on different devices: interactive table, tablet and PC, see Fig. 8. Each PU used the devices in different order, preventing biased results. Independent of the first device used, we first explained their task which was to purchase different products. At the interactive table we also briefly explained its usage. Version 1 of the MHT App was working by knocking on the interactive table, in the areas indicated by the overlay.

After the PUs had “bought” ten items on a device, SUs (who observed the PUs during this time) were asked to fill in two questionnaires together with the PUs. In total, they completed six questionnaires, two after each device. The questionnaires they had to fill in were the NASA TLX [17] and the SUS usability scale. The NASA TLX is used to measure perceived workload on different scales (Mental Demand, Physical Demand, Temporal Demand, Performance, Effort, Frustration; participants rate within a 100-points range with 5-point steps). The higher the score, the higher the perceived workload. The SUS scale is an instrument to measure the usability of interfaces with the help of ten items. Participants indicate with five response options; from “Strongly agree” to “Strongly disagree”. The participant’s scores for each question are converted to a new number (0–4) and then added together. At the end they are multiplied by 2.5 to convert the original scores of 0–40 to 0–100. Based on research, a SUS score above a 68 would be considered above average and anything below 68 is below average. The idea was to compare the three different devices regarding workload and usability.

After the evaluation results of Money-Handling Training App version 1 were computed and recommendations extracted, version 2 was tried out with a few PUs. The changes with respect to the interaction scheme with the interactive table, hovering now being possible and a more precise recognition algorithm of the table in order to prevent fore-arm detection faults worked as expected and proved the improved interaction with the interactive table.

Resulting from the recommendations extracted during Pilot 1 and Extended Pilot 1, improvements were implemented in the MHT App. This second version was previously described in Section 3.1 and our previous work [18].

Pilot 2 included the evaluation of this second version of the MHT App and additionally the Shopping App. For this second long term trial of one month the android version of the MHT App and the Shopping App were evaluated using smartphones, as part of the POSEIDON App. Nine persons with Down’s Syndrome from three countries, three from each country aged between 15 and 28 years, living with their parents participated. Three PUs were male and six were female. The pilot consisted of 4 visits. During each of these the required tasks up to the next visit were discussed and a certain set of tasks were executed by the PUs. In the case of the MHT App the tasks were to create a list of products, a shopping list. Paying for these products was subsequently trained. The same shopping list was used to go out shopping and use the Shopping App. Online-questionnaires, observation sheets, interviews and user protocols were the instruments used to gather the evaluation results. These are presented in Section 6.2.

5.3. Evaluation of the Money-Handling Training App – Comparison of device sizes

Usability testing of mobile applications involving people with DS is an issue that has not been comprehensively investigated. In this context we developed USATESTDOWN. It is a usability testing guide for mobile applications, focused on people with DS. It is based on a literature review and experience gained evaluating more than 108 people with Down’s Syndrome interacting with mobile devices.

In the following we describe the process to evaluate the usability of MHT App with the USATESTDOWN tool by describing each of the steps depicted in Fig. 9.

OPEN IN VIEWER

Fig. 9.

Workshop process for testing MHT App with USATESTDOWN.

At the beginning, we individually explained to the participants the purpose and process and requested their consent on testing the game. They agreed with the option to play with a mobile device. Additionally, we asked if they agree that their hands are recorded, to which all participants agreed.

We asked the participants to fill in a general questionnaire about age, name, and previous experience with touch screen devices.

Then, we worked with the participants individually and explained how the application works in an easy way. This helps the participants to understand a real example with a shop that the students know.

For every device we conducted three tries. This means each participant had to buy three groceries using the MHT App. There was no specific time limit to finish the task, because they may feel uncomfortable or forced to end the task, which could change the evaluations results.

After the participants did the six tries to buy (three for every device), we posed a satisfaction questionnaire, asking which device they prefer, to play with the application.

During the personal interview we asked the participants if they think the application is useful for everyone. Another question was if they think it is easy to use or not.

Maria Corredentora Centre is a special education centre in Madrid for children with DS [16]. The centre has four education levels: Early Childhood education, Basic stage education, DVA stage education, and professional programs. There are children with DS, between 5 and 16 years old. Each group has finished general skill development. We selected our evaluation group taking into account their age as our main parameter. From the remaining group we randomly chose the participants.

This section describes the execution of the previously described USATESTDOWN method applied on the MHT App.

We divided the evaluation process in two phases according to our two evaluation sessions. The goal was to evaluate the usability of the MHT App version 2.

As test systems we used two mobile devices – a mobile phone with 4 . 7 ″ screen and a tablet 10 . 1 ″ screen, with Android 6.

In the first phase, we used both the mobile phone and the tablet. The participants were 20 children between 7 and 8 years old. In this case it was not necessary to ask about touch screen experience, as the Maria Corredentora Centre has an official tool called PICCA 2 which works on a tablet. Using this app the centre teaches the children at least twice per week. We followed the process of USATESTDOWN, as described before.

We wanted to know if the difficulty of using the MHT App depends on the user’s age. We followed the same USATESTDOWN process as described before. For this we used two user groups of different age spans. The first user group was at the Maria Corredentora Center, where we worked with 18 children between 10 and 11 years. The second user group was located in the Apadema Centre, which is an Occupational Center that hosts 44 women [2]. The participants were 17 women with DS, between 26 and 56 years old.

The satisfaction questionnaire result from the evaluation comparing the device size indicated that the children prefer to work with the tablet and not with the cell phone. The main reason was the MHT App interface, as the buttons and icons are too small, to be properly displayed on a mobile phone screen. It was a major concern for DS users, as many of them have vision problems. For this reason we used a 10 . 1 ″ tablet as testing device, see Fig. 10.

First findings show that there is a difference between children and adults people with DS, managing the MHT App. Children needed more support on mathematic logic. More explanation of the basic operations was required for the children, compared to the adults. However, this had only a minor impact on the success rate.

OPEN IN VIEWER

Fig. 10.

Evaluation of MHT App version 2 with tablet and smartphone at Maria Corredentora and only with tablet at Apadema Centre.

6.1. Evaluation results and recommendations for Money-Handling Training App with interactive table

Five pilot participants answered the questionnaire on the MHT App. None of the SUs sees this game as a gimmick. Instead, most of them think that MHT App helps to strengthen the memory of bills and coins, that it promotes the safe handling of money in everyday life, that it helps to better distinguish bills and coins in everyday life and that with the help of the MHT App acquired knowledge can be easily applied in everyday life. On the one hand, these results are very promising, since strengthen a safe money handling is the major aim we try to accomplish with the MHT App. On the other hand, many suggestions for improvement were collected, which have to be taken in consideration.

Playing the MHT App was fun for most of the PU not at least because of interacting with the interactive table, because most of them got the feeling of mastering something. SUs see the MHT App as a chance to increase knowledge about coins and bills. PUs like the gamification approach. One SU argued that the PU better distinguishes the euro and cent coins after only four or five times playing. The SU also mentioned that this of course is not comparable to a real world situation but that the MHT App can help to get used to money, different prices and values. Most of the SUs also liked the colours and the design. They also praised that a feedback function is included and that interacting with the interactive table is a new form of interacting. Apart from liking the idea and the fact that one important everyday skill is supported, SUs gave a lot of suggestions for improvement, reported in Section 7.1.

Some SUs mentioned that the interactive table is big and bulky, and too big for some of the PUs rooms. But the idea of interacting with the interactive table and the game itself compensates for this disadvantage. Due to the size it seems to be ideal for schools. One SU mentioned that one advantage of the device is, that it has a fixed location and therefore became part of the daily routine, e.g. playing the MHT App before or after dinner. Some SUs mentioned, that the table did not always do what was expected. Sometimes the money did not appear when knocking on the money motive, other times it disappeared again, or they had to knock several times before the money appeared. When standing too close to the table, it also happened that the table did not behave as expected. The fact that the table was sometimes too insensitive and too unstable leads to frustration or irritation of the PU.

The arrangement of coins and bills on the surface of the interactive table was mentioned as being not perfect more than once.

SU were also asked, what they think of this learning method (using the interactive table) compared to other devices. Some of them said that they don’t have any other experiences. One SU said that using the interactive table could be more motivating and one SU mentioned that it also could work on a tablet.

The feedback from PUs in the Extended Pilot in the UK and Norway was that the interactive table was that they liked it very much and that they would like to continue using it in the future. From the SU, who answered a questionnaire most thought that the MHT App was easy to understand, half said that it is fun to use, most were not sure if it is better compared to other MHT Apps. Most SUs said that the interactive table was suited for this kind of application. However, this is in contradiction to the overall answers to the questions: How do you like the idea of using the interactive board? How do you like the idea of having an interactive board in your home? Would you like to have an interactive board in your home?

Here the SUs thought that the interactive table should be used more in the area of schools and that it was fun to use. They didn’t like the idea of having an interactive table at home. Most said it was too big, some considered a smaller version and said that a tablet would suffice.

The results from Pilot 1 and the Extended Pilot are more or less similar. The recommendations are compiled and presented in Section 7.1.

The evaluation of the Extended Pilot in Germany where the use of interactive table, PC and tablet were evaluated, resulted in results gained with the help of the NASA TLX, see Table 2. This indicates that there seem to be only small differences between the three devices for most of the scales. We were aiming for low numbers, like little frustration or physical demand. The biggest difference was found for Physical Demand which seems to be lowest for tablets (10.2) and highest for PCs (36.8). The perceived physical demand for the interactive table seems to be between the other devices (25.5). The effort for interacting with the interactive table seems to be similar to the tablet. Surprisingly, the lowest value was found for the PC. The SUS value indicates that the usability seems not to vary in dependence on the device. As a conclusion, there is room for improvements regarding usability for the MHT App.

The design and the handling of the MHT APP was considered as appropriate. Almost all think that the font (size) is appropriate and that the colours and contrasts used ensure good readability. It also became clear, that the usage of the MHT APP become easier after four weeks of testing for the PUs.

None of the SUs saw this training game as a gimmick. Instead, most of them thought that MHT APP helps to strengthen the memory of bills and coins, that it promotes the safe handling of money in everyday life, and that it helps to better distinguish bills and coins in everyday life. Furthermore, with the help of the MHT App acquired knowledge can be easily applied in everyday life. These results are very promising, since strengthening a safe money handling is a major aim we try to accomplish with the MHT App. But much work has to be done as there are many suggestions for improvement.

6.3. Evaluation results of Pilot 2 for Shopping App

All SUs thought that the Shopping App has a good design. This is true for the font, font size, contrasts and readability. However, the opinions diverge regarding joy and acceptance.

Most of the SUs think that the Shopping App helps to strengthen the memory of bills and coins, that it promotes the safe handling of money in everyday life, that it helps to better distinguish bills and coins in everyday life. Nonetheless, the majority is not sure it helps the user to easily organize daily life. Moreover, there are many who do not think that the app is enough personalized to the needs of the user.

After four weeks testing the opinions about the Shopping App became better: more SUs disagreed on the statement that the function is only a gimmick, but more SUs think that it helps the user to easily organize daily life.

6.4. Evaluation results on device size and age group comparison

We asked every participant about their device preference. 95% of participants (20 children) answered that they prefer the tablet for interaction. This was also confirmed with the participant’s behaviour when they took the mobile telephone close to their face, because it was difficult to see the different buttons and icons on the screen. Most persons with DS have vision problems, which is a real limitation for applications that have too many icons.

Furthermore, we analysed the videos looking for the success or failure rate, when buying the product. If the participant manages to buy at least two products, it was considered a successful try. At Maria Corredentora Centre we found only one student that succeeded. At Apadema Centre the analysis showed that one participant bought 2 products and another participant bought 3 products, leading to two successful executions of the task.

At both centres, 82% of the participants answered that the application is useful in the real word and they would like to play with it. Additionally, 91% of the participants answered that the application is complicated to manage. The application therefore is in principle suitable to teach people with DS, and should be extended.

7.1. Money-Handling Training App version 1 with interactive table

As a result of all these comments we have adjusted the overlay and some functionality of the MHT App. Figures 2 and 3 show the adjustment of the overlay. The coins are displayed in relative size of each other and the money was organized in value groups. We addressed the “getting money back”-functionality by exchanging the negative feedback icon of “thumbs down” with a symbol which lets the user know that she should expect to receive change, see Fig. 4(d). The recommendation to adjust the MHT App to different levels of competency could only be addressed by implementing personalizable products. The carer is thus able to add his own pictures of products and also add the price tag. Through this, difficulty in training level can be modified.

7.2. Money-Handling Training App version 2 on smartphone and tablet

Some of the recommendations can be addressed quite easily, like reinforcing positive feedback by playing a happy acoustic sound, displaying the change in form of bills and coins or having the option to undo a previous action.

By personalizing the products which could be bought, the first step towards creating multiple levels of difficulty has been reached. Extending the app by having multiple shopping lists, corresponding to different levels of difficulty, could be used to have multiple levels of difficulty available at once. However, giving appropriate guidance, meaning feedforward which is adapted to the capabilities of the user is a more complex task.

7.3. Shopping App on smartphone

Most of the recommendations regarding the improved interaction with the Shopping App can be implemented straight-forward, such as creating multiple shopping lists, having a zoomed view on the shopping items, as well as reminding messages. Other recommendations like reducing the workload of the carer which needs to keep price tags up-to-date, or having a list only with text, are extremely dependent on the capabilities of the person with DS. Putting all bills and coins on one single smartphone screen, such that the user is not confused, is also a difficult design task since many users with DS suffer of bad sight and need big enough images to see clearly.