IDMILE: An interactive didactic math inclusion learning environment for blind students

Abstract

The preparation of mathematical learning content for blind students does not include many elements of the implicit didactic information included in the visual layout of math schoolbooks, e.g. spatial arrangements for sequencing or relations, color-coded information, and other layout based content representation. This kind of information is an indispensable component for up-to-date math learning and teaching. Due to the limitation of other senses and assistive devices for blind people, this missing of explicit semantic mark-up of these implicit didactical elements is seen as one key reason for the disadvantages of in STEM education and related job opportunities.

This paper presents basic research on developing an approach for making this implicit information accessible by marking-up the implicit visual elements used in math schoolbooks, which goes beyond linear text. Describing and marking-up this dynamic content forms the base for our prototyping of an inclusive math-learning environment for blind students. Interactive dialogues guide the blind user and provide assistive functionalities for better managing the complexity when solving math problems using sequential audio, refreshable braille display, and other presentation and interaction methods. In this attempt, our prototype covers four basic arithmetic operations. We use the Eclipse SWT framework and Microsoft UI Automation to be in line with known accessibility concepts and skills of users. The prototypes have been evaluated both by blind and sighted persons.

Keywords

Interactive non-visual user interface math solving process didactics blind

1. Introduction

Within the last decades, the group of visually impaired and blind people witnessed considerable pro- gress in their ability to use modern information technology. These advances grant them access to almost every aspect of today’s information society, substantially enriching their participation in educational and vocational life as well as in entertainment and in leisure activities.

However, one area remains, in which the mentioned advances tend to have very limited effect, and this is the field of mathematics. Since many mathematical formulas are presented in a two-dimensional, graphical manner, current information technology for visually impaired and blind people, which relies on linear, text-only output modalities, cannot present mathematics in the same way as for sighted people. The situation gets even more critical when it comes to solving mathematical problems.

The aim of our research is to furnish the first contribution to support blind students when learning the process of solving elementary arithmetic tasks. The software prototype described and evaluated here is based on analytical research about teaching methods found in mainstream mathematical schoolbooks and observing teaching mathematics in schools for young blind students [1, 2]. The visual display of math encodes a lot of instructional information using e.g. non-linear spatial arrangements, jumps, color or other layout markers, which are perceived as implicit instructions how to proceed in solving a mathematical task. This implicit visual information is not made explicit (no text alternative) and no efficient techniques and tools are available for supporting the calculation process in a comparable way.

We selected three Austrian math schoolbooks and observed the math classes for blind students to identify and formally describe the methods used to teach the four basic operations of arithmetic. This allowed us to propose alternative computer and AT based methods suitable for blind students. In this paper, we present these prototypes integrated into a software suite called IDMILE – Interactive Didactic Math Inclusion Learning Environment for Blind Students.

2. Problem and research definition

The problem is a crucial one since mathematics nowadays is a fundamental component of almost every area of higher education. Being unable to deal with mathematics in a satisfactory way puts visually impaired and blind people at a disadvantage in many educational fields and in the labor market.

The present paper contains the first contribution to employing modern information technology to support blind students in very elementary tasks of mathematical learning. Before describing the basic idea, we shall outline the three levels a blind individual faces problems with when dealing with mathematics: presentation, navigation, and manipulation [3].

2.1 Presentation

Most of the mathematical formulae and their pro- blem-solving processes are represented in 2-dimen- sional, graphical notation, for instance: a fraction, where the fraction bar is written on the baseline, whereas numerator and denominator are placed somewhat above or below the baseline, respectively. This 2-dimensional representation, which has been proved effective over the last centuries for sighted people, is inaccessible to blind peoples’ output modalities, synthetic speech and braille display, which are both linear, one-dimensional techniques. Although quite sophisticated one-dimensional codes – mostly braille codes [4] – have been developed, they are often hard to learn, and they do not furnish an effective means of communication between blind and sighted people, which would be indispensable in a setting where a blind student is taught in a mainstream class.

2.2 Navigation

Navigation or “browsing” math content is the process where blind students can explore the mathematical formulae or expressions to develop a mental map and understanding. Even if a blind student may read a mathematical expression by means of representation, he/she will experience considerable difficulty to understand its structure and meaning: while a sighted student may use his/her vision to acquire an overview of a formula “at a glance”, even if it may be quite complex. A blind person needs to sequentially traverse the whole formula in order to fully read it – the graphical representation, which, by depicting the formula’s structure, assists the sighted person in understanding the structure “at a glance”, is not available to the blind person, and it has not found a suitable counterpart by now that would grant comparable support. In addition, the visual representation stays at focus where audio or braille is fugitive and has to be re-visited sequentially.

2.3 Manipulation

By nature, this is the most challenging issue. Even if a blind student may be able to understand mathematical formulae, he/she will run into additional challenges when working with a mathematical expression. Already at the level of relatively simple calculations from elementary arithmetic, one will need to do sub-calculations, such as marking elements of an expression as already dealt with, marking other parts of an expression to be dealt with later on, or many more tasks that are similar. Such tasks will quickly organize the calculation in a structure, which is much more similar to a tree, or a graph, than similar to a linear sequence, which will be extremely hard to traverse for a blind person. Since many elements of that structure will not be within the blind person’s reach (see navigation), he/she needs to memorize them, which causes a tremendous memory load, distracting him/her from the original task of learning, understanding, and practicing mathematics.

While, as can be seen from the next section, quite many efforts have been taken to overcome the first two basic problems, representation and navigation, almost no research is has been carried out to tackle the non-visual manipulation. We are convinced that modern information technology, which has been immensely helpful to blind people in dealing with textual contents, does have a strong potential to supply similar support in successfully managing mathematics. In this context, it is interesting to mention that even sighted mathematicians, rather than using the computer, tend to do calculations by paper and pencil: first because this method proved to be effective over centuries, second, because, even for the sighted mainstream, almost no software tools to support a human in calculating are used. An abundance of mathematical editors, also called formula editors, let one easily write down mathematics, and several highly developed computer algebra systems automate calculation tasks, but the layer in between, a tool to assist a human in doing a calculation him-/herself, is widely missing. Only recently some first steps towards doing math with ICT are done using handwriting recognition on touch devices, connected to computer algebra systems [7]. We are thus convinced that the research we are doing for the benefit of blind students may also be of general interest.

3. State of the art in non-visual math education

The process of learning how to manipulate or solve mathematical problems is one of the most complex combinations of science and art. Along with the logical sequential solving steps, one has to know the implicit meaning between the lines. This is because the mathematical formulae have their own syntax and they are illustrated in 2-dimensional representation, which builds up the spatial arrangement that has its own meaning depending on its position and placement. For the pedagogical purpose, both pictorial and spatial representation of mathematics is used as visual implicit didactic information, which can be observed in the theoretical part of mathematical schoolbooks designed by educators. During the learning process, a sighted student may not realize this kind of information explicitly, but it will affect the learning success of him/her, especially the schematic spatial representations [8]. Unfortunately, this implicit visual didactic information gets lost when mathematical contents are being converted for blind students for sequential use by audio or braille.

Without reasonable didactic information, a blind student would have a hard time to learn how to solve a mathematical problem. In the past, solving a mathematical problem using a mechanical braille typewriter was popular and widely spread. For navigation and finding a spatial position, a blind student can move the carriage return lever of the mechanical braille typewriter. The challenges of using this assistive technology can be depicted into four aspects: a) portability, the mechanical braille typewriter is a heavyweight piece of equipment and hardly portable, b) sufficient finger strength requirement, c) loudness, and d) restricted possibility to correct typos [9]. The electronic braille typewriters are later on invented, but they still cannot solve some problems. Nowadays, most of the blind students use a computer with assistive technologies (screen reader with audio or refreshable braille display) for studying mathematics. This combination compensates the functionalities of a braille typewriter in the old days. However, in case of learning and practicing mathematics, there still exist no suitable computer-supported tools. A blind student still would use traditional software, e.g. a text editor or spreadsheet, to solve a mathematical problem.

Following this analysis, we defined the first R&D task that is seen in analyzing visual math content to make the implicit didactic mathematical information explicit and marking it up to allow processing it for IT-based support functionalities.

4. State of the art in assistive technology for accessing and doing math

Efficient and professional IT-based assistive technologies are available for visually impaired and blind people, e.g. screen magnifier, screen reader software, refreshable braille display, tactile display, synthetic speech etc. [9, 10]. Also in terms of the three basic problems in dealing with mathematics mentioned above, researchers and companies came up with approaches matching the blind students’ special needs [11].

For representation of mathematics, several extensions for original braille codes were developed, which are called “mathematical braille codes” [4]. The advantage of these notations is that they reflect the need of blind people of having shorter expressions. The striking disadvantages of mathematical braille codes are, that:

•
Firstly, different countries developed different codes, such that mathematical material from a foreign country is often hard to read.
•
Secondly, the codes are quite hard to learn, because in the endeavor to save space, which is valuable in braille, quite sophisticated systems have been defined.
•
Thirdly, sighted people usually can neither read nor write the mathematical braille codes, such that they do not facilitate mathematical communication between both groups, which is indispensable for a standard inclusion classroom setting. Moreover, an automated translation between different braille codes or between a braille code and a code common to the mainstream is almost impossible, because braille codes typically have context-sensitive grammars.

In an attempt to overcome the isolation caused by the use of mathematical braille codes, blind people now quite often use mathematical typesetting system as LaTeX to read and write mathematical content. LaTeX is easy to learn as it is an ASCII notation, and it provides special unambiguous codes for coding mathematical formulae [11]. An additional proof LaTeX is the power of producing visually appealing mathematical text to support inclusive communication. This theoretically solves the representation problem. In practice, however, LaTeX content is hard to read for a blind individual, because it tends to become lengthy and the mathematical symbols are intermingled with formatting code. Approaches of shortening the LaTeX notation and cleaning from layout commands for blind users (e.g. Human Readable TeX [12]) helped to partly overcome the issue but asked again for more learning.

An alternative to LaTeX are web mark-up languages as MathML [13] or OpenMath [14]. The W3C standard for publication of Mathematics on the web, MathML, seems to become the best-supported and most widely used notation. While practically not human-readable, some of the modern screen readers such as JAWS [15], NVDA [16], and VoiceOver [17] are able to output formulae written in MathML by well-understandable, structure-aware speech and even in some of the braille codes mentioned above. Moreover, they offer navigation mechanisms to the user, such that he/she may traverse the tree-like structure of a formula in a convenient manner. This way, the integration of mathematical content into screen readers does not only contribute to a solution of the representation problem, but also to the navigation problem. Although not yet technically mature and widely used, we value this approach as the way forward, as it can use well-structured and semantic rich standard markup. To mention one example, the LEAN math editor provides functions for creating and editing mathematical formulae based on MathML [18]. Due to the mentioned starting trend of moving from paper and pencil to using touch devices, handwriting recognition, and computer algebra systems for doing math [7], we can expect a common platform to access math content and to implement functionalities supporting the doing of math. Another solution used for overcoming isolation caused by mathematical braille code is the use of linear text-based mathematical notations, which is provided for common mathematical expressions for learning and communication purpose in inclusive classrooms [19].

The Infty Group has been developing and maintaining a product called InftyReader [20] that uses OCR (Optical Character Recognition) technology combined with intelligent software to recognize scientific documents and mathematical formulae and then converts them into blind-friendly formats. An approach from ViewPlus [21] prints out and embosses information such as maps, graphs, etc., in braille and graphics. Handy Tech [22] provides a 2-dimensional refreshable dot system with a resolution of 24 by 16 pins. Similar to Handy Tech system but with higher resolution is a system from HyperBraille [23]. The system is a refreshable tactile system with a resolution of 120 by 60 pins for displaying text or simple graphics. The drawbacks of these devices are the heavy weight and the high price.

For navigation and understanding mathematics, the Math Genie was developed by Arthur Karshmer and his group [24]. Released already in the late 1990s, it enabled the user to traverse the tree structure of a MathML document, giving feedback through synthetic speech with lexical indicators such as “begin fraction” or “end fraction”. Another approach using sound is called ASTER system [25], which is a tool to convert LaTeX documents to a format that could be used as audio documents. A blind user can then navigate through the mathematical formulae actively. The MATHS [11] system uses the combination of both speech and non-speech audio to build up the basis for transmitting useful navigation information to the user.

Within the MICOLE project [26], a software prototype was developed to furnish a collaborative environment between blind and sighted students. The prototype enabled navigation in the tree structure of a formula through a collapse/expand mechanism. The idea was to synchronize two different views onto a formula, a braille view for a blind individual, and a visual view through standard graphical rendering. Not only the views but also pointing actions on them are synchronized, in that, if a sighted user performed a mouse click at a spot in a formula, the braille display immediately jumped to that point. Conversely, if a blind student uses the cursor routing buttons on the braille display to point to a spot in the braille view, the mouse cursor in the visual view jumps to that spot.

Manipulation of mathematics, as mentioned before, is by far the most complex task among the three defined problems. Nonetheless, we mention some activities:

Arthur Karshmer and his team developed the MathOMatic Blocks System [27], a tactile interactive learning environment for blind students in elementary arithmetic. On a pad, wooden blocks are placed, which are furnished with tactile digits and letters. By positioning the blocks on the pad, a student may write down a mathematical assignment, which, by sensitivity features on the pad, may then be stored on a computer for further processing.

A promising approach is the LAMBDA project [28]. It provides a mathematical editor with components to tackle each of the three basic problems outlined: A novel, configurable, and intuitive braille code for reading and writing mathematical content, a graphical view facilitated the collaboration of blind and sighted people and a collapse/expand mechanism to support navigating the structure of an expression. First functionalities for manipulation support were implemented, e.g. selecting parts of an expression and copy them into a buffer.

Silvia Fajardo Flores in her Ph.D. developed a software prototype to support blind students in doing elementary algebraic tasks [29, 30].

Unfortunately, neither of the tools mentioned above is explicitly designed for helping blind students learn mathematics and its solving process. Many of the above approaches stay at a prototype level and did not yet reach the attention of educators.
5. Studying the math solving process

To learn and get a better understanding of the process of teaching and learning math, we visited math courses in two elementary schools for blind students: In the Landesschulzentrum für Hör- und Sehbildung (LSZ) located in Linz, Austria, and at the Bundes-Blindenerziehungsinstitut (BBI) located in Vienna, Austria. The classes are for visually impaired and blind children from grades 2–4. Young blind students get support in understanding mathematics and learn how to solve basic mathematical operations. They used different types of tools for different arithmetic assignments. At LSZ, the young blind students learn how to use mechanical braille typewriters as a tool for solving mathematical problems. During the mathematical solving process, the young blind students spoke aloud what they were doing. The blind students at BBI are older than at LSZ and used the computer with refreshable braille display and Microsoft Word as a tool for solving mathematical problems. The solving procedure was almost the same as at LSZ, but the used tools are different. Although, similarly to LSZ, the blind students spoke aloud what they were doing, too.

We used controlled and natural observation as methods with an open protocol [31] for studying activities, behavior, and the environmental process. The principles of Heuristic Evaluation by Jacob Nielsen [32] were used to evaluate usability in interviews based on a questionnaire.

5.1 Blind students’ keyboard usage behaviors

Blind people use the standard keyboard as input device. For a blind student, a keyboard is not only the input device for inputting the text, but also for commanding and navigating. Therefore, the use of his/her own keyboard is very important, in order to find and use the particular key quickly and intuitively.

The observation of blind students using keyboard at the schools for young blind students and blind mathematical experts shows that a large number of both blind students and experts tend to use the well-known keyboard shortcuts and the TAB-key. The keyboard shortcuts help them to be at the wished location directly, and the TAB-key transfers the focus of the program from one user interface element to another. The latter enables the navigation of active elements throughout the program.

The unusual key combination shortcuts or well-known shortcuts that do not facilitate the expected functions cause difficulties for blind students. A key combination with more than two keys, e.g. Ctrl $+$ Alt $+$ K, causes a blind student to take more time to find the right keys, and he/she may press on a key too long. This can affect some unwanted program behaviors. For canceling a task or exiting a program, the ESC-key is the most popular one. Whenever a blind student does the wrong step or does not know how to go further, he/she will execute this key.

5.2 Blind students’ braille display usage behaviors

Combined with a screen reader software, a refreshable braille display displays the content on the tactile screen, even though showing only one line of text at once. Most of the modern refreshable braille displays are equipped with up and down buttons on the left- and right-hand side, navigation buttons or TAB button below the display field, and the scroll up or down on the bottom of the device. The positions and the functions of the buttons can be varied from one model to another. This affects the behavior of blind students according to the condition of the respective device. Some of the students use their right hand to press on both scroll buttons for up and down. Others use both left and right hands to do the same task.

Most of the observed blind students and experts use 8-dot refreshable braille with 40 characters. To increase the reading speed, some blind participants use both hands to read. However, the switching between braille display and keyboard is time-consuming.

5.3 Implicit didactic information in mathematical schoolbooks

The mathematical schoolbooks in Upper Austria consist of two parts, namely the theoretical part and the practical one [2]. The theoretical part contains the didactic information for the (sighted) students to learn the theories and methods of solving particular mathematical problems. This didactic information can be both pictorial and spatial presentations, which the sighted students may absorb it with or without awareness. This kind of information, especially the spatial representation, affects the learning ability of a sighted student. Unfortunately, this information will be lost if the mathematical materials are converted for blind students using traditional methods. The possible reasons for losing this kind information could be: a) the modesty of the current assistive technology, and b) the lack of technique to transfer this kind of information.

According to our prior findings [2], the visual implicit didactic information is divided into three domains: 1) implicit visual notations, 2) context-relevant instructions, and 3) spatial arrangements.

The implicit visual notations describe attributes of the used didactic teaching techniques. They highlight important information that is supposed to catch students’ attention, e.g. graphics, color-coded fonts, bold line technique, etc. This helps students to be able to distinguish between relevant information and non-relevant one. The implicit visual notation can be further differentiated into subdomains, including text attributes (e.g. color-codes, font weight, font size, decorations), graphics (e.g. characters, real-world objects, area representation, number line), symbols (e.g. bold lines, plus sign, pointing arrows), and grouping (e.g. group of ten ones to one ten, grouping by using colors, parentheses).

The context-relevant instructions are usually shown aside the ongoing calculation step in the whole solving process. By following the instructions, the students can understand the procedures described in the text. The context-relevant instructions can be differentiated into subdomains, including overview descriptions, calculation step instructions, and definitions.

The spatial arrangements describe the placement of one or more elements, defining their order (e.g. numbers) into (mostly) 2-dimensional representation. For instance, the arrangements embody table grids, columns of hundreds, tens, and ones with ones at the most right position, addition upwards, etc. These arrangements can be differentiated into subdomains, including position (the arrangement, which places the numbers into proper position. The writing of place value abbreviation, such as H, Z, or E, after the number, e.g. 5H $+$ 2Z $+$ 8E, whereas H stands for hundreds, Z for tens, and E for ones respectively.), direction (the arrangement, which guides the learners to the next point, guided arrows, etc.), and boundary (the visible border, which distinguish numbers according to their place values).

The most important finding out of the implicit didactic information analysis is the relation between different objects among the spatial representation. This kind of relation allows both sighted and blind students to build up their own mental map and link this together into meaningful knowledge model. This works in the way that a sighted student would solve a mathematical problem systematically according to the solving process, and will be reflected in our software prototype later on.

The visual implicit didactic information from mathematical schoolbooks can be prepared for blind students with suitable approaches. Since the mathematical problem-solving process relies on the sequential solving steps, we can design a software-based assistant for guiding blind students throughout the process. In addition, the assistant should provide the spatial perception through special navigation concept for blind students.

6. Non-visual user interface requirements

In contrast to developing mainstream software, selecting the programming language and environment for our IDMILE prototype had a particular focus on the following specific functionalities and supports.

6.1 Manageable focus

The system focus is the key element, which is tracked

by screen reader software, meaning that it is shown in braille and speech whenever it changes. Therefore, the easiest way to bring something to the attention of a blind user is to switch the focus to the spot in question. Usually, a standard user interface library of a programming language provides a focus on an active element at a time. This focus can be then shifted from one active element to another according to the order of the UI elements represented in the Document Object Model (DOM) by using TAB key. From the analysis above, the sequential principle of operation of our prototype relies on focus transfer from one active element to another, too. However, the sequence of the mathematical problem-solving steps is not the same as the order of the elements in the original DOM. Therefore, one of the specific properties that a programming language must have is efficient support in managing focus. Our previous analysis [1] shown that Eclipse SWT is best suited for our purposes.

6.2 Operating system accessibility API

The best way to guarantee interoperability between the developed software and assistive technologies (AT) is to let the operating system act as an intermediate platform as both access the same resource. UI Automation of Microsoft Windows supports AT in accessing the necessary information to facilitate an accessible interaction. The Eclipse SWT framework provides an accessible interface for all GUI elements and controls, which can be bound into the elements’ attributes, such as name, description, help, etc. Assistive technology can then properly access the elements’ attributes.

6.3 Different types of widgets

A curtain UI widget might be suitable in one particular scenario, but in another one maybe not. In addition, different kinds of UI widgets can be used to support various interactive tasks depending on purposes. An assistive technology might behave differently on UI widgets that seem to be alike visually. For instance, a screen reader software, e.g. JAWS, may interpret a label with text as a plaintext on a refreshable braille display (without “lb” as a hint) by default, but for interpretation of an input text field, a blank space with message “ed” (which stands for edit) would be applied [33]. This enables the software developer freely to design the software behavior using UI control elements. Furthermore, while a blind student interacts with the software, other (sighted) students and/or a teacher can follow his/her interaction steps and vice versa. This supports inclusive cooperation of sighted, partially sighted, and blind people.

6.4 Supporting APIs

For developing accessible software, an operating system, on which the software should run, should support accessibility. The three mostly used desktop operating systems use different strategies in providing accessibility functions. The Microsoft Windows Operating System provides the so-called UI Automation [34], which is available on all types of operating system that support Windows Presentation Foundation (WPF). UI Automation enables assistive technology products, such as screen readers, to provide access to UI elements. The Mac Operating System includes a screen reader called VoiceOver [17], which communicates with braille displays and speech output. The Linux Operating System has several concepts for accessibility depending on the community of respective distribution. The Ubuntu [35] distribution, for example, provides accessibility functions such as a screen reader, high contrast features and voice recognition.

Some of the GUI frameworks provide an interface that supports the accessibility functionalities of an operating system. The open source Standard Widget Toolkit (SWT) [36] from Eclipse.org employs the UI Automation on Windows operating system directly. SWT can also benefit from most of the provided accessibility functionalities from Windows.

The Java Swing benefits from the so-called Java Accessibility API (JAAPI), which is a part of the Java Access Bridge [37, 38]. Similar to SWT, Java Swing provides a programmatic framework for software developers. However, on Windows operating system Java Swing does not use UI Automation directly but uses Java Access Bridge provided by Java Virtual Machine. Some of the accessibility functionalities may therefore not be supported natively.

6.5 Control elements

Control elements can receive a focus from the operating system according to the order in the DOM. Eclipse SWT framework provides the Accessible class getAccessible to provide additional information to control elements. Assistive Technology as a screen reader can use addAccessibleListener to access this information. Methods as getName (AccessibleEvent ev), getDescription (AccessibleEvent ev), can be employed. The operating system will then prepare the accessible information for assistive technology. Furthermore, a mnemonic key can be added to an active control element by using the “&” symbol in front of a letter that should be the key.

6.6 Controlling screen reader mode

Solving math tasks includes using, setting up or adapting strict spatial arrangements and structures (e.g. the addition of two numbers asks for vertical alignment of the ones, tens, hundreds, …) for both navigating and solving tasks. Screen readers as JAWS use different modes, including a function to omit empty spaces to make sequential reading faster. This might affect spatially arranged calculations where empty spaces convey a particular meaning. For instance, a label and a text element are in the same row (y-positions of these elements have the same value), the texts of the label and text element will be concatenated into one text string. If there are spaces in between theses texts, they will be shown on braille displays causing some confusions for the users and waste of character placeholders unnecessary. This happens in JAWS when using Line Mode. Unlike the Line Mode, the JAWS Structured Mode will show only the active (focused) element on the refreshable braille display. Hence, the concatenated text string will be interpreted with shortened empty spaces and left-aligned. Therefore, for designing an interactive software, controlling the screen reader mode gets important.

Figure 1.

Logical layers of an accessibility concept.

This is also applicable for other scenarios as e.g. when table layout/mark-up gets used to support spatial arrangement and navigation. A screen reader mode might shorten and omit empty cells.

6.7 Shortcuts

Shortcuts are one of the most convenient input methods for blind users. They consist of one or more modified keys (Ctrl, Shift, Alt) and an alphanumeric key. Blind students can use them to “jump” from one element to another without long path as in the common task using the TAB key, a technique especially common with experienced students.

However, using shortcuts can be harmful to blind students, too, when the number of the shortcuts gets too big and tends to cognitive overload and disturbing the solving or the actual task. A developer should also take care not to overwrite the well-known standard shortcuts as it affects using other parts of the system.

The key combination, especially the letter, has to be logical and comprehensive. The letters used as the keys on the keyboard should be located in a close area, for more ergonomic access.

7. Prototype

7.1 Modeling

The prototype has been designed under the principle of maximizing accessibility with explicit didactic support for blind students. From the descriptions above, the most suitable combination selected are Microsoft Windows as the operating system, Eclipse SWT as an interactive GUI framework supporting non-visual access, Java as a programming language, the screen reader JAWS and a refreshable braille display. Figure 1 depicts the logical layers of such an accessibility concept.

The logical layers of this accessibility concept illustrate that the operating system (MS Windows with UI Automation) acts as an intermediate platform between accessible UI (Eclipse SWT) and assistive technologies (JAWS and refreshable braille display). Eclipse SWT provides accessible text for control elements used in the operating system via UI Automation. If needed, JAWS for Windows can acquire this information and read aloud and/or display on a refreshable braille display. In our user studies, blind people tended to prefer a combination of speech output and braille as it allows complementing and switching. Eclipse SWT UI provides interactive feedback and guides a blind user by managing the focus from one active control element to another.

Figure 2.

Software prototype architecture.

The prototype architecture structures software components into four layers following the four levels of the problem-solving process: a) Entering, b) Arrangement, c) Sub-calculation, and d) Evaluation, whereas b) and c) have been encapsulated in one layer. Figure 2 depicts the software prototype architecture.

In the Entering phase, a blind student can input a linear mathematical expression, which will be validated for completeness and operation type for calling applicable support functionalities. According to operation type, a corresponding wizard will be called. The included math engine is used to verify each step of user’s activities in the wizard allowing support at the spot and the provision of context information for better learning effects. In the Arrangement and Sub-calculation phases, the dialogue-based assistant will guide a blind student by means of the important concepts of mathematical didactics, which are implicit to the visual arrangement. A blind student can use the special navigation concept to navigate through the mathematical problem and rearrange it where necessary for building up his/her own mental map, which is important for understanding of problem-solving concept. The Sub-calculation steps for particular math tasks have been extracted from mathematical schoolbooks. After finishing all Sub-calculations, a blind student will be guided to the Evaluation phase where he/she can revise the solving steps and evaluate the answer by calling the support of the math engine. Furthermore, the last layer provides the summary, which allows a stepwise checking of the result. All of the intermediate steps will be shown, which also includes a 2-dimensional MathML Elementary representation [13] for better cooperation and communication with sighted people.

Figure 3.

Layers of an assistant.

The didactic mathematical problem-solving dialogues and processes are defined based on the mathematical schoolbooks analysis mentioned above. Figure 3 shows the internal layer of an assistant.

The dialogues between an assistant and a blind student are based on invisible relations between mathematical objects embedded in mathematical schoolbooks and will be continued until there is no solving step left. The predetermined sequential didactic solving steps are based on the atomic concept and hence, cannot be interrupted. A blind student has to go through all of the intermediate steps. In the case of a mistake or miscalculation identified by a math engine, he/she can correct it at the end of the procedure. If he/she disrupts the assistant during the sub-calculation, he/she has to restart the assistant from the beginning. Demanding for a strict following of the defined process should support a blind student in developing a mental map and procedural skills.

7.2 Implementation

The prototype is implemented using Eclipse SWT framework with JFace API in the Java programming language. Along with the dialogue of an assistant/wiz- ard concept, all of the control elements are equipped with the Accessible class providing accessibility information to the Microsoft Windows operating system. The significant information for the JAWS screen reader software will be announced with an attribute “name”. This will be read aloud by JAWS. For more information on a control element, other attributes such as “description”, “help”, can be used. This information will not be read aloud by JAWS directly, but in particular situations instead.

Figure 4.

Main window of the prototype.

Figure 5.

Entry Wizard for a long addition calculation.

Figure 6.

(a) Shows the original position of the numbers, and (b) Shows the number shifted to the correct position.

Figure 4 shows the main window of the IDMILE prototype where the students can linearly type in their mathematical problem or select one from the list in the menu. After validating the expression and identifying the type of operation, the proper entry wizard will be proposed, confirmed by the student (for learning purposes), and started. Over time, students and teachers may bypass the wizard by changing configuration in the Settings menu. Figure 5 illustrates a long addition wizard.

In the next step, the numbers are separated into two rows imitating the visual 2-dimensional representation of a mathematical problem-solving step. A blind student can use arrow keys to navigate and readjust the numbers’ position into the correct one. Figure 6a shows the original position of the numbers, and Fig. 6b shows the number shifted to the correct position.

Figure 7.

Sub-calculation for a long addition.

Figure 8.

(a) Current position of sub-calculation, and (b) Context-sensitive help message.

The Fig. 7 illustrates the sub-calculation of a long addition beginning from right to left. The representation of a sub-calculation changes from vertical to horizontal for easier access on a refreshable braille display.

For orientation and reference purpose, a blind student can call up a dialogue that is providing information where she/he currently stands in the solving process. This is useful when the expression numbers get larger and the solving process gets longer. Consequently, the blind students may lose their orientation due to the lack of external memory. Figure 8(a) shows the current position in a long addition sub-calculation. The dialogue displays the complete mathematical assignment with current numbers indicated with squared brackets. This supports a blind student in remembering and avoids memory overload. A context-sensitive, application-wide help is also included. This should support more independent working for blind students. Figure 8b illustrates such a context-sensitive help message dialogue.

After all of the sub-calculations steps are done, a summary of the detailed mathematical problem-solving process can be obtained from the main window. All of the significant information gathered during the sub-calculations is analyzed and represented. A blind student can then track the steps systematically. This refreshing procedure contributes support for better learning effects. In addition, for sighted and partially sighted students, the 2-dimensional representation of the calculation is represented using MathML Elementary Math. This should help improving cooperation and communication in the inclusion classrooms. Figure 9 depicts a summary window.

Figure 9.

Summary window of the software prototype.

8. Testing and evaluation

8.1 Test environment

The platform-specific accessibility concept requires certain prerequisites in order to guarantee the optimal functionality. Since the IDMILE prototype relies on Eclipse SWT framework and Microsoft Windows operating system, the following requirements were defined for the testing setup. Microsoft Windows 10 was used as an operating system because of its up-to-date UI Automation. The resolution of the screen was set to 1366 $\times$ 768 (full resolution is 1920 $\times$ 1080); otherwise, some of the control elements on the software prototype windows cannot be displayed. The combination of JAWS in version 17.0 with Structured Mode [39] with a refreshable braille display is used. There are four braille output modes provided by JAWS: Speech Output, Structured, Line, and Attribute Mode. Structured and Line mode are used for prototype testing. However, the Structured Mode provides descriptive information about the current dialogue and/or control and lets the user navigate faster, whereas the Line Mode displays information as displayed on the screen, including layout and format, which tends to overload. The JAWS braille table has to be set to Europe. Moreover, to prevent JAWS from saying non-labeled control elements as images, the setting of graphics with no verbosity is selected. The refreshable 8-dot braille display ALVA BC640 with 40 characters is used. This test environment ensures that the participants use a comparable working environment for exploring and examining the software prototype.

8.2 Functionality validation

We defined three groups of users: including students, teachers, and experts. The group of students consisted of three blind students and two profoundly visually impaired, all in an age range of 10–12 years old. The group of teachers consisted of one sighted teacher and one blind teacher. The group of experts consisted of two blind experts in computer science and mathematics.

The test procedure was constructed allowing all participants to do the same math tasks, which began by an introduction to the IDMILE software prototype. Thereafter, the participants were requested to choose mathematical problems from a list of prepared tasks, which contained the basic arithmetic operations: addition, subtraction, multiplication, and division. The participants went through the solving steps using our IDMILE prototype. Afterwards they were interviewed and asked to fill out a questionnaire.

None of the students used specific software for learning or doing mathematics before using the IDMILE prototype. They are used to use the mechanical aids such as braille typewriters or traditional text editors such as Microsoft Word, Microsoft Notepad for solving mathematical problems. However, some of the teachers and experts do use computer algebra systems such as Maxima, Mathematica, or MuPAD, to solve more complex math tasks.

The observation and interviews showed that all participants had some difficulties with IDMILE at the beginning, especially to remember the keyboard shortcuts of the software prototype, which is understandable because the IDMILE was new to them. For all of them, however, after a short time and when getting used to the underlying process of problem solving, they could move very quickly to solve problems with large numbers where the guidance and support for avoiding mistakes were highly recommended. Only a few times, some of the participants used the common shortcut F1 to get more information and help. One remarkable behavior, especially for more experienced participants such as teachers or experts, was that they were trying to use the TAB key for navigation, what seems to be a de facto standard in many software interfaces. In our IDMILE, this function is not supported due to the focus management discussed above. Nevertheless, this is to be considered when working on the next version of IDMILE.

Table 1
Percentage of participants’ satisfaction

Criteria	Low	Middle	High
	(%)	(%)	(%)
Students
Simplicity of learning curve to use the prototype	–	40	60
Good and logical keyboard shortcuts	–	20	80
Prototype is better compared to old methods	20	20	60
Supports the computational tasks	40	20	20
Assists the way of solving mathemati- cal problems	–	40	60
Helps to learn mathematics	20	20	60
Teachers
Simplicity of learning curve to use the prototype	–	–	100
Good and logical keyboard shortcuts	–	–	100
Prototype is better compared to old methods	–	100	–
Supports the computational tasks	–	100	–
Assists the way of solving mathemati- cal problems	50	50	–
Helps to learn mathematics	–	50	50
Experts
Simplicity of learning curve to use the prototype	–	50	50
Good and logical keyboard shortcuts	–	–	100
Prototype is better compared to old methods	–	–	100
Supports the computational tasks	50	50	–
Assists the way of solving mathemati- cal problems	–	50	50
Helps to learn mathematics	–	–	100

The mathematical problem-solving procedure introduced in IDMILE was new to all of the participants. Despite that, all of the participants could finish the task list using the guidance and support functionalities of IDMILE relatively quick, especially in the problems with large numbers. The following Table 1 outlines the evaluation results based on the criteria measured by observation and interview/questionnaire.

Table 2

Heuristic evaluation of usability in percentage

Criteria	1 (%)	2 (%)	3 (%)	4 (%)	5 (%)	6 (%)	7 (%)
Students
Simple and natural dialogue	–	–	–	–	–	66.7	33.3
Speak the users’ language	–	–	–	–	33.3	33.3	33.3
Minimize user memory load	–	–	–	–	66.7	–	33.3
Consistency	–	–	–	–	–	66.7	33.3
Feedback	–	–	–	–	–	–	100
Clearly marked exits	–	–	33.3	–	33.3	–	33.3
Shortcuts	–	–	–	–	33.3	–	66.7
Good error messages	–	–	33.3	–	–	–	66.7
Prevent errors	–	–	33.3	–	33.3	33.3	–
Help and documentation	–	–	–	–	–	33.3	66.7
Teachers
Simple and natural dialogue	–	–	–	–	–	50	50
Speak the users’ language	–	–	–	–	–	–	100
Minimize user memory load	–	–	–	–	50	–	50
Consistency	–	–	–	–	50	–	50
Feedback	–	–	–	–	–	50	50
Clearly marked exits	–	–	–	–	–	50	50
Shortcuts	–	–	–	–	50	–	50
Good error messages	–	–	–	–	–	–	100
Prevent errors	–	–	–	–	–	50	50
Help and documentation	–	–	–	–	–	50	50
Experts
Simple and natural dialogue	–	–	–	–	–	50	50
Speak the users’ language	–	–	–	–	–	–	100
Minimize user memory load	–	–	–	–	50	50	–
Consistency	–	–	–	–	50	–	50
Feedback	–	–	–	–	–	–	100
Clearly marked exits	–	–	–	–	–	–	100
Shortcuts	–	–	–	–	–	50	50
Good error messages	–	–	–	–	–	–	100
Prevent errors	–	–	–	–	50	–	50
Help and documentation	–	–	50	–	50	–	–

All of the 3 groups of participants found that the learning curve of the prototype is not high. A new user may have a little difficulty at the beginning, but he/she can learn and understand the software strategy in relative short time due to the simplicity of software structure. Despite the new keyboard shortcuts, more than 80% of all of the participants were satisfied with the interaction principles (80% of students, 100% of teachers, and 100% of experts). The majority of the participants indicated that the prototype supports the way they calculate mathematics, and this should be better and more effective than the methods they are used to. We were in particular pleased that experts and teachers underlined that this is seen as an excellent tool to teach and learn basic mathematics. Even if not used often, it considerably helps to develop a mental map and understanding of the underlying principles.

8.3 Usability evaluation

The usability evaluation was conducted under the concept of heuristic evaluation introduced by Jacob Nielsen [32]. The usability evaluation method has 10 criteria concerning user activities and program’s responses. The scales are between 1 and 7, where 1 means “very poor”, and 7 means “very good”. Table 2 shows the collected data from the participants in percentage.

The overview of Table 2 shows that the majority of the participants found that the IDMILE prototype is quite user-friendly. Most of the scores are between 5 and 7, in particular related to “Simple and Natural Dialogue”, “Feedback”, “Shortcuts”, and “Good Error Messages”. The criteria “Minimize User Memory Load” and “Help and Documentation” indicate where a next version should be improved in terms of intuitiveness. The criterion “Minimize User Memory Load” is one of the key targets in our research attempt developing IDMILE prototype, as it should allow and motivate the users to switch from traditional math working environments such as braille typewriter or text editors towards more guidance and supporting assistant. In particularly, to benefit from understanding the mathematical principles and avoid doing mistakes caused by the lack of overview and orientation when working with sequential audio or braille.

The evaluation clearly outlined where to be improved but also underlined that our work is a fruitful and promising R&D, which is in line with the movement toward ICT-based mathematics, which is also seen in the mainstream.

9. Conclusion and discussion

The concept of transferring visual implicit didactic information from mathematical schoolbooks and teaching techniques used in schools for blind students to an accessible software tool for blind students is successfully done. Unfortunately, not all of the visual implicit didactic information shown in the mathematical schoolbooks can be converted for blind students who use assistive technology. This is because of, on one hand, the modesty of the current assistive technology itself. On the other hand, until now there was no suitable software tool enabling visual implicit didactic information conversion for blind students. Nevertheless, the most important didactics – invisible relations and navigation in spatial math structure – could be analyzed, modelled, and implemented.

According to the functionality evaluation above, the IDMILE prototype could guide the blind participants through the mathematical problem solving process successfully. The visual implicit didactic information, especially the spatial one that effects significantly on learning ability of the students, was analyzed and constructed for assistive technology explicitly. The used assistive technologies, namely a screen reader and a refreshable braille display, compensate the use of pen and paper as a tool used by sighted students. In addition, during the solving process, the IDMILE prototype helped the participants to carry over their memory to the software, so that the memory overload of the participants was prevented. One remarkable observation is that the way the participants are used to solve a mathematical problem may be slightly different from the methods used in the mathematical schoolbooks. However, they managed to learn using the IDMILE in relatively short time.

Unfortunately, the Accessibility Technology is still a proprietary issue, despite standardizations set up by different leading organizations. This necessitates the determination of to-be-used technology at the beginning, and certainly, this affects only some of the particular groups of blind people. In this case, we developed the IDMILE prototype that can only run on Microsoft Windows operating system effectively, although the IDMILE itself was written in Java. On other operating systems, the benefit of UI Automation, which is the Accessibility API from Microsoft Windows, cannot be provided. Moreover, the settings of assistive technology such as JAWS screen reader can affect the representation behavior of the IDMILE on a refreshable braille display such that one Mode could be suitable for a particular situation, but another Mode may be not.

To obtain more information about converting implicit didactic information in mathematical schoolbooks to assistive technology, a long-term research should be done. The observation of blind students’ behavior during the mathematical problem-solving process will deliver valued information. Furthermore, the extension of the functionalities of the IDMILE prototype will expand the research horizon, and bring the prototype to the level of a ready-to-be-used tool for blind students in the future.

Footnotes

Conflict of interest

None to report.

References

Jitngernmadan

Miesenberger

. A comparative study on java technologies for focus and cursor handling in accessible dynamic interactions. Studies in Health Technology and Informatics. 2015; 217: 267-273.

Jitngernmadan

Petz

Stöger

Miesenberger

. Analysis of implicit didactics in math schoolbooks for interactive non-visual user interface development. In Computers helping people with special needs. 15th International Conference, ICCHP 2016, Linz, Austria, July 13-15, 2016, proceedings/Klaus Miesenberger, Christian Bühler, Petr Penaz (Eds.) Miesenberger

Bühler

Penaz

, eds. LNCS sublibrary: SL 3 – Information systems and application, incl. Internet/Web and HCI 9758-9759. Springer International Publishing, Cham, 2016; 19-26. DOI=10.1007/978-3-319-41264-1_3.

Stöger

Miesenberger

. Accessing and dealing with mathematics as a blind individual: State of the art and challenges. Enabling Access for Persons with Visual Impairment. 2015; 199-203.

Aldridge

Kappel

Samland

. Das System der Mathematikschrift in der deutschen Brailleschrift: Nach den Beschlüssen vom 30.11.2013 in Basel. SBS, 2015. [Zürich].

Gschaider-Kraner

Hanneman

Mühlbach

Ruemer

Schmid

Stanetty

Weingartner

. Änderungen in den Mathematikbüchern bei Neuerfassung ab dem Schuljahr 2015/16.

McCracken

Nemeth

Roberts

. The Nemeth Braille Code for Mathematics and Science Notation: 1972 Revision. American Printing House for the Blind, Kentucky, 1972.

Wolfram. Handwritten math recognition in Windows 7. Wolfram language documentation. 2017. http://reference. wolfram.com/language/tutorial/HandwrittenMathRecogni- tion.html. Accessed 20 January 2017.

Hegarty

Kozhevnikov

. Types of visual-spatial representations and mathematical problem solving. Journal of Educational Psychology. 1999; 91(4): 684-689.

Presley

D’Andrea

. Assistive technology for students who are blind or visually impaired. A Guide to Assessment. AFB Press, New York NY, 2008.

10.

Hersh

Johnson

, eds. Assistive technology for visually impaired and blind people. Springer London, London, 2008.

11.

Manduchi

Kurniawan

. Assistive technology for blindness and low vision. Rehabilitation science in practice series. CRC Press Taylor & Francis Group, Boca Raton, 2013.

12.

Batušić

Burger

Miesenberger

Stöger

Kepler

. Access to mathematics for the blind – defining HrTEX standard. In Proceedings of the 5th International Conference on Computers Helping People with Special Needs. 1996; Part II: 609-616.

13.

W3C. 2016. MathML. https://www.w3.org/Math/. Accessed 18 October 2016.

14.

openMath. 2016. OpenMath. http://www.openmath.org/. Accessed 20 January 2017.

15.

Freedom Scientific, I. JAWS: Job Access With Speech. Freedom Scientific, Inc, 2015.

16.

Curran

Teh

. NVDA (NonVisual Desktop Access) NV Access, Australia, 2017.

17.

Apple. Accessibility – VoiceOver. http://www.apple.com/accessibility/osx/voiceover/. Accessed 18 October 2016.

18.

Gardner

. The LEAN Math Accessible MathML Editor. In Computers Helping People with Special Needs Miesenberger

Fels

Archambault

Peňáz

Zagler

, Eds. Lecture Notes in Computer Science. Springer International Publishing. 2014; 580-587. DOI=10.1007/978-3-319-08596-8_90.

19.

Batusic

Mühlbach

Pichler

Schmid

Stanetty

. Aufbereitungsrichtlinien für barrierefreie Schulbücher, 2011.

20.

Net

. InftyReader. Science Accessibility Net (sAccessNet), 2016.

21.

Gardner

. Viewplus – A brief history and background. Empowering people worldwide. ViewPlus. https://viewplus. com/about/. Accessed 17 October 2016.

22.

Schönherr

K-P

. Handy Tech – HandyTech.de. https:// handytech.de/en/info/handy-tech. Accessed 17 October 2016.

23.

German Federal Ministry of Economics and Technology. HyperBraille. http://www.hyperbraille.de/?lang=en. Accessed 17 October 2016.

24.

Gillan

Barraza

Karshmer

Pazuchanics

. Cognitive analysis of equation reading: Application to the development of the math genie. In Computers Helping People with Special Needs, Miesenberger K, Klaus J, Zagler WL, Burger D, eds. Lecture Notes in Computer Science. Springer Berlin Heidelberg. 2004; 630-637.

25.

Raman

. ASTER – Towards modality-independent electronic documents. In Electronic Multimedia Publishing Makedon

Rebelsky

, Eds. Springer US. 1998; 53-63.

26.

MICOLE Project: Multimodal Collaboration Environment for Inclusion of Visually Impaired Children. http://micole.cs. uta.fi/, 2012.

27.

Karshmer

. MathOMatic Blocks: An Automated, Tactile, Interactive Method of Teaching Mathematics to Blind Students in the K-12 Environment. Business Analytics and Information Systems. 2007.

28.

Schweikhardt

Bernareggi

Jessel

Encelle

Gut

. LAMBDA: A European system to access mathematics with braille and audio synthesis. In Computers helping people with special needs. 10th international conference, ICCHP 2006, Linz, Austria, July 2006: proceedings/Klaus Miesenberger [et al.] (eds.), K. Miesenberger, Ed. Lecture notes in computer science, 0302-9743 4061. Springer, Berlin, 1223-1230. DOI=10.1007/11788713_176.

29.

Fajardo-Flores

Archambault

. Understanding Algebraic Manipulation: Analysis of the Actions of Sighted and Non-sighted Students. 2012; 97-105.

30.

Flores

Archambault

. Multimodal interface for working with algebra: Interaction between the sighted and the non sighted. In Computers Helping People with Special Needs Miesenberger

Fels

Archambault

Peňáz

Zagler

, Eds. Lecture Notes in Computer Science. Springer International Publishing, 2014; 606-613.

31.

Suen

Ary

. Analyzing quantitative behavioral observation data. L. Erlbaum Associates, Hillsdale, N.J., 1989.

32.

Nielsen

. How to conduct a heuristic evaluation. http:// www.nngroup.com/articles/how-to-conduct-a-heuristic-evaluation/. Accessed 1 April 2015.

33.

Freedom Scientific, I. JAWS – Help – Defining Structured Mode. CHM, 2016.

34.

Microsoft. Windows Automation API (Windows). https:// msdn.microsoft.com/en-us/library/windows/desktop/ff48637 5(v=vs.85).aspx. Accessed 17 October 2016.

35.

Ubuntu. Accessibility. https://help.ubuntu.com/community/ Accessibility. Accessed 18 October 2016.

36.

IBM Corporation. 2012. Creating Accessible Applications in Eclipse SWT. https://www-03.ibm.com/able/education/ creating_accessible_eclipse_apps.html. Accessed 18 October 2016.

37.

Oracle. Enabling and Testing Java Access Bridge. https:// docs.oracle.com/javase/8/docs/technotes/guides/access/ enable_and_test.html. Accessed 1 April 2015.

38.

Oracle. Java SE Desktop Accessibility – Java Access Bridge. http://www.oracle.com/technetwork/java/javase/tech/index-jsp-136191.html. Accessed 18 October 2016.

39.

Freedom Scientific, I. 2014. JAWS – Speech History Mode. http://doccenter.freedomscientific.com/doccenter/doccenter/ rs25c51746a0cc/2014_12_3_using_jaws_speech_history/02_ Speech_History_Mode.htm. Accessed 16 November 2016.

IDMILE: An interactive didactic math inclusion learning environment for blind students

Abstract

Keywords

1. Introduction

2. Problem and research definition

2.1 Presentation

2.2 Navigation

2.3 Manipulation

3. State of the art in non-visual math education

4. State of the art in assistive technology for accessing and doing math

5.1 Blind students’ keyboard usage behaviors

5.2 Blind students’ braille display usage behaviors

5.3 Implicit didactic information in mathematical schoolbooks

6. Non-visual user interface requirements

6.1 Manageable focus

6.2 Operating system accessibility API

6.3 Different types of widgets

6.4 Supporting APIs

6.5 Control elements

6.6 Controlling screen reader mode

7. Prototype

7.1 Modeling

8.1 Test environment

8.2 Functionality validation

Table 1 Percentage of participants’ satisfaction

9. Conclusion and discussion

Footnotes

Conflict of interest

References

Table 1
Percentage of participants’ satisfaction