Abstract
BACKGROUND:
Technology is widely used in school to assist students with visual impairments. Drawing with traditional methods is still difficult for those users. In this paper, we present and discuss an evaluation of a haptic and sound interactive drawing program (HIPP).
OBJECTIVE:
The aim of the study was to identify what aspects of the technological aid support the drawing skill development in children with visual impairment.
METHODS:
Interviews, observations and video recordings of use situations were used for data gathering. Our analysis is based on cultural-historical activity theory, and examines the mediation between the child and the object of their activity, their drawings, as well as the roles of teachers, classmates, assistants, and family.
RESULTS:
The haptic and audio drawing program supports the steps (doodling, interpretation, planning, and more intentional drawing) observed in visually-acquired drawing, although possible improvements have also been identified. Observations of the drawing program in use showed that its mix of dynamic multimodal interaction and a stable drawing feedback, enable visually impaired students and sighted teachers to jointly access a shared representation.
CONCLUSIONS:
Successful long-term use, together with the presented results of our analysis show how multimodal dynamic and stable interaction can successfully support drawing activities.
Keywords
Introduction
Actively using images and being able to produce drawings may seem an un-important skill for a person who is blind. However, living in a world where most people can see and have a relation to images creates a need to understand and be able to handle various kinds of graphics. Thus, tactile image interpretation and the creation of images are expected to be learnt in school. Tactile images are surrounded by an extensive body of guidelines, tools, and techniques for creating and interpreting them. In many cases, tactile images are made by seeing persons for the pupils who are blind. This places blind pupils in a passive role, as a recipient and consumer of what others have made available to them rather than as an active creators of images and graphics. Supporting blind pupils in creating images needs to be prioritized for two important reasons: 1. The student should be able to feel that they are doing the same tasks as the other pupils in class, 2. the creation of images by themselves helps the understanding of how images are made and understood. This is supported by Eriksson [1, pp. 15–17], who highlights the relevance of images in pedagogical settings, also for students with visual impairments.
Drawing for children with visual disabilities has been supported with several types of aides. They rely on tactile feedback but most lack the stroke movement typically involved in drawing. Several high-tech solutions exist, but few of them have been evaluated more extensively in a school environment. This article presents the evaluation of a technology (the HIPP system, see Section 1.2) that supports drawing in a dynamic way, and of its use in school environments by visually impaired children. This evaluation focuses on the technological aspects, and, by use of Activity Theory (see Section 1.3), aims to highlight the positive features and limitations of the technology through the lens of the whole activity – including the users themselves and the people around them, as well as the situations of use. The project, including the pedagogical partners e.g. Björk [2], focused on the activity involving children-led drawing and situations of learning mediated by HIPP. The objective of this study is to identify how the technological features of HIPP enable or hinder mediation of those activities.
Related work
Drawing and understanding tactile images is indeed important for pupils who are blind. Kennedy [3] argues that outline drawings are the most relevant, and goes on to describe several ways that visually impaired people employ to represent the world in drawings. His argumentation analyses the different ways the world is perceived depending on the senses used to perceive it, and shows how tactile drawings can be used to represent the world in similar ways that are used with visual drawings – as well as how the tactile images can also differ or put more emphasis on certain types of representations, like outlines. There are ways to print out relief images, but also to create them. Swell paper is often used for printing images that can be later explored in a tactile way – the darker lines and areas are raised under heat [1].
The problem of erasing, the difficulty of understanding and creating drawings, as well as the general move to digital drawing tools also for sighted children, have spurred researchers to create different variants of prototype drawing tools. Kurze [4] presents an early study that uses pen on swell paper with digital annotations. Kamel [5, 6] describes a CAD-like application where users can create drawings with the help of audio and keyboard. This is accomplished by a structured approach: dividing a drawing into subparts and enable the user to draw small segments of it. Unlike the work of Kurze [4] there is no drawing movement of the hand, but the drawing is constructed with keyboard commands.
On the topic of how people with visual impairment themselves can create an image more directly, without relying on others, there are few references – in fact Kennedy [3] is one of the few scientific sources which describe this kind of image creation. Here are three different ways of creating tactile images without computers:
One way to create lines is to press hard on a plastic pocket that is put around a paper. That pocket will then retain the line in relief allowing quick drawings. A disadvantage of this type of drawing is that a lot of force is required to draw a line; one has to push hard with the pen. The pressure required means that each line takes more time and often has to be planned. Another way to create tactile drawings involves a special wax thread that one can stick or glue to the paper. It also requires some planning as it takes time to create a line. A third way to create tactile drawings is to use so called “sponge paper” or “QuickDraw” paper. It requires felt pens and creates a raised area where the felt pen has drawn a line, by absorbing the ink.
The advantages of these techniques are that they are relatively simple to do and that it is possible to feel the result with both hands simultaneously. The disadvantage common to all those techniques as well as most non-digital drawing is the difficulty or impossibility to erase a line.
With the development of force-feedback devices c.f. [7], haptic feedback in drawing and exploring both 2D and 3D digital objects was made possible. Several different applications for school use have been developed, e.g. [8, 9]. Despite this, few have attempted to create a full drawing application with this technology that is more than a proof-of-concept or a technical feasibility test. Early work by Sjöström and Rassmus-Gröhn [10] and Hansson [11] attempted making a drawing program with several colours represented by depth or textures. The activity available in this prototype could be compared to drawing with crayons, and the design was more focused on colour and filling areas. Software for outline drawings more alike traditional tactile images are described in Rassmus-Gröhn et al. [12, 13, 14]. The HIPP system used for this qualitative study is a further development of the software in [12, 13, 14], and compared to the earlier prototypes, it is more adapted to the pupil’s own computer and the other aids installed in it (e.g. screen readers), thus being closer to a real product.
The HIPP system has previously been briefly described, together with some preliminary results, in Rassmus-Gröhn et al. [15, 16]. Pedagogical approaches with methods and guidelines of how to use the HIPP system in practice when working with pupils has been presented in Björk et al. [2, 17].
Below is a short description of the technical functionality of HIPP. The software itself has been made available on the HIPP website [15]. The haptic feedback was achieved with a force feedback device called the PHANToM [7].
The HIPP drawing program resembles a haptic variant of a simple visual drawing program (e.g. Microsoft Paint). The program works together with a haptic pen (the PHANToM), and is implemented to enable the drawing and feeling of curves on a virtual 2D surface (called the virtual paper).
HIPP allows direct drawing – it is possible to create lines by pressing a button on the pen and putting the pen in contact with the virtual surface. Drawn lines can be felt right away – a dynamic feedback of the action conducted, or the line drawn. This feedback is automatically generated. Additional automatic audio feedback is provided when the pen comes in contact with the curve. This dynamic/immediate and automatic feedback provides a repeatability so that it is possible to explore the drawing many times and always get the same feedback over time.
When drawn, the curve is immediately displayed on the visual screen, while simultaneously being engraved on the virtual paper. Depending on the sensory abilities of the users, it can thus be simultaneously seen and felt through the haptic pen. The movements of the hand, via the pen, controls where the point of contact is on the virtual paper, mimicking a real pen-on-paper interaction. It is possible to annotate each curve with text (read by the screen reader) and it is also possible to add an audio file. The user can manipulate the curves in different ways: strokes can be made thinner or thicker; there is a choice between engraved or raised lines; predefined shapes like circles, rectangles or straight lines are available; shapes can be filled; shapes can be copied, pasted and erased; shapes can be moved, scaled up or scaled down. The commands are accessible via a standard menu, which also includes “undo” actions and provides compatibility with screen readers. In addition to the line drawing, HIPP is able to import PNG files of a predefined size. An imported greyscale image will be felt as a relief where the depth depends on the darkness of the corresponding pixel. Images created with the HIPP program can be printed. If swell paper is used in combination with a swelling machine, the printed image can also be made tactile. This functionality provides another permanent and stable representation of the drawing that can be explored by both hands.
The HIPP software used in the studies is open source and available freely. It is written in C
Theoretical framework
The study takes CHAT – Cultural Historical Activity Theory – as its main theoretical framework. It is a theory rooted in the beginning of the twentieth century Marxist Russia. Vygotsky formulated a psychological theory that the development of the mind was supported by the person’s activities over time interacting with people and artefacts that also carry a cultural meaning [19]. He mentioned a specific perspective that is relevant in education: the student can do more with help than alone. This zone of proximal development is relevant to both classical education [20] and special needs education [21]. The technological side of activity theory has been utilized in relation to Human-Computer Interaction and shown useful by Kaptelinin and Nardi [22]. The activity diamond [23] puts in relation the person acting, what they are doing, their goal, as well as their environment, both artefactual and human. It was elaborated within the perspective of rehabilitation engineering and lived disability, and enables one to encompass the many parameters that enable or hinder an activity.
Method
In Sweden, children with visual impairment are integrated in a class with sighted peers. This means that some adaptations are made to let all children follow the courses at the same time. During the HIPP study, visits to the schools were made and gave the researchers a clear understanding of the human and artefactual environment in the child’s learning activity. The observations are based mainly on a weeklong study visit in one of the locations.
The study presented here is part of a project that ran for several years, in which the HIPP application was gradually developed in close collaboration with pedagogical personnel, pupils, and experts on computer aids for persons with visual impairment. The entire project had a strong foundation in user-centred design [12, 24, 25] which focuses on user involvement, iterative design and evaluations. The strongest user involvement has been in iterative testing and evaluation of HIPP, and users’ ideas and suggestions for improvement have been the main driving force behind the iterative improvement of the software.
The development of the program was conducted in parallel with the trials in school. At the beginning, an older version of the program was used. The program was then modified incrementally in an iterative process. Regular meetings were held between the teachers and the researchers involved in the project and priority lists were created with all actors involved to guide what should be developed next. This created a back and forth between coders and users, allowing functionality to be built concurrent with their involvement.
During the project, the use and uses of HIPP have been documented in several ways. Pedagogic material created by experts and teachers have been collected, some drawings made by pupils have been gathered, videos were made where pupils are seen using the system, and finally, interviews have been carried out with everyone involved. Several on-site visits have been carried out in the schools. This paper is mainly based on a longer study visit, where one of the researchers was present in school for a whole week.
Figure 1 shows the chronology of this process. The concurrent development and use period is called “the project” and lasted several years. The weeklong field-study happened during one week toward the end of the project period. It is designated as “the study”. This article focuses on the study.
The chronology of the work process.
List of interviews
The design of the study and our choice of methods were guided by CHAT. The activity checklist [26] and the Activity Diamond [23] were the two tools used that relate to this theory. The checklist provided us with a number of questions and the diamond helped us to articulate the different elements that are relevant (see Fig. 3). Before the week on-site, the project included regular meetings between the teachers involved with the users in different schools across Sweden, as well as the researchers that developed the program. Those meetings allowed us to gather practical information on the situation on-site and gave an idea of what to expect. With this in mind, the questions of the activity checklist and the diamond were applied to the school situation, considering the use of HIPP in specific drawing sessions as well as in-class.
Another perspective that was brought up during the preparation of interview questions and observation guidelines was the one of borders. The concept, as studied in Sandberg [27], can be seen as a separation that is appearing and disappearing. In Sweden, children with visual impairment go to school with children without visual impairment. One of the goals with this is to integrate and to avoid separating the children with visual impairment from the rest of society. One of our goals in the choice of methods was to study how this type of separation is manifest -- or not – in the lived reality [23]. In particular, we chose to study artefacts in use in real environments [28, 29]. Identifying the barriers between the HIPP user and the rest of the class was a good way to assess whether the use of the HIPP drawing program supported the integration goal. The physical borders as well as the time-related ones were thus included more prominently in the interview and observation guidelines.
With these perspectives in mind, the list of interview questions was created to cover expected topics while still allowing room for unexpected topics. This led to the choice of a semi-structured interview questionnaire as a guide to the interviews, Kvale [30].
The initial version of the HIPP program and hardware were the only well-known elements from the start, and the methods were chosen to understand how it did or did not mediate or integrate with its surrounding activities. It must be mentioned that another goal/focus from the week was to bring the technician (not only the researcher) closer to the users, to allow for more direct problem-to-solution moments.
Participants
We had two students with visual impairment in the study: Sam and Kim (fictional names for anonymization). They went to two different primary schools in the same Swedish town. Kim’s teacher was her personal assistant and will be referred to as Kim’s individual teacher. Sam’s assistant followed Sam in school, while Sam’s teacher also taught Sam’s classmates. Within the HIPP project, pedagogues came to Kim and Sam once a week for drawing sessions; they are referred to as Sam’s pedagogue and Kim’s pedagogue, having respectively the main direct contact with Sam and Kim. Other people involved were a teacher assistant to Sam’s teacher, and additional special pedagogues at the school.
During the HIPP study, visits to the schools were made and gave the researchers an understanding of the human and artefactual environment of the child’s learning activity. The following general observations on the human environment are based mainly on a weeklong study visit in one of the locations:
Around the child were obviously the other classmates. The teacher and sometimes assistants are organizing the lessons. A child with visual impairment will also be supported by a personal assistant. This assistant will ensure that specific needed adaptations are made, while doing for example, a great amount of audio interpretation (“syntolkning” in Swedish, see Section 3.3). People at the child’s home are also important. The family often takes part in and supports learning (and other) activities outside of the school.
During the study, the researcher brought pre-printed questionnaires as well as paper notebooks in order to take field notes. A voice recorder for the interviews and a camera in the form of the researcher’s phone were also used. This resulted in a number of physical paper notes for each interview as well as observation notes in the notebook.
Audio recordings, pictures, and notes were gathered during the week. Although the study focused on the week on-site, in order to provide a more complete picture our material was complemented by videos taken by the local participants over a period of one or two years of use.
Digital copies of the recordings could be considered part of the material traces of this study. They were the basis for the transcriptions that were then printed and used for analysis. NVivo [31] and digital data sheets were also used to help organize the analysis.
Interviews and observations
We followed the guidelines of Kvale [30] when planning and handling the interviews. The interviews were conducted with as many of the involved persons as possible, and they followed the prepared questions and format whenever possible. In one case, several of the teachers could dedicate one hour for an interview all at the same time and then had to go. We thus decided to change the format to a focus group, while all other interviews were conducted individually. All interviews were audio recorded and transcribed. The places where the interviews took place varied. When possible, the place was a calm room but still within the school. Some short interviews were done in context by the pedagogues and taken on video recording. The longer interviews amount to a number of six as described in Table 1.
A week on site was scheduled in order to arrange interviews and observation. During the observations, the researcher acted as “observer-as-participant” (Angrosino [32, p. 54]): The usual school activities were conducted and the researcher was allowed to be present and take notes on the side, as well as interact without disturbing the activity. Video recordings of previous drawing sessions were also provided to the researcher afterwards to complete the on-site observation.
Table 2 presents the list of conducted observations. When noted “whole classroom”, it includes the student in focus, their classmates, their assistant, and their teacher. One lesson was between 40 and 80 minutes in duration.
List of observations
List of observations
At the beginning of this study, an application was sent to the regional ethical review board at Lund University for evaluation. It was deemed that the study did not need an ethical board approval and could continue further. However, we still considered the ethical implications both before and after the study – see more in the discussion paragraph 3.7, notably on the role of the researcher in the setting. Informed consents were gathered, and alternative names or roles have been used to refer to the participants.
Analysis
The project had a concurrent timing mixed methods design with qualitative emphasis [33]. The Activity Diamond [23] as well as the questions of the activity checklist [26] were used as analytical constructs to organize the data obtained through the methods described above (videos clips, transcriptions, observation logs). Answers from the interviews were used to inform the observations and vice-versa: the data was consolidated by triangulation.
Since the focus was mainly on the technical mediation, special attention was paid to the features of HIPP when they were mentioned in the data. Each time a feature was used or mentioned, its place within the activity (a described activity or an ongoing activity) was examined, either within the same piece of data or informed by other sources.
According to Kuuti, p26 in [34], the smallest possible unit of analysis is the activity. Whenever considering any piece of the gathered data, it was thus necessary to relate the piece of data to the other parts of the activity within which the same data was involved. The Activity Diamond [23] provided four categories as well as the link between them to expand on when relating bits of data to each other (see Fig. 2). The activity checklist [26] also offered guidance with its four columns as follows:
Means/ends (hierarchical structure of activity), Environment (object-orientedness), Learning/cognition/articulation (externalization/ internalization), and Development (over time).
The activity diamond, smallest unit of analysis.
Possible simple representation of HIPP use in the activity diamond.
Plastic bears of different sizes.
Six-items pattern repeated with a “plastic row” tool.
To illustrate our process, we provide two examples of how this was done in practice. In our first example, we interviewed the teachers about the use of the “import” function of the HIPP program to create new material to be used with HIPP using digital images. When considering the activity of creating this material, it was necessary to take into account that the teachers had as an internalized tool the knowledge from a previous course in digital images. This was made explicit by inquiring about the detail of how the teachers acted when using images in class. This tool was then related to the interface provided by HIPP, and it later was bundled with other bits of data (other instances when the more standard interface mattered in use) to support the assertion that the standardization of HIPP’s interface facilitated its use as a mediating tool in the drawing activity.
Another example is when we observed the use of HIPP in class. The material environment was documented (Fig. 10) and the student of interest (subject) as well as the teachers and assistants (human environment) were interviewed. The internalized tools were shown more evidently in a subsequent session in the afternoon, where more time was dedicated to understand what the HIPP user could or could not follow. The observation helped to understand the link between the different parts of the activity in order to produce the school’s purpose. It also gave insight into the interaction with other classmates, and in how the desk’s organization and assistant’s placement enabled or hindered it. The noise level, being close to a classmate, having the same material prepared in an adapted way: All of these influenced how the activity was enabled or hindered.
In this section, we will discuss how the HIPP system and the research process around it supported drawing for children with visual impairments in real use situations. As a introduction to the results, we can quote from a dialogue between Sam and his pedagogues (translation from the report “Jag kan rita” [17] by the authors):
The pedagogues asked Sam if they wanted to draw but they answered: “Then I must say no, because I can’t draw because I can’t see.” After some persuasion and trials with HIPP, Sam noted: “I am the world’s best in drawing” Sam had drawn 18 drawings and the pedagogues noted that no one else had done more drawings at the school that day.
This quote shows that the HIPP system successfully mediated the drawing activity for Sam. A number of more detailed conclusions made by the pedagogues can be found in both the project’s report [17] as well as one of the pedagogues’ essay [2]. Of course, the fact that the HIPP program needed a specific combination of hardware and software to fully work, impacted the way in which the technological system mediated the activity. In this article, we will keep a focus on the technical details of the system and of the use, and how they have been mediating, and enabling the different activities identified.
The user engaging in HIPP.
A pedagogue’s hand helps the child with the use of the haptic device.
A photograph of the environment – the table and the HIPP hardware at school.
A number of drawings produced in HIPP and then printed out.
Desk equipped at school.
The black swelling machine is kept out of the way.
HIPP drawings included in a class activity about sea animals.
Figures 6 to 9 show several photographs taken during the HIPP project, selected as illustrative excerpts of the activities. They show a HIPP user, their assistants, their school, and their drawings. These excerpts can be placed relative to the Activity Diamond. For example, the user engaging in HIPP Fig. 6 would be on the left of Fig. 2. On the bottom of Fig. 2, we could place the picture of a direct help situation (Fig. 7), where a pedagogue’s hand helps the child with the use of the haptic device. On the top part the Activity Diamond, we could place a photograph of the environment (Fig. 8) – the table and the HIPP hardware at school (more on Figs 10 and 11). On the right side of Fig. 2, we could place a number of drawings produced in HIPP and then printed out (Fig. 9). They are representing both the material production of drawings but also some of the joy of having produced them or – as in Fig. 12 – to have one’s drawing included in a class-project. This activity description is the basic result of the analysis, which yielded several more detailed results and points of discussion. In the following sub-sections, we present and discuss our results relating to:
Drawing and context Learning and drawing activities: from perception to production Dynamic drawing, stable shared representation Haptic mediation Material/external outcomes Teacher’s activities
In each section, we discuss the role of HIPP with a focus on how the program and features played a role in the observed results. In our final sub-section “coder and researcher” we reflect on our own role in the activity and the process. Each title represents a theme that emerged after having created activity diamonds from the data available.
In this section, several of the identified activities are presented. This result is linked to the observations and interviews and examples of how and what HIPP was used for are presented. It will present both the context within which HIPP was used (drawing in context), as well as showing that the object of the drawings was to be found outside of HIPP itself (drawing on/about the context). Identifying the elements of the activity is a prerequisite to understanding how HIPP mediates it.
User-led activity, child’s motive in focus
The importance of the child’s initiative was emphasized in our meetings with the pedagogical staff. By letting the child take the initiative, choose what to draw for example, the learning goal within the different activities taking place had a higher chance of being effective for the child. This is explained more in detail in Björk [2] and in Fahlström and Björk [17].
The main user of the HIPP system is a child with visual impairment. In our study, two children (primary school) have used HIPP and one of them was able to share more deeply about their perspective through an interview. The pedagogues also worked with an approach that placed the children’s own goals at the centre (see Björk [2]). The system was introduced gradually, to match the abilities of the child. HIPP was first introduced without using the keyboard, relying on support from the pedagogues. The use of the keyboard was only introduced after a while, when the child was ready for learning this skill.
The child’s perspective includes all of their lives. They have a life outside of school and it was elicited during the drawing sessions by making the weekend events the theme for the drawing – the class started the week with the writing of a weekend diary, and HIPP drawing was introduced as a regular activity to complement the writing. The child’s interests both in and out of school are relevant to HIPP use, as were their wishes and capabilities.
A lengthy interview of one of the users gave insight into this user’s motives. The drawing activity was something novel because of how difficult it had previously been to draw with non-digital tools. The user’s interest for technology as well as for the “drawing production” in itself surfaced. We could also observe that users seemed very engaged in the sounds used during the drawing activity (HIPP allows the association of a sound file to a curve). Recounting events that had happened in the outside life through the drawing sessions, both our participants engaged in re-telling/re-creating parts of their own life during the drawing activity. For example, the drawing sessions for Sam referred to out of school activities such as eating popcorn at grandma’s, cycling with the family, or gardening with a parent. This can be described as drawing on or about the user’s context.
HIPP in the context of the physical environment
The children were not only surrounded by people. The physical environments of the schools were of importance and affected the activities conducted there. Artefacts and the school’s structure played a role too. The physical context is also one they were “in”. Several pieces of equipment were often present in schools around the child with visual impairment: A Perkins machine to write braille, a computer where some of the material could be spoken out, a swelling machine used to produce swell-paper drawings etc. All this took space around the child. In the case of HIPP use, we could especially note the amount of devices (technological or not) that cluttered the desk (Fig. 10) of the child with a visual impairment. Sometimes the problem was solved by keeping the biggest pieces of equipment away (Fig. 11) from the desk except when actually using it.
Sometimes, to focus on specific tasks, parts of the lessons were done in a separate room. The child would then have had to move away from the well-known space where all the usual equipment resides. This also happened for several of the school topics, e.g. sport, woodworking, sewing, music. During the HIPP project, the program was installed on Sam’s classroom computer. The HIPP program could also be “brought to the school” by the pedagogues who would bring the robot with them and let the child draw during a drawing session. The different ways of using HIPP implied different timeframes for the usage. It was different to use HIPP “on the fly” to illustrate a concept during the teacher’s lecture in common with all other students, than to get a separate “drawing session” away from the main lesson schedule.
In both cases, the clutter or the lack of access to the equipment could have an impact on the activity. For example, clutter might have made it more difficult to integrate with the rest of the class when the class was organized differently. Alternatively, keeping an equipment in another room might have made it more likely to forget about its availability in the middle of an activity. It was also harder to include HIPP use when class happened outside of the classroom.
HIPP in the context of other activities
What about HIPP in relation to other activities not mainly focused on drawing? The system in Sweden is to have an integrated education for all children as much as possible. The children who have visual impairments are in the same local schools as children without visual impairments and get a specific help to follow school as similarly as the rest of their classmates. In this context, the concern that the HIPP system’s use might be isolating was raised. We have seen that different contexts and uses of the program might lead to different results.
It is also to be noted that in general children with visual impairments have already many additional activities in their school schedule compared to the other pupils. Learning Braille is one example. HIPP could be considered as “one more thing” and, as any new technology/software introduced, it did take time to learn its use, especially at first. Here are some more detailed accounts that illustrate different takes on this question in the reality of our project.
In the visited school, the HIPP program was prominently used to produce drawings that then got used in similar ways that drawings might be used by the non-visually-impaired classmates. For example, one of the drawings was chosen to be showed to the child’s family and sent as a letter to the child’s grandmother. If we take CHAT as a support for analysis here, we can notice that this specific activity highlights the importance of the human environment, here the family.
Another example was the use of HIPP to produce a drawing of sea animals. The class activity resulted in a huge wall display (Fig. 12) composed of every pupil’s contribution on a sea background. When we analyse in terms of activity, this specific use of HIPP could be mediating the object of participating in a class project, with a possible outcome of a feeling of belonging to the class.
Using HIPP during class was also practiced. However, that use was constrained by outside restrictions. For example, Sam’s assistant was able to make use of HIPP during the observation in day 2, but lacked time to prepare and let Sam explore all the material available to the sighted students. This was remedied by an extra session in a separate room later in the day.
In other occasions, the use of HIPP was impossible or limited to preparation exercises. During the week on site, the observer followed the observed pupil out of class both during lunchtime and during a sports lesson. Those two instances involved crossing the schoolyard as well as navigating the cafeteria in one instance, and participating in the physical activity in the other.
It was not possible to use HIPP at all during those times, mainly because of the limited portability of the setup. HIPP had however been used previously, in exercises involving a map of the schoolyard. The exploration of the map in HIPP has been done with cane exploration in mind, and is useful in the school activities described just above.
HIPP features identified as relevant
To focus on the HIPP features relevant regarding the fact that drawing had more aims than just the act of drawing, we can emphasize the importance in HIPP of:
the system’s portability in different settings (helped by the use of portable computers, but still limited by the space required by the haptic device and additional devices (printer, swelling machine, speakers). the direct print possibility of the system onto swelling paper, which enables black and white as well as relief drawings to be produced from the HIPP system in one shortcut.
HIPP is meant to support the development of 2D representations in both perception and production. Use and production of 2D representations is seen in the school’s curriculum (see [35, p. 22]) as a useful skill for many areas of life. Nevertheless, in this case, when we consider the child to be the subject, we also need to see the goal from their perspective.
From the analysis made by the pedagogues involved in the projects, we can pick up possible goals or motivations that drive the child to act. Björk [2, p. 12] identifies “to be able to do independently” as a motivation for the child, and she explains how this was then taken into account by the pedagogues in HIPP use. The pedagogues themselves try to recreate a Zone of proximal development [36] with HIPP to enable a more independent creative drawing activity. Björk [2, pp. 49–50] explains the distinction between drawing and exploring and how that is important for the child and the success of the activity from the creativity perspective: Drawing is linked to a more active child while exploring a finished drawing leaves the child more passive.
A key observation is that HIPP enabled that specific distinction by complementing the drawing movement with immediate and dynamic feedback. The combination of movement and immediate feedback was not available to the child in a drawing activity without the HIPP technical system. Without HIPP, only passive drawing explorations or delayed drawing creations were available. According to our analysis, this dynamic characteristic of the HIPP system – in combination with the human pedagogical support – was what contributed most to the successful mediation of the drawing/learning activity. That HIPP successfully mediated the drawing activity is further confirmed by the continued use of the program over a long period of time (two years once a week), the use of HIPP in the classroom, and the requested loan of HIPP during summer holidays by Sam.
Dynamic drawing, stable shared representation
While supporting dynamic drawing, HIPP also provides a stable shared representation. Haptics have been shown [37] to be able to support the creation of “common ground” [38], and we suggest the HIPP system can be considered a place for the pedagogical staff and the child to meet, while embodying this meeting in the form of a 2D multimodal drawing. In this meeting, the child and the pedagogical staff are co-constructing an object of inter-subjectivity. One of the goals is of course to learn, and the 2D drawing is envisioned as a support to externalize parts of the dialogue so that learning can occur.
The interviews emphasized the importance of human audio interpretation both for the child’s integration during usual school activities and during drawing sessions [39]. Audio interpretation is to render in speech the visual world to the child with a visual impairment. The person doing this is often the assistant, but can also be the pedagogue in HIPP. The audio interpretation cannot render everything that happens in the visual world, so the interpreter will of course apply a filter to how and what to render. The relevant or most pregnant elements of what happens around are often interpreted back to the child – including non-lesson related details. With HIPP, the choice is slightly different. The pedagogue can choose to render at several levels: a pure geometrical description or an interpretation of what the lines might look like. That step – going from the lines to the representative drawing with the help of HIPP and audio interpretation – is described in more detail in Björk [2].
Stages of drawing development quoted and translated from Lantz and Mélen [41].
Let us turn to how this “stability” of the representation – or externalized part – is supported by the technical HIPP system: the multimodal feedback is a reliable and externalized representation. It encompasses the visual on screen, the audio feedback for each curve as well as the haptic feedback. Specific features could be introduced or enhanced to increase HIPP’s support:
The automated curve numbering was shown helpful in supporting the human audio interpretation by giving a way to differentiate curves. HIPP also has a feature that allows the users to change the name to reflect something more meaningful (for example “sun” or “circle” instead of “curve 1”) – that feature also supports the human audio interpretation. Automated geometrical form identifications in the automatic names is a possible future enhancement of the automated curve numbering. The observation showed a heavy use of directional words. Except for the kinaesthetic feedback, the program does not currently provide specific directional pointers. A potential future development could be some kind of “up/down” and “left/right” audio scale in independent use (early tests of such a feature was done in [40] Rassmus-Gröhn et al.).
In conclusion:
The HIPP system provides different views (haptic, visual, audio, and relief-printed) to a single medium: the drawing. Through this multimodality, HIPP mediates the access to the drawing to users both with and without visual impairment. In this way, it supports the co-construction of the drawing between different people (intersubjectivity) as argued in Björk [2]. The support of naming the curves as well as consistently giving back their names (via screen reader) supports the construction of a coherent internal image (internalization). The visual to audio interpretation is a central part of the pedagogue’s role during the drawing. This “syntolkning” (audio interpretation) accompanies the audio and force feedback given by the program and integrates as yet an external representation of the drawing to the child.
HIPP uses a haptic device and does not provide a 2D-surface contact. How does that impact the use? A dynamic surface representation could be seen as conveying more information than the unique point of contact available with the haptic device. At the same time, the lack of focus on one point in a surface display might make it impossible to add sound feedback to the drawing. The movements of a haptic device are also a specificity that can be thought as similar to the movements required in drawing for sighted peers. The video observations showed an example of this: Sam used many times the haptic pen to finish off his drawings by a specific move to the right, as if “signing” the drawing. The haptic device is handled mostly as a pen. It is possible to study what kind of handgrip the user has on the pen-implement of the PHANToM [7]. In early uses or in breakdown situations, just holding the pen in a way that one can feel and draw with it can be the whole focus of the activity. In those cases, the tool is not mediating the activity of drawing properly, but only the prerequisite step of holding a writing implement. Later on, when the grip is learned, or when the breakdown has ceased, the grip can be internalized (learnt so that it is unconscious and automatic) and the focus can be on the drawn lines themselves.
One identified missing feature is the possibility to mark dots only (and not just lines) with repeated “push” movements of the pen, which is the stage previous to doodling in learning how to draw according to Lantz and Melén [41] (see p. 10 in Björk [2], or Fig. 13).
Material/external outcomes
During the project, a number of 2D creations were made by both teachers and students using the HIPP program.
The students produced numerous drawings, seen in piles in Fig. 9 on the bottom right. This is something to report specifically on, since being able to produce many drawings apparently brings out a better grasp of drawings in general, and is something specifically enabled by using digital-and-haptic supported drawing instead of traditional drawings. The production of these drawings involved a pedagogical process to support the students’ creativity.
One explicit goal of the project was to create a base of ready-made resources for HIPP, to be used by the teachers in school. The teachers involved in the project chose relevant illustrations, graphs, and 2D tables and created them in HIPP. They also made use of existing documents (drawings from the National Agency for Special Needs Education and Schools SPSM [42] and other existing pictures) that they could include thanks to HIPP functionality that allows the inclusion of background images, as well as the choice of a file format based on SVG. This work resulted in a portfolio both of students’ drawings on the one hand, and of teachers resources on the other that illustrate and facilitate HIPP use for potential new users.
The portfolio directly addresses one issue identified during the study, namely that coordinated preparation of material between Sam’s teacher and Sam’s assistant was crucial for functional and concurrent HIPP use in the full classroom setting. The time required for preparation diminishes when a portfolio is given, enabling HIPP concurrent classroom use to a larger degree.
Teacher’s activities
All the people around the child have diverse approaches to HIPP use. They have their own motives and abilities that will influence the child’s use through the way they support her/him. In this article, we have tried to focus on the user’s activity – the student with a visual impairment who uses HIPP. It was however interesting to consider the activity where the teacher or assistant is the subject. The interviews with teachers/assistants allowed us to gather their perspective on the use of HIPP.
During class, the material required for HIPP needs to be at least in part prepared in advance as we saw in the shadowing observation and interviews. Chronologically, the teacher and the assistant must have met before the lesson, in order to know what it would be about. This is required to allow the assistant to be able to adapt the material to be used for the student with visual impairment. In the school observed this happened because of the well-functioning group dynamic between the teachers, but a practical limitation is the amount of time available. This indicates that it is important to have a well-functioning collaboration – and time allocated to it – for a successful use of HIPP during class time.
The observation also showed the difficulty of conveying the visual information at the same rhythm as the visuo-tactile exploration in HIPP. This points to the benefit of having access to two ways of doing the lessons – both in class (the assistant following the teacher’s lead) and in a side room (the assistant doing the teaching), combining a feeling that all students are included in the same class activity, while still allowing for extra time on a specific lesson when required.
One suggestion that came from the shadowing observation would be to add more audio feedback into the visual exercises HIPP might be used for, in order for the concept to be recognized faster. Music is also a powerful source of examples for mathematical concepts. That would be doable in HIPP but would require thinking out of the box when rewriting the exercises for HIPP.
In Björk [2] the analysis of HIPP in this case of use is made by one of the pedagogues involved. It is argued that the pedagogical-supported use of HIPP enabled the creation and usage of images in all the ways that are usually pointed out as relevant in the Swedish school system (as a preparation for life). The role of the pedagogical scaffolding is also pointed out. HIPP is discussed as an enabling medium where this collaboration and scaffolding, including the technical mediation towards a shared representation (or common ground), is made possible.
During the project, the assistants, teachers, and pedagogues were the most natural relays to us as coders-researchers. They would ask for help when something was difficult to understand about the program, or when a bug happened. In return, our help would let them explore a new way to use HIPP that was not thought about before. We also explicitly designed the project to be iterative, meaning that we had several back and forth between use of the program, reporting on it, and further development.
In that respect, our interpretation is that we created within the project a Zone of Proximal Development [36] for the teachers and assistants in their learning of HIPP use. We as researchers also had a learning process about what the school activities entailed. It is our understanding that those two learning processes were supporting each other, one creating the circumstances for the other to expand and vice versa, both working in their Zone of Proximal Development. Jönsson et al. [28] highlight the importance of putting the technology in the situation and see what happens. The project was influenced by participatory design [43] which also advocates this process. Thus, on the psychological level, the existence of a mutual Zone of Proximal Development between the participants in the development may likely be a pre-requisite for productive participatory design processes.
The project being iterative meant that it was possible to let the users – students and teachers – learn to use the program in its current state, and then the practice developed could further our understanding, allowing us to propose improved functionality, which in turn let the users discover new possibilities of use.
The drawing program was implemented from the start of the project with an emphasis on using standards. Commands that are common in other programs (for example copy/paste/cut) were implemented consistently. This was an positively appreciated, as confirmed by one of the assistants during interviewing. The focus group interview brought up another dimension of this result: a common set of teacher tools.
Of course, the teacher’s, child’s, parents’ previous knowledge and affinity about or towards technology must have played a role in the actual use of HIPP. In the case of the teachers, the interviews and project meetings shed light on some common elements that teachers might be able to relate to, like a nation-wide technology course (PIM – Praktisk IT och mediekompetens) offered to teachers. It is useful to note that since these standards are taught nationally for all teachers, the progress of making HIPP more similar or compatible with them is a progression that can be generalizable nationally, as it is embedded in a national structural system.
Coder and researcher
In considering the results of the observations and interviews, a specific bias had to be considered: We represented both the developers of the program as well as the researchers. This double role influenced the answers we received and the focus we have had in the work. It was for example often natural to attend to a bug just before or after conducting an interview, just because of the practicalities – it is sometimes easier to group the things to be done on site during the same timeframe. The interviews and the observation might have been in general more positive or more negative depending on the coding actions. The interview questions and the observations were often on parts of the usage that the researcher as a coder ignored – namely the environment and the pedagogical practices. It was a way to bring back a fuller picture and avoid a negative effect of the coder-researcher bias.
It might be, however, an inevitable and maybe even desirable bias, as long as it is kept in mind. In p. 123 of [44], the importance of considering, as Latour (chapter 6 in [45, 46]) that “technology is society made durable” [47], can be taken at the heart in such projects. It was thus essential to include more development alongside the introduction of the technology in schools. That way, there was time to see how ideas that were embedded in the development were received in the testing. This process provides room for change of the embedded ideas by changing the technology itself. By being both a coder and a researcher, it is possible to code, see how the code is received, and use it to reflect back on the values that we have embedded in the code. It is thus possible to change them explicitly, because the analysis can be put into action through the researcher’s coding competence.
As a last note on this subject, we can consider something that was learned during this study but that was very counter-intuitive from a coder perspective: Errors in cultural specificities might have more importance than functional errors. The lack of specific Swedish letters was reported multiple times while the device misbehaving was not mentioned and shrugged off during observation as: “It does that sometimes, we have to restart”.
Conclusions
During this project, visually impaired children used the HIPP drawing system in a school setting. The study was done after a long-term use of HIPP: the program was used weekly over one to two years by two visually impaired students accompanied by their teachers and assistants. These successful long-term uses as well as the interview results show that HIPP supported the drawing activities successfully.
Through observations and interviews, we were able to confirm the successful mediation between the user and the object of their activity, their drawings. We considered the user with its human environment: teachers, classmates, assistants, and family and have showed how HIPP enabled them to share an understanding of the drawings under focus, to reach common ground in the drawing and learning activity. The HIPP multimodal system mediated the drawing and interactive learning activity by allowing the dynamic exploration, creation and editing of a stable and multimodal (haptic, visual and audio) representation. Additionally, the system provides static representations (tactile printouts) that can be used outside of the program or in complement to it. Both dynamic and stable representations were seen to be important; the dynamic representation provided direct feedback, allowed seamless gestures, exploration and creation, and supported interactive learning, while the static printouts allowed results to be shared for example as letters and also incorporated into joint presentations at school. Both representations – dynamic and static – were stable in that they provided automated feedback that any user could come back to numerous times through different modalities, in order to support grounding between several users.
HIPP allowed teachers to make use of pedagogical methods better suited to the learning of 2D graphics (Björk [2]). It is a takeaway that not only the children, but also teachers and assistants, need to be actively involved in the development of any new technical pedagogical/accessibility aid, in order for it to be used effectively to the full extent of its possibilities. One aspect of this is that the software needs to fit into the existing environment at school and preferably not add extra work for the teachers and assistants – whenever it is possible to adopt standard solutions (e.g. standard menu shortcuts), it should be done.
Future work
A possible future study would be to compare the usefulness of the HIPP drawing system against other drawing options available to visually impaired users in specific 2D creation and perception tasks. It would be possible to design a test case by letting a number of visually impaired students enrol in a two-year program where half of them are given weekly drawing sessions with HIPP and half of them weekly drawing sessions with other tools. Standard drawing and 2D material interpretation tests could be done at the beginning and end of the test period.
Another possible way to extend this study would be to identify and integrate the parts of the wider social system more systematically into the HIPP program, and vice versa. For example, it could benefit to be part of the teacher training when it comes to drawing as an example of how to make lessons accessible. It would then be needed to reintegrate into HIPP whatever modifications come up that make the use of HIPP more seamless (or internalized) in the classroom routines – like using the digital versions of the drawings shown on the smartboard into HIPP.
Finally, we identified an interesting topic for further analysis of the video observation data: How does the grounding [38] process happen during the learning and drawing activities where HIPP is used, and how does it affect learning? Some similar studies have been done in different settings [37], but relating that analysis to an activity model would give us insight into the process in which the technology mediates the learning and drawing activity within the child’s zone of proximal development.
Limitations
We have described how the methods and theory were used to gather as much data as possible from this study. The main limitation, however, is that this is a qualitative study. In itself, it is not possible to reproduce. It was not possible to control the parameters and test exhaustively to understand the correlations of cause and effect precisely.
Because of the nature of the methods used – interview and observation, qualitative methods – the bias of the observer/researcher has to be taken into account. Precautions were taken as named above (accounting for cultural and professional bias, use of analytical constructs) – but the subjectivity of the interpretation is nonetheless a limitation that should be named.
Another possible way to act would have been to make the analysis more explicit by relating each bit of data gathered to each of the possible parts in an activity diamond or categories in the activity checklist extensively. A more structured method of analysis would have yielded a more reproducible analysis. Making the categorizations of the data points more explicit would have also been a way to improve on the current analysis.
Another limitation inherent to qualitative analysis of this type is the one that language and communication creates. Language and direct relationships within interviews are great tools to understand more of the activity system, but they are dependent on the language and communication skills of both the interviewer, the observer and the participants. In this study, the native language of the participants was different from that of the researcher who conducted both interviews and observations. This combined with the communication abilities of the participants limited the possible interviews on site. This was mitigated by transcribing the content of the interviews’ and video observations’ recordings.
Footnotes
Acknowledgments
We gratefully acknowledge the participants, who tested and used HIPP over several years. This includes the users and their assistants, teachers, pedagogues, classmates and family. We also acknowledge previous work done in the Micole project, supported by the EU IST 6th framework program, on which the HIPP program was based. The HIPP project was supported by funding from Arvsfonden.
Conflict of interest
None to report.