Haptic and interactive illustrations for visually impaired children: A literature review

Haptic feedback for storytelling

S. Zhao, N. Yannier, and A. Israr, in collaboration with Disney Research, introduced two interfaces for vibrotactile perceptions in story listening technologies. The first is a haptic vest equipped with a matrix of vibrators positioned on the back.^10,11 The second interface is designed to attach to the back of a tablet, providing vibratory feedback to the hands.^12,13 These technologies deliver vibrations corresponding to specific words and their synonyms within a story. For instance, terms like rain and downpour are paired with intermittent vibrations of varying intensities, mimicking falling raindrops. Tests with 29 children aged four to six for the vest ¹⁰ and 44 children aged six to nine for the tablet-attached interface ¹² show that children around 6 years old are able to associate haptic input with lexical content, memorize it, and better understand the narrative. Children were able to associate synonymous words with corresponding vibrations when repeated, which aided their understanding of the narrative during the question-and-answer phase. This finding is particularly relevant for using vibration-based feedback in accessible illustrations for visually impaired children, especially in narratives designed to enrich vocabulary acquisition. However, results vary across vocabulary items, primarily due to the narrative context, which plays a crucial role in the protocol. The project should therefore implement a protocol that compares different narratives for the same vocabulary and associated interactions.

In parallel, A. Israr and his collaborators worked on the concept of stereohaptics across various devices, including the two previously mentioned. By using at least two vibrators, they create the illusion of vibrations moving through space, perceivable on the back or hands. These studies emphasize immersive entertainment applications,^{14,15,16,17,18,19,20} including virtual reality video games, cinemas, and theme parks. These projects gradually involved an increasing number of participants, starting with 4 colleagues, then 10 participants, and later 18 participants. The results indicate that participants correctly perceived the dynamic movements of the vibrations. The concept of stereohaptics is particularly promising for creating dynamic illustrations, enabling professionals specialized in accessible content design to represent concepts that evolve over space and time, such as energy or transformation.

Haptic feedback for verbal communication

Communication through haptic perceptions is a non-visual solution for improving accessibility. The MISSIVE project (Multisensory Interface of Stretch, Squeeze, and Integrated Vibration Elements), led by N. Dunkelberger, J. Sullivan, and others, in collaboration with Facebook, developed an armband interface.^21,22 Worn on the back of the arm, the interface combines vibration, pressure, and skin-stretching patterns to communicate forty phonemes of the English language. Based on an experimentation involving a group of 12 participants, they obtained results which tend to demonstrate the effectiveness of combining these three haptic modes. J. Chen, A. Israr, and collaborators tested a similar device concept for the forearm. ²³ This system associates vibration patterns with phonemes to communicate approximately one hundred words. Participant studies revealed effective memorization of these patterns, emphasizing the importance of a guided learning phase in the successful use of such devices. Although the memorization of patterns appears effective, it still requires a significant cognitive effort, which must be taken into account.

Haptic feedback for virtual reality

E. Pezent and colleagues, with Facebook Technologies introduced Tasbi (Tactile and Squeeze Bracelet Interface), a multisensory bracelet featuring two haptic modalities. ²⁴ Tasbi combines vibrations and uniform pressure applied around the wrist. Linear vibrations are generated by six vibrators, while pressure is applied using a tensioned cord driven by a motorized coil. Tasbi’s primary function, designed for use in virtual reality (VR), to assist with the manipulation of virtual objects and enhance virtual proprioception. For instance, in VR environments, the bracelet vibrates upon contact—collision—when pressing a virtual button and applies pressure to the wrist proportional to the button’s resistance. The Tasbi project has been refined through various research avenues, including the rigidity of virtual objects ²⁵ and the exploration of different vibration frequencies. ²⁶ The research follows a participatory approach, progressively engaging study panels—for instance, pressure calibration was conducted with twelve participants, ²⁷ and bracelet design iterations involved panels of six participants.^28,29 Depending on context—such as the action performed or object manipulated—users interpret the same haptic feedback differently. Visual cues appear to significantly influence the effectiveness of the experience. In light of the themes explored in this review, adapting the Tasbi system—or similar multisensory interfaces—for visually impaired users would represent a meaningful step toward more inclusive virtual environments.

Synthesis and discussion of the subsection

Papers related to this subtopic have been published between 2010 and 2022, reflecting the relatively recent use of haptic perception in electronic devices to support rich communication—particularly in language, storytelling, and virtual reality. The development and design of these projects have advanced rapidly, leveraging cutting-edge technology supported by major companies such as Disney and Facebook.

These studies are still in the early stages of design and research, generally involving small and homogeneous participant panels—except for the storytelling studies, which include more participants, notably children under 9 and 6 years old. While the phoneme communication and virtual reality studies primarily target adults, they remain valuable for exploring the potential of vibration-based interaction. Research on armbands and phoneme-based systems is inspiring for the development of non-visual verbal communication systems, which could complement braille for visually impaired readers. In the same vein, the virtual reality studies take an approach to building inclusive virtual environments through multimodal interaction, including vibration feedback. Future research involving larger and more diverse participant panels, particularly those including individuals with visual impairments, would help generate new hypotheses and guide the refinement of these systems.

Haptic illustrations

The concept of haptic and removable illustrations—objects has been studied in the literature by researchers such as F. Bara, D. Valente, and collaborators. ⁷ This concept allows readers to manipulate removable illustrations, such as opening a door to represent a house, pulling a flush handle to signify a toilet, and other similar interactions. The goal of these manipulations is to stimulate sensorimotor memory and proprioception. ³⁰ These studies advocate for 3D illustrations—referred to as 2.5D by the tactile designers, as the illustrations remain flat within the pages of the books. This method complements tactile perceptions textures, by incorporating a reading experience enriched with actions, such as gestures. In one study, three six-year-old children with visual impairments from a specialized center participated in reading sessions with an educator. ² A comparative analysis, involving three stories presented with and without 2D and 3D illustrations, indicated a modest improvement in story recall when tactile elements were included.

Imitation gestures

Imitation gestures present another solution to enhance accessible illustrations for young visually impaired readers. They consist on using hands and fingers—index and middle finger—to embody a character or an object within the narration and the haptic illustration as support. Imitating actions provides an approach to learning vocabulary related to verbs and their associated objects—illustrated supports—for the visually impaired. D. Valente confort the use of this concept, she used 3D miniature—illustrations—designed to engage children to imitate such actions; like jumping on a trampoline, using a swing, climbing stairs, and more, exemplify this approach. ³¹ In Ref. 32, D. Valente has studied the concept of ASFGs (Action Simulation by Finger Gestures) by involving panels of eighty sighted participants and one hundred sighted participants, who were asked to recognize ASFGs produced by twenty blindfolded individuals and seventeen early and late blind participants. Her research aims to explore the relationship between ASFGs, proprioception, and visual perception, with a particular focus on how the ability to “feel” and “see” one’s body contributes to the re-enactment of actions. Sensorimotor memory is activated through haptic experiences facilitated by tactile and removable illustrations, as well as through the execution of finger gestures.

Textured illustrations

C. Mascle and F. Bara evaluated three approaches to illustrating vocabulary with two panels of children—36 sighted and eighteen blind. ³³ They evaluated the effectiveness of textures and tactile pictograms. Tactile pictograms are raised dots arranged as an illustration to represent a word.^34,35,36 They conducted a comparative study using lists of ten words (e.g., objects and animals), they associated each word with three types of illustrations: (1) a single textured circle, (2) a composition of cut-out and sewn textures for a visual effect, and (3) tactile pictograms. Their results favor the use of textures, which were easier to recognize and memorize. In contrast, children struggled to understand and retain the meaning of tactile pictograms.

Building on these findings, the researchers adapted the story of Little Red Riding Hood using the textured circle approach. In a new study involving seven visually impaired children, they evaluated their appreciation and comprehension of textured circles. ³ Results showed that the children could identify characters and follow the story’s progression with mediation. Mediation was necessary to introduce the textures before the storytelling, conducted as a shared reading activity with an adult.

In another initiative, C. Mascle and D. Valente organized a workshop involving twelve families, each with a visually impaired child and their parents. ⁶ Together, they designed tactile textured illustrations to enrich their personal libraries. The participants’ preferences led to the creation of a tactile illustration kit.

These studies share a common methodological approach: the use of simple textures as a central solution for illustration—whether comparing tactile techniques, adapting stories for shared reading, or involving visually impaired children in participatory workshops to select preferred textures for their personal libraries.

Augmented tactile illustrations

Refs. 9 and 37^9,37 present our project IMI—Illustrations Multisensorielles Interactives (Interactive Multisensory Illustrations). We designed a prototype by modeling and producing an illustration page. This page integrates textures and electronic components to apply oral reading and autonomous interactions. Thirty blindfolded sighted children participated in testing the prototype, embodying the main character using their index and middle fingers. The prototype’s oral narration was divided into several segments. After each segment, the children were immersed in the story’s universe through stereo sound and touch-sensitive surfaces. They could interact with the narrative by: moving toward characters based on their spatial audio cues, hearing the sound of footsteps in grass, feeling vibrations that represented the movement of bees, and experiencing the ambient sounds of a forest. the use of stereophonic sounds and vibratory feedback appeared effective for supporting children’s imitation and exploration of the haptic illustration. During narrative recall, children relied on the experienced interactions and sensory cues they had perceived and heard as key reference points to reconstruct the story. Though the final aim of this project is toward children with visual impairment, these papers are focusing on the design of the technology; the next step involves visually impaired children.

Synthesis and discussion of the subsection

The design of tactile illustrations for visually impaired children is not a new research topic. However, it has primarily been explored within the field of developmental psychology. ³⁸ Over time, the literature reveals an evolution driven by earlier studies and advancements in technology. This progression spans from complex raised illustrations to simple textures and also from simple haptic and removable components to haptic scenarios and, finally, to the integration of electronic devices. Throughout, the consistent focus has been on accessibility and children’s literature. Moreover, the vast majority of these projects have involved the LDQR association—Les Doigts Qui Rêvent (The Dreaming Fingers)—in the design and production of illustrations and handcrafted prototypes, which were subsequently tested with visually impaired children.

Although current interactions remain limited by technical constraints, emerging multisensory technologies—such as vibration feedback—may represent the next step in advancing accessible illustration design. A key strength of these projects lies in their user-centered design, focusing specifically on visually impaired children and evolving in response to their needs.

Multisensory book

The specialized institution Elizabeth School for the Blind in Malaysia collaborated on a project led by H. Nishino and collaborators. They presented a multisensory book that folds like an accordion.^39,40 The narration is based on the cat from the tale Alice in Wonderland. The cat is hidden, and the goal is to find it as it progresses through the story across six interactive illustrations. The thick pages use touch-sensitive, conductive surfaces to provide interactive audio feedback through speakers. About ten educators and 25 children experienced this narration through adapted texts: enlarged characters and Braille. They searched for the cat by touching its fur, its food, and objects in the scenery that concealed it, all while listening to its presence and movements. Feedback was enthusiastic, with participants asking for more interactions and extended narrations.

eTextile crafting

I. Posch, a researcher at The Institute for Design and Assessment of Technology at TU Vienna, has undertaken several projects involving eTextiles (Electronic Textiles), also referred to as smart fabrics.^41,42,43 Inspired by childhood memories with her grandmother, she created a book using her knowledge of eTextiles and embedded systems. ⁴⁴ Her book consists of twelve illustrations crafted with conductive and smart fabrics. These measure temperature, humidity, and light intensity using an Arduino Nano electronic board. Two children explored her illustrations, to: make a parrot talk thanks to an integrated microphone and speaker, brush the hair of a doll that opens and closes its eyelids, feel the purring of a cat with vibrators, light up a Christmas tree with LEDs, and engage in many other interactions.

P. Moorthy and her colleagues developed a similar book featuring three interactive illustrations.^45,46 Their research focuses on the precognitive development of young children, with a hypothesis emphasizing the role of multisensory interactions. Eighteen children under the age of seven, accompanied by their parents, explored three illustrated animals. Each illustration triggered different interactions based on touch: the fish’s tail produced a melody trough a buzzer; the bear’s belly, equipped with a pressure-sensitive velostat sensor, emitted vibrations; and the firefly’s tail, detected by a photosensitive sensor, illuminated LEDs representing stars.

Augmented tactile books

The LTA project—Livres Tactiles Augmentés (Augmented Tactile Books)—integrates electronic devices into tactile illustrations within the pages of a book. The first prototype is titled Petite main se promène, which stands for Little hand walks. ⁸ The process enables an immersive reading experience, through touch-sensitive surfaces that detect readers’ gestures. The book prototype was tested with two groups of visually impaired children - 5 blind and 6 with a low vision. ¹ These previous studies culminated in the creation of a sturdy book, titled Kapi Kapitaine, authored by Lucie Félix, inspired by the haptic scenarios of D. Valente. ⁴⁷ In which, the left-hand pages feature adapted texts, including enlarged print and Braille. While the right-hand pages contain augmented tactile illustrations, that combine textures with removable objects. The electronic devices are integrated into the book’s thick pages. The book is powered by a rechargeable battery and connects through Wi-Fi to a separate box for interactive audio feedback. Its illustrations encourage engagement through activities such as: jumping into puddles, opening a window to hear birds singing, dancing on a carpet to the rhythm of music, and navigating a boat across the ocean. Currently, Kapi Kapitaine is produced in twelve copies, to be tested in about ten inclusive and specialized institutions for visually impaired children.

Crossover multisensory picturebook

The Crossover Multisensory Picturebook ⁴⁸ by S. Karaoğlu and colleagues involves elderly residents in retirement homes as active participants. The book’s illustrations are augmented with electronic components, including a button and a potentiometer, programmed using an Arduino electronic board. Seniors contributed to the book’s design by sharing their memories and preferences, which shaped its content. Within the pages, they can turn on a gas lamp or a radio to play music from their era and insert a token into a machine reminiscent of ferry rides. Feedback is provided through light and sound, using an LED and a speaker. This project aims to stimulate memories while fostering shared reading experiences between seniors and younger generations.

Synthesis and discussion of the subsection

Among the projects reviewed, the earliest article dates back to 2016, while three are from 2024, reflecting the relatively recent emergence of augmented books in the literature. These projects are predominantly in the early stages of design. Over time, it is noticeable that they incorporate more complex and abundant interactions. This evolution is not only due to the increasing accessibility of electronic components, but also to the growing ability to embed richer interactions—such as synchronized sound, vibration patterns, and tactile feedback—directly within the pages or illustrations. These improvements help overcome previous technical constraints, allowing non-engineering professionals to more easily design and prototype such projects.

The book format presents significant design challenges due to the physical constraints of turning pages and the requirement for illustrations to remain relatively flat, limiting the integration of fully three-dimensional movable components. In the reviewed projects, the pages often appeared thick, and the overall ergonomics of the books—particularly for young children—seemed inadequate. Considering that visually impaired children may interact with illustrations more vigorously, this aspect raises further concerns. There is a clear need for additional research focused on the ergonomic design of augmented books to support the continued development of accessible and durable formats. Moreover, the various objectives of these projects are not all centered on accessibility, although they do highlight the broader appeal of augmented books to a wider audience—including young children and elderly people in retirement homes. These projects do not present comparative studies or formulate hypotheses related to narration or vocabulary acquisition. Instead, they focus on the expertise and creative input of multiple participants to design interactive and multisensory book prototypes.