Abstract
This paper presents the design of a glasses system for the blind and visually impaired that incorporates various technologies to enhance their daily living experience. The glasses are equipped with a speaker and haptic feedback, allowing users to receive audible and tactile notifications about their surroundings. We also implemented image understanding for identifying multiple unique objects, people, and environmental obstacles. Additionally, the glasses can read signs and printed materials via text recognition technology. Finally, the glasses incorporate navigation tools, helping users to find their way in unfamiliar environments. The system has the potential to significantly improve the independence and quality of life of those with visual impairments, as it could be a valuable assistive technology tool for a wide range of applications.
Keywords
Introduction
World Health Organization (WHO) defines visual impairment on various levels based on the individual’s better eye visual acuity level, such as Blind (worse than 3/60), Severe (6/60 to 3/60), Moderate (6/18 to 6/60), and Mild (6/12, 6/18) (WHO, 2022). According to The International Agency for the Prevention of Blindness (IAPB), the number of blind people worldwide will increase from 43 million in 2020 to 61 million in 2050, with moderate to severe from 295 to 474 million (The International Agency for the Prevention of Blindness (IAPB), 2023). These user groups require special assistance as they cannot navigate an unknown environment without the help of assistive technology (Agrawal et al., 2022). Researchers are investigating how to make the lives of people with disabilities and older adults, of which a significant portion has a visual impairment to some extent, comfortable, independent, and free of participation in social activities and research (Brinkley, 2021; Brinkley, Daily, et al., 2019; Brinkley et al., 2017, 2018, 2022; Brinkley, Posadas, et al., 2019; Enam et al., 2023; Gluck et al., 2020). Though the advancements of technologies are expanding at a rapid rate, a cure for blindness still seems far from development (Alam et al., 2022).
Therefore, traveling for blind people requires some assistance, and new emerging technology might provide a solution. Autonomous vehicles could allow blind people to travel to any desired destination (Brinkley et al., 2022). However, indoor movement or walking to reach an autonomous vehicle needs guidance, especially in an unknown environment (Khan et al., 2022). To address this issue, researchers have developed technologies using a global positioning system (GPS) to locate the position of a blind user, ultrasonic sensors to detect objects, and computer vision technologies for sidewalk edge detection (Astler et al., 2011; Terven et al., 2014). However, technologies have not been realistic enough to help individuals navigate in real-world traffic and under various uncertainties. Researchers continue to explore how to increase the self-sufficiency of blind people in their daily lives.
Keeping in mind the portability, convenience, and safety, in this paper, we present the design and development of a glasses system for the blind and visually impaired that incorporates speakers, haptics, image understanding, text recognition, and GPS for navigation. The glasses have the potential to significantly improve the daily living experience of those with visual impairments, allowing them to navigate their environment freely, access information, interact with others more effectively, and envision themselves working in or traveling to a distant location. Overall, the glasses represent a significant advancement in assistive technology for the visually impaired and have the potential to make a meaningful impact on the lives of people worldwide.
Related Work
Most blind people use a white cane to move indoors and outdoors (Bousbia-Salah et al., 2011). However, it is insufficient to navigate in an environment where the paths are unknown and uncertain and may cause the blind person to stumble. An electronic white cane with RFID technology provides blind users the ability to move around in an unfamiliar environment that continuously takes data from the geographic information system (GIS) (Faria et al., 2010). However, it is limited as it does not detect objects outside the cane’s range, such as a tree branch or hanging signboard. An autonomous robot is another option for blind people to navigate by holding a handle that provides feedback through vibration to the users’ hands or a speaker (Guerreiro et al., 2019; Lu et al., 2021). While most of these robots use ultrasonic or solar sensors, some use laser range sensors. The issue with this solution is that it takes space for the robot to move. Also, it burdens the users as they must carry the robot using private or public transportation to reach a destination.
Smartphone-based systems solve the portability issue as they are easy to carry (Ahmetovic et al., 2016; Ganz et al., 2014). These mobile phone solutions provide turn-by-turn navigation to the users. However, the users must open the application and hold the phone while navigating. Additionally, there is the risk that the phone will ring in the middle of the navigation, distracting the user. Wearable devices, such as head-mounted devices, glasses with integrated cameras, and ultrasonic sensors, are another popular solution for blind users’ navigation in an environment (Bai et al., 2018; Zare et al., 2022). Though it has the potential to solve issues related to portability, it has a few limitations. For example, Bai et al. designed glasses that use cameras to find the path using an ultrasonic sensor to detect objects in front of the user while navigating (Bai et al., 2018). However, the issue is that this device is bulky and cumbersome for regular use.
Our glasses design for blind users to navigate is different from other currently available solutions as we focus more on the convenience of the user, such as 1) Integrating the capability to read text, 2) Detecting sidewalks and objects using a small camera, 3) interactive button support to trigger voice user interface and image understanding functionalities. Moreover, the small and lightweight glasses can be worn like regular glasses.
Glasses Design
External Structure
We designed and fabricated the glasses throughout multiple iterations. We used a Polylactic Acid (PLA) 3D printer to print most of the body of the glasses, and a Stereolithography (SLA) 3D printer to print the glasses cover. Even though we developed the complete glass structure, including the temple, we later used a strap to get a perfect fit. The entire setup was a little heavier than regular glass as the commercially available components we used were not fabricated for our use case. Figure 1 demonstrates the development’s initial and final versions.
Glass body design using AutoCAD.
Internal Components
The internal structure consisted of eight unique components, a few on either side (Figure 2). We used Raspberry Pi Zero-Wifi as our main computer inside the glasses, that maintains communication with the server. We used speakers and haptics for auditory and haptic feedback. We used the I2S MEMS breakout (SPH0645LM4H) for the microphone, the tri-axis magnetometer breakout (HMC5883L) for orientation, and the iPhone-7 speaker for auditory feedback. We also used a 1000mAh battery to run the device wirelessly.
Internal components of the glass system.
Implementation
The glasses system is integrated into the cloud backend with various ATLAS microservices (Brinkley, Daily, et al., 2019), as demonstrated in Figure 3. However, the glasses feature three main components: Perception, Localization and Positioning, and Mode of Interaction. These sections are explained below.
System Architecture.
Perception
Sidewalk Detection
Perception is a primary difficulty for blind individuals in achieving their independence. With our glasses, we implemented sidewalk detection (Figure 4) using semantic segmentation using a deep lab v3 (Chen et al., 2017) trained on the cityscapes dataset (Cordts et al., 2016). We achieved a framerate of nearly five frames per second (FPS) while on the Internet and about 25 FPS connected to a faster 5G or WIFI network. We set up SSH tunneling to access the glasses’ camera image from the machine learning pipeline’s backend server. We deployed the backend locally and accessed it remotely using University anonymized gateway. Additionally, this system could be deployed on a large scale using any other web service.
Visual representation of the detected scene.
Image Understanding
We used Google’s Vision API (Google, 2023a) for image understanding. We provide information on multiple objects using this service. The user can select the number of object information to render using the mobile application to configure each user profile. This technology provides the object detection functionality of the glasses (Figure 5).
Object Detection using Google Vision AI (Google, 2023b).
Text Recognition
Optical character recognition (OCR) helps the user read the text in front. Our findings echoed Toyama et al.’s findings, showing that the head-mounted orientation method performed better than phone-based techniques (Toyama et al., 2014).
Localization and Positioning
We used a magnetometer for on-device orientation, the user’s smartphone for GPS and accelerometer, and Gyroscope data. We used GPS + IMU fused coordinates to localize a person. A mobile application was developed to communicate with the server to get the navigation information, which then forwards this information to the glasses (Figure 6). We also account for random head movement so that the head’s orientation closely represents the body’s orientation with additional phone magnetometer readings.
Mobile Application for the Glasses.
Mode of Interaction
The glasses feature two different modes of interacting with the user. One uses a voice user interface (VUI), and the other uses haptic feedback (Figure 7). Auditory feedback delivers information about detected objects, recognized text, navigation cues, and the system’s status (e.g., Wi-Fi not connected). For haptic feedback, we used an 8mm haptic motor on both sides of the glasses to deliver directional cues (e.g., left, right, and front) using different haptic configurations.
in-vehicle control and real-world navigation using the different interaction modes.
Discussion
The following use cases were identified for the glasses we developed.
Ingress and Egress to and from Autonomous Vehicles
The initial goal of developing the glasses was to create an assistive technology to navigate a blind or visually impaired (BVI) individual to and from a vehicle. As technology advances, the glasses could assist with efficiently using and interacting with future autonomous vehicle technologies. Also, we later figured out that adding multiple functionalities, such as image understanding, increased the usability for entering and exiting a vehicle. Figure 5 shows how the mobile application records the coordinates of the parked car so the user can navigate to or from the vehicle using the system’s VUI.
In-vehicle Usage
The glasses were also developed to aid users in obtaining real-world information about their surroundings. The direction the user’s head faces determines what information the glasses deliver, as the glasses camera only looks forward. Through the glasses, a user’s situational awareness is increased, thus increasing their independence. Additionally, the glasses were designed to eventually allow users to interact with the in-vehicle system using the glass’s VUI (e.g., Playing music) via Bluetooth connectivity.
Low Cost of Fabrication
We identified that the fabrication of such a system is relatively low cost compared to various accessibility aids for BVI individuals. With server-side information processing, the on-device components require minimal computing power, thus decreasing the overall size. Therefore, when produced in mass, such devices would not cost more than devices readily available Internet of Things (e.g., Internet protocol cameras).
Lessons Learned
Through our development, we have also identified that processing haptic and auditory information can sometimes result in information loss. For example, the user might not always be able to detect haptic feedback while in motion unless the vibrational intensity is high to offset any vibration from the act of walking itself. Additionally, auditory information like “left” or “right” take some time to be rendered and recognized, thus reducing the amount of auditory input that can be delivered at a time. With this, we suggest future research to explore using spatial audio, which in our opinion, may reduce the cognitive load and provides a more intuitive experience to the user. Finally, we observed a potential benefit of glasses compared to the hand-based technique: it solves the issue of the user being unable to point the phone in the right direction. This design of integrating these functionalities in the form of glasses is much more convenient, intuitive, and less cognitively taxing.
Artificial Intelligence (AI) is the Ultimate Solution
Various efforts, like Aira AI (Bin et al., 2021) and Be My Eyes (Be My Eyes, 2023), have been to help BVI individuals with navigation and daily activities. Still, we believe AI is the ultimate solution for implementing navigation for blind people. We question whether such technologies with a live agent to guide a BVI individual would be a sustainable alternative to AI. Also, most current implementations involve the user holding a phone while acting in scenarios. This method can introduce difficulty for a BVI individual to navigate as they are accustomed to using a cane in one hand. We thus suggest such enterprises focus more on AI-based assistance and try to make it hands-free to provide an easy and intuitive user experience.
Limitations and Future Work
One major limitation of our implementation is the server-side processing of graphical information. Even though this worked very well in the facility where we tested it, it might not work well in locations with poor Internet connectivity. We also tried to perform the segmentation task on the phone but could not due to various limitations on the available SDKs. We are still working on processing the information on-premises on the user’s phone so that a delayed response does not lead to catastrophic consequences. We are also currently working on various other hands-free navigation mechanisms for indoor environments.
Face recognition is another implementation we want to include in our future development. We initially thought of the device as a navigation aid and then later were able to use the visual information gathered to detect objects and read text. We aim to create a profile or contact-based architecture that assigns labels of detected faces among added contacts. One significant difficulty in such scenarios can be adding facial information on the go without increasing overhead to the user’s interaction. Subsequent implementation could implement a phone number assigned to the name and image that could be treated as a packet and used differently in different scenarios. The complexity of this feature can be seen when trying to add a person to contact in a noisy street or sub-optimal lighting conditions. This information relies on the person speaking their name and number, and the camera detects their face.
Finally, after we achieve these amendments, we plan on conducting user experiments to measure task performance and cognitive load to identify usability issues and iteratively refine the system to meet accessibility standards.
Conclusion
In conclusion, the design of glasses for the blind is an innovative and exciting assistive technology development. These glasses can significantly improve a BVI individual’s quality of life by providing a new level of independence and mobility. These glasses can help the wearer navigate their environment and detect obstacles by incorporating features such as cameras, haptics, and audio feedback. Additionally, the glasses can provide real-time information about the user’s surroundings through the image understanding feature. While there are still challenges to overcome, such as the need for further testing and refinement, the potential benefits of these glasses are clear. With continued research and development, this technology will become more accessible and widely available, improving the lives of visually impaired individuals worldwide.