A Virtual Environment-Based Training System for a Blind Wheelchair User Through Use of Three-Dimensional Audio Supported by Electroencephalography

Abstract

People with disabilities encounter many difficulties, especially when a diagnosis of more than one dysfunction is made, as is the case for visually impaired wheelchair users. In fact, this scenario generates a degree of incapacity in terms of the performance of basic activities on the part of the wheelchair user. The treatment of disabled patients is performed in an individualized manner according to their particular clinical aspects. People with visual and motor disabilities are restricted in independent navigation. In this navigation scenario, there is a requirement for interaction that justifies the use of virtual reality (VR). In addition, locomotion needs to be under natural control to be successfully incorporated. Based on such a condition, electroencephalography (EEG) has shown great advances in the area of health by employing spontaneous brain signals. This research demonstrates, through an experiment, the use of a wheelchair adapted to have the support of VR and EEG for training of locomotion and individualized interaction of wheelchair users with visual impairment. The objective was to provide efficient interactions, thus allowing the social inclusion of patients who are considered otherwise incapacitated. This project was based on the following criteria: natural control, feedback, stimuli, and safety. A multilayer computer rehabilitation system was developed that incorporated natural interaction supported by EEG, which activated the movements in the virtual environment and real wheelchair through adequately performed experiments. This research consisted of elaborating a suitable approach for blind wheelchair user patients. The results demonstrated that the use of VR with EEG signals has the potential for improving the quality of life and independence of blind wheelchair users.

Introduction

People with disabilities face a daily battle, as the lack of accessibility and social inclusion is still a big problem.^1,2 There currently exist a significant number of people with disabilities; therefore, solutions that will help these people are constantly sought, whether for mobility, rehabilitation, communication, or digital inclusion. In Brazil, there are still many structural issues in the area of therapy for the disabled. For example, the 2010 Census³ shows that 45,606,048 of Brazilians, 23.9% of the total population, have some kind of disability—visual, auditory, motor, mental, or intellectual. Visual impairment had the highest incidence rate, affecting 18.6% of the population. Second is motor impairment, occurring in 7% of the population, followed by auditory in 5.10% and mental disabilities in 1.40%.³

The disability knowledge area offers categorical challenges in helping people with severe needs, and their impact on the right to come and go. This constitutional right is often violated due to the lack of accessibility,⁴ especially when dealing with patients who have more than one significant disability. Those patients are classified as incapacitated for independent living. Motivated by this scenario, this work is about the difficulty of locomotion of the blind wheelchair user. This class of patients, by far, depends on the use of wheelchairs for the rest of their lives. In fact, individuals with multiple disabilities, on average, are shown to suffer greater exclusion from new technologies. In this case, these computational systems need to be supported by a type of interface that provides the user with a more cognitive form of training. This emphasis is placed upon virtual reality.⁵ VR can be visualized as a system used to create an artificial environment, in which the user has the impression not only of being inside this environment but also enabled, with the ability to navigate the same, interacting with its objects in an intuitive and natural way.⁶ In addition, studies demonstrate that creating a functional space for the tetraplegic individual through a computer system is favorable for the training of the wheelchair user.⁷ Accordingly, through the already mentioned studies and the long search for social inclusion, newly assisted technologies are being sought and developed.⁸ One of the main functional interfaces is the brain computer interface (BCI), which is a technique that aims at interpreting electrical signals from the cortical surface of the brain without going through nerves and muscles. This technique called electroencephalography (EEG) is used to acquire brain signals to interact with the external environment through devices, seeking to interpret thoughts toward movements without the need for real movement.⁹

Currently, the major VR systems for wheelchair training through brain waves do not support visually impaired wheelchair users.¹⁰ Therefore, the synchronization between acquisition of the signal and the virtual environment (VE) presents itself as a challenge to be overcome.¹¹ The proposal behind this study is to, therefore, approach the development of a tool for training blind wheelchair users.¹²

Overview

Each deficiency has prerequisites for implementation of computational systems and experimental approaches for training and treatment solutions. One notes here that all projects are focused only on one deficiency, leaving aside patients with more than one dysfunction, which could be treated together, being necessary only to redefine the appropriate requirements for both disabilities. Table 1 presents the list of studies categorized with specific criteria required for each impairment, these projects presented important attention points.

Table 1.

Describe Requirement for Each Deficiency

	CONTROL	USER FEEDBACK	STIMULUS	SIMULATION	AUDIO 3D	SECURITY	USER INTERFACE
Motor deficiency
Ding et al.¹⁵		✔	✔
Palmom et al.¹⁶			✔	✔
Fiore et al.⁷	✔
Albellard et al.¹⁷		✔		✔
Carlson et al.¹⁸	✔			✔		✔
Visual deficiency
Rui et al.¹⁹		✔			✔
Alm et al.²⁰				✔			✔
Cheein et al.²¹			✔		✔		✔
Mulloni et al.²²		✔			✔
Lokuge et al.²³				✔
Rehabilitation
Rodriguez et al.²⁴			✔	✔
Maver et al.²⁵		✔	✔	✔			✔
Niniss et al.²⁶				✔
Harrison et al.²⁷				✔		✔	✔

3D, three-dimensional.

Case Study

The patient is 33-years-old, suffered a car accident on returning home after a day's work. He was diagnosed with cranial polytrauma in the presence of a loss of encephalic mass in part of the parietal lobe. Consequently, speech loss, neck movements down, and vision completely compromised. The only means of communication with the patient is through blinking.

System Prototype

A VE for locomotion training for blind wheelchair users was developed, using hardware, software, and computational techniques to improve interaction and inclusion of the patient. For this adaptation, it is necessary to investigate the requirements for the construction of the prototype; according to Grant (2004), the architectures are derived from the main components as visual simulation, physical simulation, and control¹³ (Figure 1).

Fig. 1.

Overview of virtual brainy chair.

The proposal of this work promotes a pretraining on the bed, and as this advances, it changes to being performed in the wheelchair. The visual simulation consists of providing a simple or complex model depending on what is most appropriate, thus providing criteria and applying the necessary techniques to the prototype.

Architecture

The architecture is divided into three layers: hardware layer, Emotiv Epoc Application Programming Interface (API) and user interface. In the first layer, hardware architecture, are the devices responsible for capturing, acquiring, and executing commands for movements. The second Emotiv Epoc API layer performs the processing of captured raw EEG signals (Figure 2).

Fig. 2.

Solution architecture.

There is a sublevel with a wrapper for communication with the Arduino microcontroller, which has the logic to feed (force) and perform the locomotion activities next to the motor for wheelchair movements to occur. The connector receives the information generated by the user to drive the motors located simultaneously in the lower right and left hand side of the wheelchair, defining the speed and direction of rotation of the motor. The user interface layer is where the control for movements and application of the techniques of audio three-dimensional (3D) is implemented, so that the actions in the VE are reproduced. The integrations of these three layers formed a single block, the project is composed of three primary axes: (1) EEG capture, (2) classification and sending of commands, and (3) replicate movement in the VE and in the wheelchair. In addition, the Logitech surround sound speakers Z506 delivers 75 W of power with 3D stereo, as shown in Figure 3.

Fig. 3.

Functional architecture.

Virtual Environment

The VE was designed for the training of blind wheelchair users in the process of locomotion, and navigation takes place first through 3D sound orientation to indicate what will be the course and the movement to follow, so the user must perform a facial expression, in order that the required movement occurs (Figure 4).

Fig. 4.

General bird's eye view of the VE. VE, virtual environment.

The training sessions were designed to be conducted on public roads or sidewalks around the house, with obstacles and interactions to simulate locomotion. The movements are guided by 3D sound instructions to the patient and triggered by the brain signals based on the facial expressions and then by means of a wheelchair avatar using the wheelchair in the VE, as shown in Figure 5.

Fig. 5.

Three-dimensional model of patient in the VE.

Facial Expression and Movements

The solution developed will translate Expressiv Emotiv Epoc API results into commands to move the virtual avatar and wheelchair. Table 2 lists the commands based on facial expressions.

Table 2.

Commands Based on Facial Expressions

ACTION	FACIAL EXPRESSION
Forward	Blink (2 × )
Backward	Raised eyebrows (2 × )
Right	Look right (2 × )
Left	Look left (2 × )
Stop	Clenched teeth (2 × )

Experiments

The brainy virtual chair project aims at enabling the user to move independently in the VE, with the purpose of going around the house along the sidewalk, as shown in Figure 6.

Fig. 6.

Route in VE.

The experiment was performed with a blind patient in two stages: No. 1, in bed (Fig. 7); No. 2, wheelchair (Fig. 8). The VE is used in both situations, using 3D sound techniques. To evaluate the performance of the system and verify whether the patient is able to perform the training, sessions are performed in the VE, accompanied by a health professional.

Fig. 7.

Simulation in progress, with patient on bed.

Fig. 8.

Patient preparing for starting a trial session.

The patient will move through the VE using Emotiv Epoc, making the facial expressions as described in Table 2. In this first part of the session, the circuit is activated with the patient on the bed, and soon after completion, the user initiates phase 2 of the training with the wheelchair. In Figure 8, the patient is shown prepared to begin the training with the wheelchair, and then beginning the use of the adapted wheelchair.

Results and Discussion

This work is qualitative, characterized as a descriptive study with empirical basis, systematic observation being used as a data collection instrument, based on Clemente et al.¹⁴ The experiment began by measuring the communication time between the collections of brain waves to the movement represented in the VE.

EEG–Wheelchair Interactions

The integration of EEGs in wheelchair control aims to reach a high level of data related to usability, this to compare the BCI interaction allowing for satisfactory control and experience. Table 3 demonstrates that Emotiv presents a false positive for detection (error 5%) for detections of facial representation, against 2% for Emotiv API; this result, however, does not compromise the experiment, as each movement can be validated by a visually validated facial expression.

Table 3.

Electroencephalography–Wheelchair Interactions

	TIME NEEDED FOR SIGNAL CAPTURE (S)	ERRORS OR FAULTS FOR EXECUTION (%)	SYSTEM CONTROL/FEEDBACK
Emotiv Epoc	1	5	External
API	0.5	2	Dynamic
Virtual environment	3	1	Forcing
Wheelchair and Arduino	5	0.1	Robotic

API, application programming interface.

Virtual Environment

The VE presented several modifications during the pre-established conceptions, but for such a comparison, two tests were carried out contemplating the VE and the wheelchair, according to Table 4.

Table 4.

Results of Sessions

TRIALS (NO.)	EEG CLASSIFIED CORRECTELY	USING REAL WHEELCHAIR	AVERAGE TIME NEEDED (MIN)	NUMBER OF COLISIONS (VE)	INTERVENTIONS FOR INSTRUCTIONS	ERRORS OF SYSTEM/PATIENT	MOVEMENTS ACCURACY (%)
Prototype No. 1—Initial tests
1. Trial (session 1)	57	No	21	43	18	1/9	61
2. Trial (session 1)	63	Yes	25	57	8	1/6	65
Prototype No. 2—Included more instructions
3. Trial (session 2)	84	No	18	24	2	0/1	82
4. Trial (session 2)	87	Yes	17	16	3	1/2	85

EEG, electroencephalography; VE, virtual environment.

In the first session, we noted that the patient had a low level of correct EEG classification due to the lack of step-by-step instructions to guide each movement in a more detailed way. For this reason, it was evidenced that they had ∼62% accuracy level for the required level of assertiveness. Trial No. 2 presented better results especially due to a better implementation of audio instruction commands in the VE. The training was conducted with timeframe of 30 days for these testing and duration of ∼60 minutes per session during 2 months. Before beginning actual test sessions, 12 training sessions were conducted relative to trial periods of 30 days. This level of practice could be improved with sessions one time per week that will improve accuracy and memory of the user.

Patient and Doctor Results

The vital signs were considered, wherein it was possible to use these as an instrument to measure the impact on the patient's emotional and physical factors. According to the results of the physiological data, those readings taken after the training sessions show an increase in the vital signs of the patient as given in Table 5.

Table 5.

Vital Signs Collected During Trials

	SESSION NO. 1	SESSION NO. 2
Before session
Temperature	36°C	36.1°C
Heart rate	78	86
Blood pressure	13/9	12/9
After session
Temperature	36.4°C	36.5°C
Heart rate	93	104
Blood pressure	13/9	14/10

During the experiment, AF3, F3, F7, F4, T7, and T8 channels were those with the highest activity and represented that these are responsible for the reasoning and control of movements, as given in Table 5.

Conclusion

In this work, a computational system was developed involving hardware, software, and computational techniques to improve integration in the training of visually impaired wheelchair users. In the case study evaluated, it was demonstrated that the patient was able to test the tool and perform clinical practices. The test subject was still able to walk through predefined actions through use of the wheelchair and reproduce the actions trained in the VE. This research demonstrates, through an experiment, the use of a wheelchair adapted to have the support of VR and EEG for training of locomotion and individualized interaction of wheelchair users with visual impairment. The objective was to provide efficient interactions, thus allowing the social inclusion of patients who are considered otherwise incapacitated. This project was based on the following criteria: natural control, feedback, stimuli and safety. A multi-layer computer rehabilitation system was developed that incorporated natural interaction supported by EEG, which activated the movements in the Virtual Environment and real wheelchair through adequately performed experiments. This research consisted of elaborating a suitable approach for blind wheelchair user patients. The results demonstrated that the use of Virtual Reality with EEG signals has the potential for improving the quality of life and independence of blind wheelchair users.

Footnotes

Disclosure Statement

No competing financial interests exist.

References

Fiegenbaum

. Accessibility in the school context making inclusion possible. [In Portuguese]. Master Thesis. Special Education Department, Federal University of Rio Grande do Sul, Porto Alegre, Brazil, 2009. Available at: http://www.lume.ufrgs.br/handle/10183/33297

Aguiar

. Public school and the accessibility dilemma. [In Portuguese]. Master Thesis. Medicine Department of University of Rio de Janeiro, Rio de Janeiro, Brazil, UFRJ, 2014. Available at: http://www.medicina.ufrj.br/acessibilidadecultural/sitenovo/wp-content/uploads/2014/07/A-escola-publica.pdf

Brasil

. Census Primer 2010—Disabled People. Luzia Maria Borges Oliveira. Secretariat of Human Rights of the Presidency of the Republic (SDH/PR); National Secretariat for the Promotion of the Rights of Persons with Disabilities (SNPD). General Coordination of the Disability Information System. [In Portuguese]. Brasilia, Brazil, 2012.

Silva

. Learning and accessibility of students with physical disabilities in public schools. [In Portuguese]. Monografia de especialização lato-sensu. Medicine Department of University of Rio de Janeiro, Rio de Janeiro, Brazil, UFRJ, 2014.

Tori

, Kirner

, Siscoutto

. Fundamentals and technology of virtual and augmented reality. [In Portuguese]. Belém: VIII Symposiun on Virtual Reality, 2006.

Cardoso

, Lamounier

, Kirner

, Kelner

. Technologies and tools for the development of virtual and augmented realitysystems. [In Portuguese]. Universitaria Press. Brazil: Federal University of Recife, Brazil, 2007.

Fiore

, Coben

, Merritt

, Liu

, Interrante

. Toward enabling more effective locomotion in VR using a wheelchair-based motion platform. Joint Virtual Reality Conference of EGVE—EuroVR 13, 2013.

Berretta

. Ambientes. Virtual environments to assist the development of spatial cognition in the blind: a natural interaction approach. [In Portuguese]. PhD Thesis, Department of Electrical Engineering, Federal Univesity of Uberlândia, UFU, Brazil, 2015.

Costa

, Oliveira

. A study about brain-computer interface. [In Portuguese]. Computer Engineering Department, University of Brasilia-UNB, Brazil, 2012.

10.

Folane

, Autee

. EEG based brain controlled wheelchair for physically challenged people. Int J Innovative Res Comput Commun Eng, 2016; 4.

11.

Bagacina

, Bedaño

, Goy

, Oppus

, Tangnan

. Peripheral control using EEG signals and facial artifacts. Philippines: Department of Electronic, Computer and Communications Engineering, University Quezon City. 2014.

12.

Salatin

, Grindle

, Wang

, Vazquez

, Cooper

. Design and development of the personal mobility and manipulation appliance. Assist Technol, 2012; 23:81–92.

13.

Grant

, Harrison

, Conway

. An haptic interface for wheelchair navigation in virtual worlds. Cambridge: Presence, MIT Press, 2004.

14.

Clemente

, Rodríguez

, Rey

, Alcañiz

. Assessment of the influence of navigation control and screen size on the sense of presence in virtual reality using EEG. Expert Syst Appl J, 2014; 4:1584–1592.

15.

Ding

, Parmanto

, Karimi

. Design considerations for a personalized wheelchair navigation system. Proceedings of the 29th Annual International. France, 2007.

16.

Palmon

, Oxman

, Shahar

, Weiss

. Virtual environments in design and evaluation. Laboratory for Innovations in Rehabilitation Technology, Department of Occupational Therapy, University of Haifa, Israel, 2011.

17.

Abellard

, Randria

, Abellard

, Khelifa

, Ramanantsizehena

. Electric wheelchair navigation simulators: why, when, how?. Université du Sud Toulon/Ecole Supérieure Polytechnique d'Anatananarivo. France/Madagascar, 2012.

18.

Carlson

, Swan

, Stredney

, Blostein

. The application of virtual wheelchair simulation to the determination of environmental accessibility and standards compliances in architectural design. Proceedings of the Symposium on Computer & Innovative Architectural Design, The 7th International Conference on Systems Research Informatics & Cybernetics. Baden-Baden, Germany, 1994.

19.

Rui

, Yuanqing

, Yongyong

, Hao

, Shaoyu

, Tianyou

, Zhenghui

. Control of a wheelchair in an indoor environment based on a brain. 2013; Computer Interface and Automated Navigation. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2016.

20.

Alm

, Arnott

, Newell

. Prediction and conversational momentum in an augmentative communication system, vol. 35, no. 5. USA; ACM, 1998:46–56.

21.

Cheein

, Cruz

, Filho

, Carelli

. Maneuverability strategy for assistive vehicles navigating within confined space. Int J of Adv Robotic Syst. 2011; 62–75.

22.

Mulloni

. Experiences with the impact of tracking technology in mobile augmented reality. MobileHCI MobiVis, 2012.

23.

Lokuge

, Madumal

, Kumara

, Ranasinghe

. Indoor navigation framework for mapping and localization of multiple robotic wheelchairs. ISMS 2014: Proceedings of the 5th International Conference on Intelligent Systems, Modelling and Simulation, 2014.

24.

Rodriguez

. Development of a wheelchair simulator for children with multiple disbilities. Laboratoire de Informatique–Robotique et Microéléctronique de Montoellier, 2015.

25.

Maver

, Harrison

, Dall

, Granat

, Grant

, Conway

. Development of a wheelchair virtual reality platform for use in evaluating wheelchair access. BioEngineering Unit, Rottenrow, University of Strathclyde, Glasgow, United Kingdom, 2000.

26.

Niniss

, Takenobu

. Electric wheelchair simulator for rehabilitation of person with motor disability. National Rehabilitation Center for Person with Disabilities, Japan, 2009.

27.

Harrison

, Conway

, Grant

. Wheelchair simulation. Telepresence Research Group–University of Strathclyde, Glasgow, United Kingdom, 2012.