An assistive IOT smart control system for people with severe mobility disabilities

Abstract

According to Dr. Margaret Chan [29], approximately 15% of the world’s population faces significant challenges in life due to severe disabilities. The majority of these individuals have mobility impairments, significantly impacting their mobility and self-care abilities. Two main groups affected by these disabilities are individuals with Amyotrophic Lateral Sclerosis (ALS) and those who have had a stroke. In a study referenced as [28], it was revealed that ALS accounts for 80% of cases involving severe mobility impairments. The incidence rates of ALS and stroke are reported to be 4/100,000 people and 600/100,000–1,000/100,000 people, respectively. In Vietnam, the ALS incidence rate is approximately 2–3 cases per 100,000 people, while over 200,000 individuals suffer from strokes each year. Roughly 25–30% of individuals with mobility impairments are at risk of experiencing communication difficulties, which can manifest as speech difficulties, slurred speech, or aphasia [20]. In addition to limited mobility and communication challenges, interacting with the living environment poses a significant obstacle for these individuals. Therefore, in addition to assistive communication systems, it is crucial to develop an effective and user-friendly control system for home IoT devices which enable people with severe mobility impairments to gain greater independence in their daily lives. This study presents the development and testing of an assistive IoT smart control system that allows individuals with severe mobility impairments to control familiar electronic devices using their eyes and brain.

The test results demonstrate the realistic and feasible nature of the proposed system. Users rated the system above 2.78 out of 5.00 according to the HMI questionnaire. The score for attitudes, which reflects users’ trust in the system, was 3.76 out of 5.00. Therefore, the proposed approach holds promise in assisting individuals with severe mobility disabilities to effectively interact with the IoT devices in their surroundings.

Keywords

Smart assistive product assistive communication smart IOT control

1. Introduction

In the World Health Organization’s World Report on Disability in 2021 [29], statistical data was compiled to assess the number and situation of individuals with disabilities. The objective was to develop strategies for improving treatment options and providing services that enhance the quality of life for people with different disabilities. Beside the need for treatment, there is a significant demand for assistive technologies (ATs) to support individuals with disabilities in Vietnam and worldwide. Consequently, people with disabilities require assistance at various levels. Communication and interaction are crucial areas where support is needed, as they enable individuals with disabilities to express their needs and integrate into the community. Physical disabilities, such as speech and language impairments, deafness, blindness, and motor impairments, often necessitate the use of non-invasive assistive devices. For individuals with speech and language impairments, alternative methods such as text or sign language can be used for communication. Hearing aid systems are commonly used by individuals who are deaf, while Braille systems meet to the needs of the blind. Wheelchair systems and robots are employed by individuals with mobility impairments. However, there is a lack of dedicated assistive devices for individuals with multiple disabilities, particularly those with limited motor function and an inability to use verbal communication. Recognizing that eye contact communication can be a viable option for individuals with severe mobility impairments and no verbal communication, our research team has developed an Augmentative and Alternative Communication system (AAC system) for Vietnamese individuals with severe mobility disabilities and difficulties in verbal communication [15], utilizing eye signals as input.

In addition to the fundamental need for communication to interact with others and express personal needs, individuals with disabilities also require the ability to interact with and control household devices in order to enhance their independence and integrate into family life. According to [1], there has been a growing focus on assistive technology over the past three decades. As a result, individuals with disabilities are increasingly placing their trust and reliance on advanced technologies to increase their independence and expand their interactions in daily life. Besides, controlling and mastering home appliances not only meets physiological needs but also helps individuals with disabilities adapt to their living environment, fostering a sense of optimism and self-reliance. In recent years, there has been significant attention from scientists and developers towards research on smart home applications for individuals with disabilities. Many of these studies focus on controlling smart homes through touch interactions or voice commands. For instance, Paul Panek et al. [21] developed a method that utilized a head-mounted hat with a handle to control smart devices via a computer keyboard, which yielded positive results. Users were able to control the handle by using their head to type on the keyboard. However, this approach is not suitable for individuals who have limited neck and head mobility. Ali I. Hussein et al. [7] proposed an IoT system for individuals with disabilities based on voice control. They employed a CNN network to identify and warn of fire hazards, while an RNN was used to learn and provide recommendations for daily repetitive tasks such as taking medicine, showering, turning on lights, and operating the television. The accuracy for predicting daily repetitive task recommendations was 80%, while fire prediction accuracy reached 95%. Commercial products like Alexa and Google Home have also served as control centers for smart homes, receiving commands and controlling other devices. Progress Mtshali and Freedom Khubisa [18] developed a smart home appliance control system specifically designed for individuals with disabilities and the elderly. They utilized advanced technologies such as smart plugs, smart cameras, Amazon Alexa, Google Home, and Apple Siri to enable users to control the smart home system through voice commands. Most of the aforementioned studies utilized input methods such as voice, keyboard, and mouse. However, there is a lack of research suitable for individuals with severe mobility and verbal impairments. Recently, Wu et al. [30] developed a wireless home assistive system (WHAS) to assist people with severe impairments in communicating with others and controlling home devices. They focused on Morse code translation to convert various types of input signals into different output modes of assistive context-aware toolkit, keyboard mode, mouse mode, and pad mode. Experimental results showed an accuracy of approximately 92% in typing tests for communication and control of IoT devices. The insertion error, deletion error, and substitution error were reported as 1.71%, 2.99%, and 5.13%, respectively. They employed diverse assistive input methods such as sensors (EOG, Touch), switch signals, and eye signals with different types of users, including those with severe mobility impairments. However, the system was noted to be more complex, and the human-machine interface of WHAS featured all device control buttons on a single window, resulting in limited control buttons.

In this paper, we present the development of an assistive IoT smart control system designed to support individuals with severe mobility impairments and difficulties in voice communication (referred to as “users” hereafter) in controlling IoT devices. The system primarily utilizes eye tracking and ElectroEncephaloGram (EEG) signals (ET/EEG signal), but it can be expanded to accommodate other forms of input, such as voice, mouse, and keyboard, in order to meet the specific needs of disabled users. The ET/EEG signal processing module is connected to an IoT gateway, through which users interact with an interactive module to control IoT devices. Our objective has been to create an innovative technical system that empowers individuals with severe mobility disabilities and their caregivers to benefit from state-of-the-art smart home technology. The contributions of our work can be summarized as follows.

The proposed system has been developed with the aim of wide usability, utilizing eye tracking and EEG signal communication. It supports most low-cost eye trackers that are affordable for the majority of users. Furthermore, the system is designed to be flexible, allowing for easy expansion with other communication methods such as keyboard, mouse, and voice.

Rather than relying on complex and cumbersome equipment, our proposed system introduces an IoT gateway that is extremely small, lightweight, and portable, offering enhanced convenience for users.

The proposed system can be seamlessly integrated into any existing assistive eye tracking-based communication system.

The remaining sections of this paper are organized as follows. Section 2 provides a background on the development of the system, covering topics such as the application of ET/EEG signals, smart IoT assistive technology, and the human-machine interface. Section 3 presents the details of the proposed system. Section 4 presents the discussion of a series of experiments, along with relevant discussions. Finally, Section 5 concludes the paper.

2. Theoretical backgrounds

Individuals with severe mobility disabilities who are unable to communicate verbally but have intact cognitive abilities can utilize ET/EEG signals for communication and interaction. The user interface designed for them should be comprehensive, simple, and tailored to their physical condition. Similarly, the IoT gateway should be compact and suitable for the user’s living space. In the following sections, we present the relevant theoretical foundations associated with these aspects. Furthermore, we analyze the basis for constructing questionnaires and outline the evaluation method employed for the proposed system.

2.1 ET signal and its applications

As mentioned in [1], assistive applications encompass various domains such as Cognition, Communication, Environment, Hearing, Mobility, Self-Care, and Vision. Our assistive IoT smart home control system falls under the category of Communication, aiming to support individuals with mobility impairments in controlling IoT devices using their eyes and brain. Additionally, the system still retains the option of using mouse input as usual. The proposed system is built upon eye tracking technology which is an external device to the human body. Eye movement serves as a crucial real-time input medium for human-computer communication, together with traditional methods like keyboard, mouse, and voice. This is especially significant for individuals with physical disabilities, including those with limited mobility and speech capabilities. Eye tracking technology, which has gained importance in user interaction, relies on an eye tracker to measure eye movement and position. Research on eye tracking techniques primarily focuses on integrating eye movements into multimedia communication with computers in a convenient and natural manner. In [16], the authors designed a system that simulated mouse right/left-click actions by blinking the left or right eye, thereby controlling the mouse cursor. In [19], researchers developed a mouse that utilized eye and nose movements, enabling users to communicate with the computer. Masta Lomaster et al. [14] created a low-cost prototype of an eye control system compatible with popular commercial eye trackers such as Tobii EyeX and Eye Tribe. Zhang et al. [31] proposed an eye tracking-based control system for user-computer dialogue, combining mouse and keyboard functions to achieve almost all computer inputs. Minh Hoa Nguyen et al. [15] introduced an augmentative and alternative communication system for individuals with severe mobility and speech disabilities in Vietnam. The system, which is eye-controlled and capable of converting text into speech, underwent evaluation with three groups of subjects. Users expressed satisfaction when using the system at home, achieving average typing speeds of 22.1 and 26.3 characters per minute with average error rates of 1.3% and 0.8%, respectively. In these studies, two main approaches were employed to select keys on virtual keyboards using gaze. The first approach involved scanning each key on the screen, with the user signaling the system to select the desired key during the scanning process. While this scanning method achieved accuracy, it incurred a significant waiting time for the desired key to become available. The second method allowed users to randomly select on-screen keys by using gaze to determine the position and signal the system to recognize the selected key. Eye blink signals were used to confirm key selections.

2.2 EEG signal and its applications

A Brain-Computer Interface (BCI) system has been developed to enable individuals to control or communicate with computers or machines using EEG signals [27]. BCI technology offers potential communication and control options in human-machine interaction. The BCI system consists of three components:

(i)
Brain signal acquisition: Specialized hardware is used to measure brain activity, and dedicated software is employed to capture data from the hardware.
(ii)
EEG signal processing: Various machine learning and deep learning methods are employed to process the EEG signals.
(iii)
User feedback presentation and model activation: The processed information is presented to the user as feedback, and the BCI system is activated accordingly.

BCI systems have been gaining increasing attention from researchers. Some significant examples of completed BCI systems include BCI2000 [25] and OpenViBE [24]. BCI2000 has been used to develop systems that assist individuals with severe disabilities, often resulting from conditions like ALS or locked-in syndrome, in tasks such as word processing, email communication, environmental control, and general communication. OpenViBE, on the other hand, is an applications mainly in healthcare, supporting disability aid, real-time biofeedback, neurofeedback, real-time diagnostics, multimedia development for virtual reality, video games, people-machine interactions, and other areas related to brain-computer interfaces and real-time neuroscience. In [6], the authors review and analyze the use of EEG signals in developing frameworks for dyslexia. In [23], Arrigo P. et al. present applications of EEG-based BCIs using motor imagery (MI) data for wheelchair movement and control. They primarily focus on analyzing research on motor imagery EEG data and testing the data in the context of wheelchair control.
2.3 User interface design for the disabled

The disabled, especially people with severe mobility impairment, require that the interface design be simple, easy to use and full of features. According to [17, 32], when designing an interface, it is necessary to clearly define a specific object. While implementing the design, attention should be paid to independent problems including the user space (age, background knowledge, gender, culture, and health condition), the need space (user needs to determine objective behaviour of the UI), and the application space (software solutions in Computer Vision, Natural Language Processing, Signal Processing etc.) Once the need space is identified, we would analyse and choose object identification, object detection, distance between objects and distance to the objects when designing UI. Different target groups need to use different approaches such as touch-based, vision-based and speech-based methods. In [10], authors showed desirable characteristics of an efficient UI for the disabled. They are intelligent and adaptable properties, user intention prediction, profile management, interaction tracker, behaviour pattern tracker, and cognitive learning assistance. To achieve these desired characteristics, the objects that appear on the UI need to be useful, usable, reliable, findable, accessible, and valuable. Objectives of interaction include communication, control, monitoring, and entertainment with different devices in the form of operation, text input and remote controller. With diverse representations of user interface such as menus, checkboxes, buttons, sidebars, ratio buttons, etc., each of these elements needs to be carefully designed in terms of size, shape, colour, position. Consistency in the layout across the interfaces of a product will help users easily remember when learning and using the product.

2.4 IoT gateway

The IoT gateway serves as the “brain” of a smart home system, facilitating the connection, management, and control of smart devices through user interactive software with various input options. It comprises an IoT gateway box and developed software. The design and manufacturing of the IoT gateway box can be achieved through discrete components or by connecting functional modules. In [5], the authors emphasize that the IoT gateway should encompass essential functions such as data management, resource management, quality of service management, device management, security management, and protocol transactions. To ensure seamless communication among the different components of an assistive IoT smart control system, there are several communication technologies [11] and protocols to choose from. These include Wi-Fi, Bluetooth, Zigbee, Ethernet, as well as protocols such as Message Queuing Telemetry Transport (MQTT), HyperText Transfer Protocol (HTTP), Constrained Application Protocol (CoAP), Advanced Message Queuing Protocol (AMQP), Extensible Messaging and Presence Protocol (XMPP), and Representational State Transfer (REST). Among the available protocols, MQTT and HTTP are commonly used. They offer lightweight packages, consume less bandwidth, and are less likely to become overwhelmed during delivery. Furthermore, the MQTT protocol is simple to set up and use. On the other hand, the HTTP protocol is capable of handling large data streams effectively.

2.5 Assistive product assessment

Assessments for assistive devices, which are designed for use by individuals with disabilities, involve measurements and tests conducted in the fields of ergonomics, medicine, or psychology. These assessments aim to evaluate the effectiveness, efficiency, and user satisfaction in achieving specific goals within particular environments [8]. To measure changes in body parameters during product use, specialized equipment is often employed. Additionally, there are various methods for assessing assistive devices: (i) Informal expert review: Experts in the field provide their evaluation and feedback on the device; (ii) Usability evaluation: Prototypes or final products are tested by users to assess their usability; (iii) Written surveys: Users respond to written questions prepared by the designer to gather feedback; (iv) User interviews: Designers engage in discussions with users to gather insights and feedback; (v) Focus group discussions: Wide-ranging discussions involving stakeholders, researchers, and users take place to gather diverse perspectives; and (vi) Environmental information collection: Information is gathered about the specific environments in which the assistive devices will be used.

For individuals with severe disabilities, active participation in discussions can be challenging. Therefore, the creation of Likert-type questionnaires is a practical choice for two reasons. Firstly, it allows for quick and easy selection of responses by users. Secondly, designers can create groups of questions with different objectives. Several standardized Likert scales are available, such as SUS [2], QUIS [3], UMUX [4], SUMI [9], UMUX LITE [12], USE [13], and QUEAD [26]. However, whenever a new scale is defined, it is essential to evaluate its effectiveness.

3. Our proposal of assistive IoT smart control system

In this paper, we propose an assistive IoT smart home control system integrating easily into the AAC system. The proposed system is depicted as shown in Fig. 1.

The system that enables users to control IoT devices through eye-gazing and brain activity consists of three primary components: signal receiving equipment, interactive software, and an IoT gateway. The interactive software comprises a user interface displayed on a screen, an ET/EEG signal processing module, and an interactive module. In the initial version of the system, input is obtained from eye signals captured by a Tobii eye tracker attached to the on-screen display, EEG signals from an Emotive device worn by the user’s head, and mouse signals (optional). The ET and EEG signals are collected and analyzed by the ET/EEG signal processing module, which converts them into selectable buttons on the designed on-screen interface. The interactive module is responsible for acquiring and processing the selected buttons for communication or command data for controlling IoT devices through the IoT gateway. The interface of the interactive software is designed as a grid of buttons, with button sizes and colors suitable for individuals with motor impairments. For convenience, the IoT gateway is designed as a compact and lightweight smart box called the Smart Home Controller (SHC). The SHC is based on a Raspberry Pi 4, which houses the system management module, control module, device database, and APIs.

Figure 1.

Our proposed IoT smart control system.

Figure 2.

Main interface in Vietnamese.

3.1 ET/EEG signal processing

In addition to perceiving visual information, users in this system utilize their eyes to express their desires and needs. To facilitate this, we employ a key selection method proposed in [12], which involves utilizing the parameter of dwell and an on-screen interface. Through this method, the user selects a key on the on-screen interface by focusing their gaze on the corresponding content key for a specific dwell time. The time counter for each key starts when the user’s gaze is directed towards the key, and once the counter value reaches the dwell threshold, the key selection is confirmed. It is important to note that the dwell parameter value can be adjusted by users based on their individual health and preferences. A smaller dwell value allows for faster key selection but may come with a higher error rate.

If the user chooses to wear an Emotiv device, the EEG processing module detects and analyzes the EEG signal each time the user selects a button [33]. This analysis takes into account the user’s concentration level to expedite the selection process or promptly cancel the selection if the user hesitates. The user’s concentration level is integrated to the key selection method to reduce dwell time and enhance the speed of key selection.

3.2 Development of interactive module

We have developed an interactive module using the QT framework, along with MQTT and HTTP protocols. This module processes the selected keys from the on-screen keyboard into control commands that activate specific functions of the corresponding IoT devices. The IoT devices are divided into two groups: (i) IR control-based devices, such as air conditioners, fans, and TVs, and (ii) on/off devices. For on/off devices like lights and fans, we utilize a smart switch connected via Wi-Fi to the router for control. To design the buttons on the interactive interface for each specific IR IoT device, we learn the common and basic function

Figure 3.

Interface of light control in Vietnamese.

Figure 4.

Interface of fan control in Vietnamese.

buttons found on the remote controller of that device. The user interface, which is a component of the interactive module, enables direct interaction between the user and the system for controlling IoT devices. When designing interfaces for assistive devices, it is crucial, as mentioned in [32], to consider the specific target user group and address specialized user interface (UI) challenges. Our product targets individuals with severe motor impairments who are unable to speak. Their primary means of interaction and communication is through their eyes. Therefore, while designing the interface, we have taken their health conditions into account to provide a comfortable user experience and utilize the system’s capabilities effectively. Based on the user’s health condition, we have designed three levels of interaction (high, medium, and low) that cater to the user’s concentration and eye gaze speed. Besides, we have chosen button elements in the form of keys on an on-screen keyboard to seamlessly integrate our system with existing AAC communication systems. On the buttons, we display simple texts representing the control commands, along with familiar icons that are easily recognizable to the user. The icons adhere to principles of human-machine interaction to promote user familiarity. We have selected a default dark blue color for the buttons, which is prominent, easy to see, and has a minor impact on blood pressure management. However, we also provide users with the option to choose their desired colors, as color can be a visual stimulus, especially when it comes to eye contact. The buttons are arranged in a consistent structure and simulate the layout of the remote controller for each device, enhancing user familiarity and convenience. For example, when designing buttons for a TV remote controller, we consider the user’s habits and preferences, dividing them into functional groups (as shown in Fig. 2), such as volume control, channel selection, and digit-based channel selection. The back button is always placed on the top right side of the on-screen interface [22].

Figure 5.

Interface of TV control in Vietnamese.

Figure 6.

Interface of air conditioner control in Vietnamese.

Figure 7.

Interface of TV control in English.

The system supports two languages, English and Vietnamese (as shown in Fig. 7), to accommodate to both domestic and international users. Users can control their registered devices through the interface and have the option to select either English or Vietnamese as the language of their choice. In addition, users can customize the colors of the interface according to their preferences and select a control speed that suits their individual physical condition.

3.3 Development of IoT gateway

To develop a small, lightweight, and portable IoT gateway for individuals with motor impairments to control IoT devices in their homes, we have taken several factors into consideration to ensure safety and durability during the device’s use. These factors include: (i) Allowable temperature: We have ensured that the IoT gateway box can withstand a range of temperatures without compromising its functionality or causing any safety hazards; (ii) Power socket standard: The design of the IoT gateway box accommodates the standard power sockets commonly used in homes, ensuring compatibility and ease of use; (iii) Voltage standard: The IoT gateway box is designed to support the standard voltage levels found in residential settings, providing a safe and reliable power supply for the device; and (iv) Electromagnetic compatibility: Measures have been taken to ensure that the IoT gateway box has good electromagnetic compatibility, minimizing any interference with other electronic devices in the surrounding environment.

For our project, we have utilized the Raspberry Pi 4 as the main component and deployed a related software system to create the IoT Gateway (as shown in Fig. 8). The Raspberry Pi 4 was chosen for its stability and portability. To enable the transmission of infrared (IR) signals to devices controlled by remote IR controllers, we added an infrared transmitter module (T) and an infrared receiver module (R) to the Raspberry Pi 4. All of these components were enclosed in a grey aluminum alloy case. Inside the case, the Raspberry Pi 4’s processor, along with a heatsink, is securely fitted. The heat generated by the active Raspberry Pi is transferred to the case, effectively turning it into a passive heatsink. Additionally, a 30 mm integrated circuit board fan is attached to the case to provide direct heat sinking and further enhance cooling. To facilitate user convenience, a power button is mounted on the case, enabling easy on/off control of the device.

Figure 8.

IOT gateway components.

We have designed a Smart Switch that is capable of automatically controlling the opening and closing of electricity. Users can easily control electrical devices through simple operations such as touching touch keys or controlling via an IoT gateway using capacitive touch technology integrated with Wi-Fi wireless technology. The Smart Switch consists of four components: the ESP8266 Wi-Fi Module, a 3 Channel 10A Optical Isolation 5V Relay Module, a Touch Sensor Module, and a Power Module. The ESP8266 Wi-Fi Module is a popular development kit based on the ESP8266 Wi-Fi SoC chip, known for its user-friendly design. It is widely used in IoT applications that require Wi-Fi connectivity, data collection, and control. The Arduino compiler can be directly used on the module for programming and code uploading. The Touch Sensor Module receives touch signals from the user and converts them into digital signals compatible with the ESP8266 Wi-Fi Module. The ESP8266 module then recognizes and issues commands to control the corresponding relay connected to the on/off devices. The Relay Module is responsible for receiving control signals from the IoT gateway via Wi-Fi, rather than touch signals. The Power Module is used to switch between different voltages and to separate the AC powered circuit from the DC powered circuit.

The IoT gateway software includes device management and control modules, such as the user manager, IR device control and management, and a device database for storing data of controlled smart devices like TVs, lights, fans, and air conditioners. We have developed APIs for data exchange and communication between the interactive module, IoT devices, and among IoT devices themselves. The control module, an embedded module, enables the control of TVs and air conditioners via infrared waves. The system management module is used for managing IoT devices and their attributes, such as adding, deleting, and modifying devices. For IR device control, we utilized the open-source LIRC (Linux Infrared remote control) libraries, which allow users to receive and send infrared signals to our Raspberry Pi. We also developed a module for building new IR signals for IR device models that are not listed in the LIRC library. The software backend associated with the IoT gateway includes a database of devices and APIs. The backend consists of a relational database based on MariaDB (called device database), which includes tables such as Device_table, Device_data_table, Supported_device_table, and Device_command_table. The Device_table manages information of registered devices in the system, while the Device_data_table stores signals collected from device controllers or sensors. The Device_command_table contains information about the mapping between controlling commands and theirliteral functions. The Supported_device_table stores information about the physical device’s serial or model used in the IR signal recording process. The management backend includes several APIs: authentication API, manage device API, manage device command API, manage device data API, manage supported device API, and manage LIRC config API. These APIs allow the system to query the device database and support the use of the LIRC library for acquiring and processing IR signals. In cases where a remote configuration file cannot be found in LIRC’s database, we have developed a remote-control command learning function. For on/off control devices, the system receives control commands from the interactive software and sends them directly to the Smart Switch.

Briefly, our IoT gateway system and smart switch provide convenient and efficient control of electrical devices while incorporating advanced features such as touch technology, Wi-Fi connectivity, and infrared device control.

3.4 Method of assessment

We have decided to employ a questionnaire for the usability evaluation of our product, targeting both users and individuals closely related to them (such as family members, caregivers). This method offers a convenient way to gather feedback from a wide range of users and related individuals, allowing for a comprehensive review of the product from different perspectives. The questionnaire called HMI (Human Machine Interaction) includes questions not only for the direct users but also for relevant people, as family members often observe the users using the product and can provide valuable insights. The questionnaire is designed using a Likert scale with a 5-point rating system. It consists of 30 questions categorized into five groups: Perceived Usefulness (PU), Perceived Ease of Use (PEU), Emotions (E), Attitude (A), and Comfort (C). Once users have had an opportunity to experience the product, we will collect their feedback through the questionnaire.

Given the relatively small number of ALS patients compared to the overall population, conducting experiments exclusively within this patient group can be challenging due to limited opportunities. Therefore, we are considering gathering feedback from a second group consisting of healthy individuals. Participants in this group will be instructed to simulate the restricted mobility and speech abilities experienced by ALS patients during the experiment. By collecting response data from this second group, we can evaluate the reliability of the HMI questionnaire and assess the degree of intercorrelation among its questions. This approach will enhance our understanding of the usability and effectiveness of the product, enabling us to make informed improvements based on the feedback received.

4. Experiments and discussion

4.1 Test scenario

Two groups of subjects (refer to Fig. 9) were involved in the experiment. Group 1 consisted of two ALS patients, one male and one female, who experienced limited mobility and speech impairments. Based on their medical records, these users had (i) vision in the range of 5–10, (ii) mental clarity, and (iii) paralysis of limbs. In order to broaden the assessment of our system, we also formed Group 2, which comprised 27 individuals in good physical health. Prior to conducting each experiment, we provided an explanation of the system and asked the subjects to imagine themselves in the role of a patient who is unable to speak or move. The subjects did not exhibit color-blindness.

For our experimental setup, we utilized the SHC as mentioned earlier. The software programs offered basic functionalities, including the control of various smart electrical devices within a household such as televisions, air conditioners, fans, and lights. Users were also given the option to customize the language and color of the program interface. From the main interface, users could select any registered device for control while accessing the configuration interface to choose their preferred language, color, and speed settings.

Table 1
Test scenario steps

Steps	Names	Details
1	System introduction and manual explanation	Users are given an introduction to the system
		Users are provided with an explanation of the user manual.
2	System experiment participation	Users open main interface
		Users config setting interface
		Users switch the light On/Off
		Users switch the fan On/Off Users cycle three fan states (low, medium, high)
		Users switch the TV On/Off Users change the TV channels Users adjuste the TV volume
		Users switch the airconditionner On/Off Users change the airconditionner temperature Users change the airconditionner modes
3	Questionnaire completion	Users complete the HMI questionnaire

Figure 9.

Subjects in group1 (left), in group2 (right).

The experimental task encompassed three main components: program interface customization, device control, and questionnaire completion. The task was carried out according to the following process:

Step 1: System introduction and manual explanation. Step 2: System experiment participation. Step 3: Questionnaire completion.

Figure 10.

Assessments of subjects in group 1.

In System Experiment Participation Step, users actively engaged with the system and performed the experiment. They were instructed to control electronic devices in the room sequentially. The recommended control order was as follows: light, fan, TV, and air conditioner. However, the actual control order was flexible and dependent on the user’s specific needs. The light device served as an on/off device, and users were required to switch it on/off. The remaining devices, namely the fan, TV, and air conditioner, were IR devices. Users were tasked with turning them on/off and modifying their states. More specifically, for the fan device, users were asked to cycle through three states: low, medium, and high. For the TV device, they were instructed to change channels, select a specific channel, and adjust the volume. For the air conditioner device, users were prompted to adjust the temperature and operating mode. Finally, following their interaction with the system, users were directed to complete the questionnaire, providing their responses and feedback. By following this task process, we aimed to evaluate the system’s usability, user control experience, and gather valuable insights from the participants.

4.2 Results

All subjects successfully completed the evaluation within the designated timeframe. The evaluation methods included the utilization of an electronic version of the HMI questionnaire. Each subject was provided with a questionnaire comprising 30 task-related questions. Participants responded to these questions by selecting a rating on a 5-point scale (ranging from strongly disagree to strongly agree, represented by values 0 to 4). Feedback from subjects in Group 1 (Fig. 10) regarding various aspects of the HMI questionnaire was predominantly positive, with the exception of ease of use and comfort. All subjects highly valued the system’s usefulness, as it contributed to their increased self-sufficiency in self-care. They expressed a willingness to utilize the product on a daily basis and recommended it to others within their community. The assistive IoT smart control system demonstrated high reliability and usefulness for both of the subjects.

Table 2
Means and Cronbach’s $\alpha$ of Group 2’s feedbacks

Scale	PU	fPEU	E	A	C	HMI
Mean	3.16	3.13	2.78	3.76	3.56	2.87
Cronbach’s $\alpha$	0.97	0.84	0.78	0.82	0.74	0.86

$\alpha\geqslant$ 0.7 acceptable; $\alpha\geqslant$ 0.8 good; $\alpha\geqslant$ 0.9 exellent.

Figure 10 also indicates that while the comfort aspect received relatively positive ratings, it did not reach the highest levels. This suggests that users still experienced some degree of discomfort when using the system. Due to the question in Group C, this discomfort could be attributed to the fact that users control the devices using their eyes, leading to eye strain during prolonged use. Moreover, if an emotive device is worn, users may find it cumbersome. Another factor contributing to user discomfort is that during the experiment, they were required to continuously control four devices, whereas in real-life scenarios, people only interact with devices as needed. However, these observations provide valuable suggestions for us to enhance the interface, develop a customizable interaction system tailored to each user, improve the flexibility of on-screen controls, and improve speed key selection on on-screen. Generally, the evaluation results reflect the positive reception of the system by the subjects, highlighting its utility and potential for further refinement to enhance user comfort and satisfaction.

In Group 2, we utilized the HMI questionnaire to measure perceived usability and calculated the mean values for different aspects, along with the Cronbach’s Alpha coefficient (Cronbach’s $\alpha$ ). The highest value of Cronbach’s $\alpha$ obtained was 0.97, which indicates excellent consensus among the subjects who evaluated the questions related to Group PU for assessing the system. This high value confirms the questionnaire’s reliability. Moreover, the Cronbach’s $\alpha$ coefficient for the HMI questionnaires was found to be 0.86, indicating good internal consistency.

The experimental results, presented in Table 2, demonstrate the subjects’ positive reception of the system in Group 2. They expressed appreciation for the system’s usefulness, convenience, and ease of use. The findings further support the effectiveness and user satisfaction associated with the system’s attributes as perceived by the participants in this group.

5. Conclusion

In summary, our efforts have resulted in a successful and well-received smart IoT control system, contributing to improved quality of life and independence for individuals with specific needs.

We have developed a unique proof-of-concept smart IoT control system that is both cost-effective and efficient. The interactive interface plays a crucial role in this system as it enables users to effectively communicate with the system, allowing them to utilize its functions seamlessly. User reviews indicate that the system provides a positive and enjoyable experience, with users expressing satisfaction and positive emotions. The comfort level, as rated on a 5-point Likert scale, ranged from agreement to strong agreement, indicating that the interactive interface has been well-designed.

Feedback from subjects in both groups, as obtained through the HMI questionnaire, was overwhelmingly positive. Users confirmed that the system was useful and easy to use, further highlighting its effectiveness and user-friendliness.

The SHC system deserves high-quality smart IoT control devices to empower its users, enabling them to lead more independent and engaged lives within their families and communities. We have successfully developed a compact and lightweight SHC device tailored specifically for individuals with mobility and speech impairments. The device is designed to be affordable, flexible, and easily expandable. A comprehensive engineering analysis was conducted to ensure that the device meets safety and durability standards. The software functions operate correctly and efficiently, and the overall system is user-friendly and easy to navigate. System testing was conducted with two groups of users, and the feedback received from them was highly positive.

Footnotes

Acknowledgments

We would like to acknowledge the project “Augmentative/Alternative communication system using electroencephalogram and eye tracking signals for severe motor disabilities” for their significant support to this paper.

Appendix. HMI questionaire

Scales	Notations	Questions
Perceived usefulness (PU)	PU1 PU2	Is this system helpful in controlling the devices? Does the system assist you in controlling devices quickly/easily?
Perceived ease of use (PEU)	PEU1 PEU2 PEU3 PEU4	Do you find the system easy to use? Is it easy to learn how to use the functions in the system? Does the system interface look very intuitive, doesn’t it? Is this control system compact and convenient?
Emotions (E)	E1 E2 E3 E4	Would you like to use this system? Do you feel completely comfortable using the system? Are you feeling nervous when using the system? Is the system useful to you?
Attitude (A)	A1 A2	Are you going to use this system daily? Would you recommend this system to others?
Comfort (C)	C1 C2 C3	You don’t need high concentration when using the system, do you? You don’t need using force when working with the system, do you? Don’t you feel eyestrain while using the system?

References

World Intellectual Property Organization. Assistive technology. WIPO Technology trend; 2021.

Byjohn

. Chapter: SUS: A Quick and Dirty Usability Scale. 1st ed. In: Book “Usability Evaluation In Industry”. ImprintCRC Press. eBook ISBN9780429157011. First Published; 1996.

John

Virginia

Kent

. Development of an Instrument Measuring User Satisfaction of the Human-Computer Interface. ACM CHIi’; 1998.

Kraig

. The Usability Metric for User Experience. Interacting with Computers; 2010; pp. 323-327. doi: 10.1016/j.intcom.2010.04.004.

Gunjan

Anita

. A systematic literature review on IoT gateways. Journal of King Saud University – Computer and Information Sciences. 2021; pp. 9541-9563. doi: 10.1016/j.jksuci.2021.11.007.

Perera

Shiratuddin

Wong

. Review of EEG-based pattern classifcation frameworks for dyslexia. Journal of Brain Informatics. 2018; 5(2): 4. doi: 10.1186/s40708-018-0079-9.

Ali

Mehdi

Mirna

Walid

. Smart Home Design for Disabled People based on Neural Networks. Procedia Computer Science. 2014; (37): pp. 117-126. doi: 10.1016/j.procs.2014.08.020.

John

Timothy

Bhaskar

Mahantesh

Vidhyashree

. Assessment and Application of EEG: A Literature Review. Journal of Applied Bioinformatics & Computational Biology, 2021.

Jurek

Mary

. SUMI: The Software Usability Measurement Inventory. British Journal of Educational Technology. 2006; (24): pp. 210-212. doi: 10.1111/j.1467-8535.1993.tb00076.x.

10.

Kunal

Usman

. Critical Analysis of the User Interfaces for the Disabled Community. International Journal of Computer Applications. 2012; pp. 40-44. doi: 10.5120/5779-8165.

11.

Bansal

Kumar

. IoT ecosystem: A survey on devices, gateways, operating systems, middleware and communication. Int. J. Wirel. Inf. Networks. 2020; 27(3): pp. 340-364. doi: 10.1007/s10776-020-00483-7.

12.

James

Brian

Deborah

. UMUX-LITE: when there’s no time for the SUS. Conference on Human Factors in Computing Systems – Proceedings; 2013; pp. 2099-2102. doi: 10.1145/2470654.2481287.

13.

Arnold

. Measuring Usability with the USE Questionnaire. Usability and User Experience Newsletter of the STC Usability SIG. 2001.

14.

Masta

. bkb – a program to control keyboard/mouse with eyes. GitHub. https://github.com/MastaLomaster/bkb; 2015.

15.

Hoa

Duyen

Hung

Mao

Tuan

. An augmentative and alternative communication system for Vietnamese with severe speech motor disabilities. Published by Oxford University Press on behalf of The British Computer Society; 2021.

16.

Missimer

Betke

. Blink and wink detection for mouse pointer control. Proceedings of the 3rd; International Conference on PErvasive Technologies Related to Assistive Environments (PETRA ’10); 2010.

17.

Mona

Dhanashri

Bhagyesh

. User Experience for Person with Disabilities. International Journal of Computer Applications. 2018; doi: 10.5120/ijca2018917141.

18.

Mtshali

Khubisa

. A Smart Home Appliance Control System for Physically Disabled People; 2019. pp. 1-5. doi: 10.1109/ICTAS.2019.8703637.

19.

Nehete

Lokhande

Ahire

. Design an eye tracking mouse. International Journal of Advanced Research in Computer and Communication Engineering. 2013.

20.

Xuyên

Tòng

. Phu̇c hồi chúc năng sau tai biến ma̧ch máu não (Rehabilitation after cerebrovascular accident); NXB Y Hȯc (Vietnamese medical publisher), 2008.

21.

Paul

Wolfgang

Christian

Gottfried

. Smart Home Applications for disabled Persons – Experiences and Perspectives, 2001.

22.

Seungho

Yoojin

Kiyoshi

. Design Proposal for Smart TV Interface and Remote Controller. International Journal of Asia Digital Art and Design. 2018.

23.

Palumbo

Vera

Barbara

Nicola

. Motor-Imagery EEG-based BCIs in Wheelchairs Movement and Control: A Systematic Literature Review; 2021; doi: 10.36227/techrxiv14916537.v1.

24.

Renard

Fabien

Gibert

Marco

Emmanuel

Vincent

Olivier

Anatole

. OpenViBE: An Open-Source Software Platform to Design, Test, and Use Brain – Computer Interfaces in Real and Virtual Environments. Presence: Teleoperators and Virtual Environments. 2010; 19(1): 35-53. doi: 10.1162/pres.19.1.35.S2CID6345088.

25.

Gerwin

Dennis

Thilo

Birbaumer

Jonathan

. BCI2000: a general-purpose Brain-Computer Interface (BCI) system. IEEE Trans. Biomed. Eng. 2004; (51): p. 1034. doi: 10.1109/TBME.2004.827072.

26.

Schmidtler

Bengler

Dimeas

Campeau-Lecours

. A Questionnaire for the Evaluation of Physical Assistive Devices (QUEAD) – Testing Usability and Acceptance in physical Human-Robot Interaction. 2017; doi: 10.1109/SMC.2017.8122720.

27.

Stancin

Cifrek

Jovic

. A Review of EEG Signal Features and their Application in Driver Drowsiness Detection Systems. Sensors (Basel). 2021; 21(11): 3786. doi: 10.3390/s21113786.

28.

Tuan

Hinh

. Awaji criteria for the diagnosis of amyotrophic lateral sclerosis in Hanoi, Vietnam. Journal of Neurological Sciences. 2021.

29.

World Health Organization. World report on disabilities; 2011.

30.

Yeou-Jiunn

Shih-Chung

Chia-Hong

. Wireless Home Assistive System for Severely Disabled People. Applied Sciences. 2020; (10): p. 5226. doi: 10.3390/app10155226.

31.

Xuebai

Xiaolong

Shyan-Ming

Shu-Fan

. Eye Tracking Based Control System for Natural Human-Computer Interaction. Journal of Comput Intell Neurosci. 2017. doi: 10.1155/2017/5739301.

32.

Yashar Abbasalizadeh Rezaei, Gernot Heisenberg, Wolfgang Heiden. User Interface Design for Disabled People Under the Influence of Time, Efficiency and Costs; 2014.

33.

. Patent: Method for processing data to adjust data inputing speed in human-computer interface system controlled by eye gaze and electroencephalography data. United State. Application number: 18210342. Application type: Utility under 35 USC 111(a); 2023 [Receipt date 2023].