Fuzzy logic-based connected robot for home rehabilitation

Abstract

In this paper, a robotic system dedicated to remote wrist rehabilitation is proposed as an Internet of Things (IoT) application. The system offers patients home rehabilitation. Since the physiotherapist and the patient are on different sites, the system guarantees that the physiotherapist controls and supervises the rehabilitation process and that the patient repeats the same gestures made by the physiotherapist. A human-machine interface (HMI) has been developed to allow the physiotherapist to remotely control the robot and supervise the rehabilitation process. Based on a computer vision system, physiotherapist gestures are sent to the robot in the form of control instructions. Wrist range of motion (RoM), EMG signal, sensor current measurement, and streaming from the patient’s environment are returned to the control station. The various acquired data are displayed in the HMI and recorded in its database, which allows later monitoring of the patient’s progress. During the rehabilitation process, the developed system makes it possible to follow the muscle contraction thanks to an extraction of the Electromyography (EMG) signal as well as the patient’s resistance thanks to a feedback from a current sensor. Feature extraction algorithms are implemented to transform the EMG raw signal into a relevant data reflecting the muscle contraction. The solution incorporates a cascade fuzzy-based decision system to indicate the patient’s pain. As measurement safety, when the pain exceeds a certain threshold, the robot should stop the action even if the desired angle is not yet reached. Information on the patient, the evolution of his state of health and the activities followed, are all recorded, which makes it possible to provide an electronic health record. Experiments on 3 different subjects showed the effectiveness of the developed robotic solution.

Keywords

Gesture control human robot interaction internet of things rehabilitation robotics

1 Introduction

According to the World Health Organization, stroke is one of the leading causes of death worldwide. In fact, 15 million people experience stroke worldwide every year. Among them, 5 million die and 5 others are permanently disabled [1]. Survivors may experience general disabilities, including movement and speech difficulties, as well as the way they think and feel. Post stroke rehabilitation has a role in stimulating plasticity and preventing and treating complications. The first goal is to work on brain elasticity: it is a matter of brain stimulation so that healthy nerve networks are reorganized to ensure the largest possible role of lost neurons in cerebral necrosis. The dose of this rehabilitation still implies uncertainty, but it is now certain that we must combine intensity and repetition, because it is a real re-learning. In this context, robots are essential in the rehabilitation process to promote motor recovery after a stroke. Indeed, in recent years, robots and serious games have evolved greatly in this field in order to increase the patient’s functional recovery. Robots can step up treatment, aid patient movement and provide feedback. As for serious games, they make learning fun and stimulating, and the difficulty level in the game constantly adapts to the patient’s performance. Systems dedicated to automated rehabilitation have benefited from numerous technological advances in robotics, such as sensors, actuators and control approaches [2 –6]. In this context, we aim to take advantage of the progress made in remote control systems, gestural control, the IoT-based system design, Fuzzy logic-based decision-support systems to design a remote wrist rehabilitation solution based on a gesture control approach. Much attention was given to the development of robotics for teleoperation purposes [7 –9]. Several human robot gesture recognition interfaces have been developed [10, 11]. Vision-based telemanipulation seems to be a solution for the aforementioned problems. The Kinect camera showed excellent vision capabilities. Thus, it was used in many applications such as people assistive systems [12-14], rehabilitation and stimulation [15, 16] and game-based learning systems [17]. The Kinect device is able to recognize human movement. The recognized human patterns have been used in several computer environments to provide the corresponding skeleton coordinates such as LabVIEW [10] and MATLAB [18]. In [19], a Kinect gesture recognition approach was proposed to improve the gesture implementation process. Also, an eigenspace-based approach was introduced in [20] to identify and recognize human gestures using 3D Kinect. Furthermore, to recognize the movement of people suffering from Parkinson’s disease, Microsoft Kinect for Windows SDK was used in [14]. Another Kinect-based tracking approach was presented in [21]. This approach is mainly for pick and place application. Thus, the Kinect sensor offers excellent capabilities to be used for remote handling purposes.

In addition to the vision-based solutions, EMG was widely used in robotic applications [13 , 22–26]. A real time telemanipulation technique was presented in [22] using EMG based arm tracking and markless images. Based on EMG signals a myoelectrically controlled robotic system was proposed for elbow training assistance [13]. In [23], EMG signal was used to control in real time a robotic arm system. In [24], the authors designed a rehabilitation training system based on virtual reality and EMG feedback. A novel concept of robot therapy using EMG thresholds was presented in [25]. A review on the EMG-based control of the human upper limb motion was provided in [26].

Also, Fuzzy control was extensively used in many robotic applications [27, 28] because of its power in modeling uncertainty and vague data. In this paper, a fuzzy logic controller is employed in the proposed rehabilitation architecture in order to model the patient pain during the rehabilitation therapy. Incorporating the fuzzy controller into the developed rehabilitation system gives the system ability to adapt efficiently and reliably to changes in the rehabilitation conditions including patients’ conditions. Many researches have been focused on fuzzy logic based control schemes for rehabilitation robots. An adaptive admittance controller based on fuzzy logic was developed for 2-DoF actuated parallel ankle rehabilitation robot [29]. A PID controller based on a fuzzy inference system for rehabilitation of shoulder flexion/extension was presented in [30]. A fuzzy-impedance control law was developed to estimate the human-robot interaction force [31]. A robotic system providing a fuzzy-based adaptive assistance was presented in [32]. In contrast of the works cited above adopting fuzzy logic as a control law for trajectory tracking [29, 30] or for assistance [31, 32], the proposed solution integrates the concept of fuzzy logic into the control architecture to build a safety system that takes into account pain that could be felt by the patient. Two fuzzy reasoning systems are implemented in the control architecture. The first one aims to estimate the muscle contraction. The second one uses this estimated muscle contraction as input, among others, to estimate the pain felt by the subject.

Using gesture in control has become a trend. Indeed, tablets and smartphones have helped popularize the idea of using movements to control devices. By using IoT concept, the system parts/objects can be accessed anytime anywhere. Incorporating robots and computers in the rehabilitation process is a must as it highly increases the effectiveness of the whole process, reduces costs, and encourages patients to be rehabilitated. Remote robotic rehabilitation can be used to relieve stress in healthcare facilities by limiting patient move to clinics. The tele-rehabilitation systems will allow the therapists to remotely control and monitor their patients at home. We note that in the current situation where humanity is fighting against coronaviruses, the most used keywords are “stay at home” and “don’t touch”. The proposed solution includes these two goals: gesture control, which means no direct contact and Internet-based control, which means rehabilitation while staying at home.

The available rehabilitation systems are very large, complicated, immobile, unwieldy, and very expensive. In contrast to the previous cited works [2 –5], the proposed solution is light and portable since the objective is to design small robots with a well-defined target in order to be accessible for use at home. In addition, we present a new design where we managed to minimize the number of motors within the robot. In fact, the novelty of this design is that by using a single actuator it covers two movements around 2 different axes. Compared to [13 , 22–25], the developed system is a vision-based telemanipulation solution that avoids the inconvenience of the EMG-based solution such as signal sensitivity. Many works on robotic rehabilitation have not taken into account the pain felt by the patient in the robot control architecture [33 –35]. However, this pain can prevent a person from interacting positively with therapy. To cope with this problem, some studies suggest incorporating a current return, where the peak may reflect resistance exerted by the patient due to a feeling of pain [27]. However, this solution seems limited because the engine start already presents a current peak which can mislead the decision system. In other works, the authors rely on the detection of muscle contraction to estimate fatigue and therefore pain [36, 37]. In the other side, one cannot avoid certain pain in the advanced steps of rehabilitation. Therefore, the decision-making system must allow robot to go ahead even in presence of a current peak. In this situation, we need a combination of inputs and a table of rules. In this developed control architecture, we propose a sensor fusion to estimate the degree of pain. Indeed, the patients’ safety during the therapy is achieved by integrating a cascaded fuzzy logic controller that estimates patients’ pain. EMG signal, current sensor feedback and the position encoder feedback are used to provide the fuzzy controller with needed data to estimate pain. This measured pain is fed back into the control loop and processed to generate safe robot actions. In most of the research presented in the literature [38 –40], robotic rehabilitation is carried out via a control panel in the robot control station. Gesture control makes the control process easier and more natural. Using the gesture of control, the physiotherapist can control more than one robot at the same time. The developed solution applies a complete system that allows the physiotherapist to act by gesture from a distance to control the robot and get visual feedback. The architecture implemented based on the MQTT protocol solves the delay problem often encountered with remote control systems.

2 Methodology

2.1 Mechanical design

The mechanical structure is designed taking into account the anatomical constraints of the wrist and its range of movement presented in Figure 1. The 3D design of the robot is shown in Figure 2 while its different components are explored in Table 2. The robot is very light given its material and its small size. It is made up of 5 steel parts. Unlike similar solutions, the robot is powered by a single motor and this also has an impact on the mass of the robot. indeed, the device developed is dedicated to the rehabilitation of the wrist movement which means abduction / adduction and flexion / extension. We are therefore interested in two degrees of freedom (DoF). We offer a new solution providing 2 DoF using a single engine instead of 2. Based on a flexible structural transformation, the robot ensures the two fundamental movements of the wrist: abduction / adduction and flexion / extension. In fact, the hand support can be unscrewed and then move from the horizontal position serving the abduction/adduction to the vertical position serving the flexion/extension as shown in Figure 3. The proposed wrist robot is practical and easy to use. In fact, it is a portable device, not complicated at all and which did not need training to configure or repair it. In addition, it is controlled simply by using the gesture. The wrist robot designed is actuated by a high torque step servo motor which is controlled according to the architecture illustrated in Figure4.

Fig. 1

Abduction-adduction (a) and Flexion-extension (b).

Fig. 2

3D design of the robot.

Fig. 3

Abduction-adduction movement (a) and Flexion-extension movement (b).

Fig. 4

Wrist joint control.

2.2 Control design

The main objective of this work is to provide a robotic rehabilitation solution at home. Such solutions could motivate patients to follow their rehabilitation protocols in more comfortable situations and could relieve pressure in clinical centers.

As shown in 5, the control strategy uses several blocks such as an adaptive controller, a fuzzy logic system, an IoT-based interface, a human-machine interface. Moreover, the proposed architecture integrates several techniques such as the gesture recognition algorithm, the extraction of signal characteristics and the implementation of the MQTT protocol. More details on these different components and techniques are presented below.

Gesture Control Using the Kinect Sensor:

An RGB camera, like any webcam, provides an image representing the light reflected by the elements of the scene. The physical principle of Kinect capture is different. Kinect is a 3D video camera, that is, it provides the distance between the camera and the objects in the scene. An image depth is presented as a monochromatic image (that is to say in grayscale) which associates for each pixel a value representative of the physical distance between the point of the scene and the camera. The camera has the ability to track, at the same time, up to 6 skeletons with 25 points for each. Kinect is an active stereoscopic 3D camera with an infrared light source and an infrared camera with a resolution of 512 x 424 pixels. In addition to the stereoscopic 3D vision component, the Kinect device also has a color camera with a resolution of 1920 x 1080 pixels. For each pixel of the image, it finally provides 4 information: the 3 color components and the depth. These specifications are more than sufficient for the dedicated application. The only concern is the distance (depth) between the operator and the camera which is taken into account by the developed software as discussed in more detail in the results and discussion section.

EMG Feature Extraction:

Feature extraction is a process of selecting useful information and deleting unwanted EMG parts. In order to correctly extract the information conveyed by the EMG signal, algorithms for signal processing and pattern matching are developed, such as the extraction of characteristics and the classification of patterns. These algorithms depend directly on the characteristics used to represent the acquired signals. The main features in analysis of the EMG signal are: time domain (TD), frequency domain (FD) and time-frequency or time-scale (TS) representation [41]. In order to correctly extract the information conveyed by the EMG signal, algorithms for signal processing and pattern matching are developed, such as the extraction of characteristics and the classification of patterns. The TD features, and unlike the FD and TS features are the easiest to calculate and implement since they do not require any other additional transformation and they are extracted directly from the raw EMG signal. Indeed, the TS and FD features require additional transformations which impose a very complex computation and consequently dedicated processors [51].

In this paper, the TD features are used as inputs of the first block of the fuzzy logic controller to estimate the muscle contraction. Indeed, the EMG signal is used to extract muscle contraction which is used further to estimate patient pain during the rehabilitation session for a safety purpose. The following features (EMG-RMS, EMG-SSI, EMG-vorder, and EMG-Logdetect) are extracted within an N sample analysis time window.

The Root Mean Square (RMS) is modelled as the amplitude modulated Gaussian random process where the RMS is related to the constant force, and the non-fatiguing contractions of the muscles. $RMS = \sqrt{\frac{1}{N} \sum_{i = 1}^{N} X_{i}^{2}}$ (1) where X is the signal

The Simple Square Integral (SSI) expresses the energy of the EMG signal as a useable feature: $SSI = \sum_{i = 1}^{N} | X_{i} |_$ (2)

EMG_v-order provides an estimate of the exerted muscle force:

$v - order = \sqrt{E | X_{k} |}_$ (3) where E is the expectation operator applied on samples in one analysis window.

The log-Detector (logDetect) also provides an estimate of the exerted muscle force:

$\log detect = e^{\frac{1}{N} \sum_{k = 1}^{N} \log (| X^{i} |)}_$ (4)

Adaptive Control The developed system uses one link robot which follows the following model: $m \ddot{q} + b \dot{q} + g (q) = τ$ (5) where

m is the link mass of the one-link robot with load.

q is the joint variable.

$\dot{q}$ , and $\ddot{q}$ are the joint velocity and acceleration.

b is the dynamic coefficient of friction.

g is the gravitational force.

In general, the adaptation of a system to its environment lies in the possibility of reacting to the variations that this environment may undergo. In control theory, adaptive control aims to react to variations in the system. In the non-adaptive approach, the development of control algorithms is done by considering an invariant model. However, in the case of systems whose parameters are unknown, sophisticated control approaches, such as adaptive control, are applied to deal with this model uncertainty. Within the framework of our application and in addition to the uncertainty of the robot model, working with arms of different weights confirms this uncertainty of the dynamic model of the system. Consequently, we use this adaptive approach in order to real-time react by minimizing the error between the setpoint and the output even in the presence of system variations. Actually, the mass m presented in the model does not only represent the link mass but must take into account of the patient’s forearm and the human arm-robot coupling. This is could be considered as non-modeled dynamics. Friction could be considered as an unmodeled dynamic as well. For this reason, we adopted an adaptive controller, where we assume that m and b are unknown.

Equation 5 can be written as: $m \dot{z} = y ϑ - τ - bz$ (6) where $z = γ e + \dot{e}$ (7)

e = q_d - q is the tracking error, q_d represents the desired trajectory of the joint variable, and γ > 0. $y ϑ = m (\ddot{q_{d}} + γ \dot{e}) + b (\dot{q_{d}} + γ e) + g (q)$ (8) where y represents the regression matrix

y = [y₁₁ y₁₂]

ϑ is 2x1 vector of the unknown constant parameters

The following controller is proposed: $τ = y \hat{ϑ} + k_{v} z$ (9) where k_v is a scalar value representing the control gain

With the adaptive update rule: $\dot{\hat{ϑ}} = - \dot{\tilde{ϑ}} = λ y^{T} z$ (10) where $\hat{ϑ}$ is a vector representing the estimate of the unknown parameters $\hat{ϑ}$ is 2x1 matrix ( $\hat{ϑ} \in R^{2 x 1}$ ), $\tilde{ϑ} = ϑ - \hat{ϑ}$ $λ = (\begin{matrix} λ_{1} 0 \\ 0 λ_{2} \end{matrix})$ , and λ_i > 0.

The stability of the proposed controller can be achieved using the following Lyapunov function V: $V = \frac{1}{2} z^{T} m (q) z + \frac{1}{2} \tilde{ϑ^{T}} λ^{- 1} ϑ^{2}$ (11) $\dot{V} = z^{T} \dot{z} + \frac{1}{2} z^{T} \dot{m} z + \tilde{ϑ^{T}} λ^{- 1} \tilde{\dot{ϑ}}$ (12)

By substituting from Equation 6 into Equation 12, we get: $\dot{V} = z^{T} (y ϑ - τ) + z^{T} (\frac{1}{2} \dot{m} - b) z + \tilde{ϑ^{T}} λ^{- 1} \tilde{\dot{ϑ}}$ (13)

Considering that $(\frac{1}{2} \dot{m} - b)$ yields: $\dot{V} = z^{T} (y ϑ - τ) + \tilde{ϑ^{T}} λ^{- 1} \tilde{\dot{ϑ}} - z^{T} bz$ (14)

Substituting Equation 9 into Equation 14 yields: $\dot{V} = - z^{T} k_{v} z + \tilde{ϑ^{T}} (λ^{- 1} \tilde{\dot{ϑ}} + y^{T} z) - z^{T} bz$ (15)

Substituting from Equation 10 into Equation 15 yields: $\dot{V} = - z^{T} k_{v} z - z^{T} bz < 0$ (16)V is lower bounded by zero, $\dot{V}$ is negative semidefinite and $\ddot{V}$ is bounded so by Barbalat’s Lemma $lim_{t \to \infty} \dot{v} = 0$ (17) By the Rayleigh-Ritz theorem, we can prove that $lim_{t \to \infty} z = 0$ (18) which means that $lim_{t \to \infty} e = 0$ (19)

Fuzzy Logic-based pain estimation Unlike classic Boolean logic, which only allows two states: TRUE or FALSE, fuzzy logic expresses different levels, rather than just 1 or 0. Actually the fuzzy logic is a branch of mathematics that allows a computer to model the real world in the same way as people. In robotic rehabilitation applications, the classic control could be applied by defining a pain threshold for each case. This threshold must be defined by the physiotherapist at the start of the rehabilitation process. However, the system, in this case, is not intelligent because we can reach this threshold due to other parameters other than pain. For example, for small angles, starting the motor rotation causes a current peak by default. If the controller considers this current request as resistance from the patient and therefore an indication of the sensation of pain, the system must be stopped and the reached range of motion will be considered as a limit for this rehabilitation step. Consequently, the need to adopt a concept of partial reality where the decision must be the result of a set of fuzzy constraints. Indeed, the current peak often caused by a resistance force applied by the subject could not firmly confirm the presence of severe pain because it could be caused by other factors. However, the presence of this peak accompanied by a muscular contraction gives a strong indication of the level of pain that can be felt by the subject. The pain itself, in some cases, may not be a cause of movement cessation or change of direction. Indeed, at certain stages of rehabilitation, the sensation of intense pain can be considered as a natural symptom. In this case, the fuzzy decision support system must have a third input function to reach a confirmed decision. Similarly, in the proposed architecture, the fuzzy system is based on a combination of the patient’s resistance, the feeling of fatigue, and the last performed RoM. Thus, the implemented fuzzy logic system could allow the robot to continue the rehabilitation process as long as the RoM is already reached previously, even in the presence of severe pain. The proposed fuzzy logic controller consists of two blocks. The first block is for estimating muscle contraction based on four signals extracted from the EMG sensor namely; EMG-RMS, EMG-SST, EMG-vorder, and EMG-logdetect. The second block consists in estimating the level of pain as a function of the muscle contraction, the patient’s resistance and the last RoM reached. The fuzzy architecture is presented in 6. The membership functions for the inputs and for the output of the first fuzzy system are shown in Figures 7 & 8, respectively. While the membership functions for the inputs and for the output of the second fuzzy system are shown in figures Figures 9 & 10, respectively. In the design and implementation of this rehabilitation protocol, assistance was provided by a medical expert who led the rule-making phase based on his medical expertise. The first tests are carried out in order to test the proposed rules. Then a second series of tests were conducted in order to refine the suggestion. Finally, the team arrived at the rule base defined by Table 2 (L: Low; M: Medium; H: High; X: Don’care.).

Fig. 5

Control Architecture Overview.

Fig. 6

The fuzzy system architecture.

Fig. 7

First block inputs fuzzy membership functions: (a)EMG-RMS; (b) EMG-SSI; (c) EMG-v-order; (d) EMG-Log.

Fig. 8

Muscle contraction estimation example.

Fig. 9

Second block inputs fuzzy membership functions: (a) Muscle contraction; (b) Current; (c) Carried out RoM.

Fig. 10

Pain estimation.

Table 1

Robot mechanical components

Article number	Piece name	Quantity
1	Box	1
2	LCD Box	1
3	Box cover	1
4	Higher back arch support	1
5	Lower back arch support	1
6	Drive disk	1
7	Sliding bar	1
8	Forward arch	1
9	Handhold	1
10	Handhold support in L	1
11	Handhold support	1
12	Higher front arch support	1
13	Intermediate front arch support	1
14	Lower front arch support	1
15	Back arch	1

Table 2

Rule base

Rule	Contraction	RoM	Current	Pain
1	L	L	X	L
2	L	X	L	L
3	X	M/H	M	M
4	L	M/H	H	M
5	M	H	H	H
6	H	X	H	H

IoT-based control architecture:

The control process begins with a simple gesture launched by the physiotherapist and taken up by the Kinect. Image processing and recognition algorithms are implemented in the human machine interface (HMI). After the required processing, the desired wrist angle is communicated to the adaptive controller block as well as to the 3D virtual environment block. The calculated torque is sent via the MQTT protocol to the wrist robot. The information collected from the current sensor, the EMG sensor, and the encoder is sent back via the MQTT protocol to the control station. The raw EMG signal acquired goes through extraction algorithms to detect muscle contraction. The EMG features and the current sensor feedback are used as input to the fuzzy system to estimate the pain felt by the patient. Position feedback is used in the control motion block as well as in the fuzzy logic input to make the appropriate decision. The parameters setting and the performed results are stored in the database. System operation is described by the flowchart presented in Figure 11. Although HTTP (HyperText Transfer Protocol) is the most widely used protocol. But in recent years, MQTT (Message Queuing Telemetry Transport) has quickly gained traction. The most important characteristic of the MQTT protocol that really distinguishes it is the real-time aspect; a criterion that avoids the problems due to latency. Also, the MQTT protocol is easy to implement and enables high speed data transmission by means of publish/subscribe operations, which are essential in medical applications, and ideal for machine-to-machine communication.

Fig. 11

Operating system flowchart.

3 Results and discussion

The proposed robot is light, practical and easy to use, which makes it a very suitable solution for home use. The fact that it is remotely controlled by gesture makes it easier for the patient, who has nothing to do with the handling of the device. In fact, in post-stroke cases, the patient is generally unable to do anything and is often passive and the initiation of the gesture itself becomes a concern with these types of cases. A Kinect gesture recognition approach is proposed to improve the process of implementing gesture recognition. Indeed, the system developed is a vision-based remote handling solution that avoids the disadvantages of EMG-based solutions. The Kinect camera is used to recognize the gesture movement of the therapist. The recognized movement is transferred through the internet to the wearable robot at the patient side). The use of Kinect is very beneficial for the therapist during the initialization of the gesture: no need for manipulation or explanation. The patient is completely passive and the robot must repeat the action of the physiotherapist. This kind of exercise maintains mobility and above all flexibility. In the early stages of the rehab protocol, we are often faced with cases that cannot be active and therefore passive ROM (range of motion) exercises can reduce joint stiffness and prevent it from freezing due to lack of movement. Using a fuzzy control approach, the system can adapt effectively to the patient’s conditions. Indeed, the solution integrates a current sensor in order to have an indication of the patient’s pain. The presence of a current peak can be explained by a resistive force. The resistance force occurs naturally and unconsciously when the patient experiences pain. The fuzzy controller takes these current levels and the last angle into account to estimate whether or not the pain felt requires stopping the action. The developed software in this paper is done on the Laboratory Virtual Instrument Engineering Workbench (Lab-VIEW) environment. By using Lab-View, we benefit from the ability to integrate user interface design into the development cycle and take advantage of numerous supports allowing easy interfacing with cameras, instruments, boards and other devices. To identify the exoskeleton, the recognition system is based on comparing a gesture that is presented with all registered models in order to find a match. To detect its exoskeleton, it is sufficient for the user to appear just in front of the sensor ensuring that his head and body are visible. Processed data must pass through a mapping block before being used in the control law set point. The pain estimation is defined according to the implemented fuzzy rules. In the implemented block of the nonlinear control a test is done to identify the state coming from the Fuzzy block. Based on this state, the adaptive controller adopt the desired angle or keep the previous angle.

This developed product features such a user-friendly HMI containing three different control screens. The first shown in Figure 16 has the role of configuration and supervision. At this step, the interface provides an authentication service. This service improves the security aspect of the networked system. Since we are in a context of vision-based control using a 3D camera, the position of the user must respect specific parameters to be correctly recognized. To solve this problem, the developed software provides some indicators on the user 3D position. When the position of the physiotherapist facing the 3D camera is not suitable for control, the indicators are displayed in red as shown on the first screen of Figure 12(a). In order to get the actual information, the Kinect camera requires a predefined position. If the user shows up outside this position, the system interface displays an invalid position alert (See Figure 12(a)). When the appropriate position is detected, the three indicators of the user’s position become colored green, as shown in Figure 12(b).

Fig. 12

Setup sequences: (a) Invalid position for control; (b) Suitable position for control.

The Figure 13 shows that the system offers real time streaming as well as a virtual environment. An integrated IP camera in the developed control architecture is used to send videos to the router. The physiotherapist computer connected to the router displays a real time video of the patient during the rehabilitation process. Such visual interaction could facilitate the process of rehabilitation. By controlling the robot in a gestural way, the video of the patient following the robot’s actions is not enough for the physiotherapist. The physiotherapist must know the real time angle achieved by the wrist in motion. To this end, the joint angle recorded during the exercise is displayed in real time on the physiotherapy interface, as shown in Figure 12. To ensure safety, an indicator is displayed on the physiotherapist’s station to offer information about the pain felt by the patient. Both videos of Kinect and IP camera are transmitted to be displayed in the HMI as shown in Figure 13. Besides, this interface displays an emulation of the physiotherapist’ arm behavior. However, this specificity can be extended, if necessary, to add other digital components to the real scene in order to further develop the rehabilitation process. Indeed, virtual reality and interactive video games are new therapeutic approaches for stroke patients. Today, the digitization of files (the conversion of documents into electronic format) concerns all sectors. Health is not spared and even out of these new methods. Indeed, electronic health records have many benefits for the medical profession as well as for patients. Grouping and remote access to patient records are allowed. It is therefore easier to transfer a patient from one rehabilitation center to another and to follow the evolution of his medical file. In this context, the software developed, as shown in Figure 14, offers the possibility to record patients’ personal data, performed exercises and reports on patient follow-up.

Fig. 13

Real time streaming.

Fig. 14

Data base interface: (a) Add new subject; (b) Save exercise; (c) Generate report.

Experiments are performed on 3 subjects, all with a wrist fracture, just one week after removal of the cast. The subjects underwent a complete rehabilitation protocol to recover the abduction/adduction movements for 10 days. The exercise is controlled by the physiotherapist who tilts the hand in one direction maintaining the position for 5 seconds. The exercise is repeated for 3 sets of 12. The connected robot follows the physiotherapist’s instructions. The first tests carried out with the 3 subjects showed that the subjects had lost approximately 66% of the RoM of normal wrist. Subject 1 was received on February 12, 2020, as shown in Figure 15(a). He underwent a 10-day work protocol punctuated by a day of rest (5 days of work, 1 day of rest, 5 days of work). He obtained a RoM of 43 degrees during the last day (February 22, 2020) with a Rehabilitation Progress Factor (RPF) equal to 7.1. This rate is calculated according to the maximum RoM that can be reached by a healthy subject (Flexion:80-90; Extension:70-90; Radial:15 and Ulnar:30-45). Likewise, subject 2 was received on February 27, 2020 and he underwent the same protocol. He completed his last session with a RoM of 38 degrees and a RPF of 6.6, as shown in Figure 15(b). Whereas subject 3 began his rehabilitation protocol on March 19, 2020 and managed to reach a RoM of 41 degrees on March 29, 2020 with an RPF of 6.8, as shown in Figure 15(c).

Fig. 15

Generated reports: (a) patient 1; (b) patient 2; (c) patient 3.

4 Conclusion

The last decades have presented interesting advances in the development of rehabilitation robotics. By integrating these devices, specialists aim to support function recovery by exploiting technical engineering solutions. In this context, we are developing a robotic wrist rehabilitation device. The proposed solution is portable and easy to use. It is a 1 DoF robot capable of serving both wrist movements (flexion-extension movement and ulnar-radial movement). Due to the current circumstances of the coronavirus spread, providing remote services is becoming a real challenge. In this context, we offer a remote rehabilitation solution. The control is done by gestures that make it simpler and more natural. For a comfortable and safe rehabilitation process, a fuzzy logic based decision making system has been integrated into the control architecture to monitor the pain level. The pain level estimate is based on the fusion of data from different sensors: biofeedback sensor, current sensor and position sensor. The developed HMI not only provides a control and monitoring interface, but also offers a virtual environment for emulation and above all a database for storage and reporting.

Footnotes

Acknowledgments

The authors would like to acknowledge the support of the Deanship of Scientific Research at Prince Sattam Bin Abdulaziz University under the research project # 2020/01/16466.

References

World Health Organization. Stroke, cerebrovascular accident. http://www.emro.who.int/health-topics/strokecerebrovascular-accident/index.html. Accessed: August 3, 2020.

Samaee

and Kobravi

H.R.

, Predicting the occurrence of wrist tremor based on electromyography using a hidden markov model and entropy based learning algorithm, Biomedical Signal Processing and Control 57 (2020), 101739.

Moghaddam

M.M.

, Moshaii

A.A.

and Niestanak

V.D.

, Fuzzy sliding mode control of a wearable rehabilitation robot forwrist and finger, Industrial Robot: the international journal of robotics research and application, 2019.

Tran

V.D.

, Dario

, Mazzoleni

and Posteraro

, Wrist robot-assisted rehabilitation treatment in subacute and chronic stroke patients: from distal-to-proximal motor recovery, IEEE Transactions on Neural Systems and Rehabilitation Engineering, pages 1889–1896, 2018.

Giannoni

, Sandini

, Squeri

, Masia

and Morasso

, Wrist rehabilitation in chronic stroke patients by means of adaptive, progressive robot-aided therapy, IEEE Trans Neural Syst Rehabil Eng 22 (2) (2014), 312–325.

Abdallah

I.B.

, Bouteraa

and Rekik

, Web-based robot control for wrist telerehabilitation. In Proceedings of the IEEE 4th International Conference on Control Engineering Information Technology (CEIT), pages 1–6, 2016.

Yang

, Su

, Luo

and Liu

, A robot learning method with physiological interface for teleoperation systems, Applied Sciences 9 (10) (2019), 2099.

Jiang

, Peng

, Hossain

M.S.

, Miao

and Muhammad

, Telesurgery robot based on 5g tactile internet, Mobile Networks and Applications 23 (6) (2018), 1645–1654.

Hooman

, Michael

and Daniel

, Robot teleoperation with augmented reality virtual surrogates, International Conference on Human-Robot Interaction (HRI), At Daegu, Korea. ACM/IEEE, 2019.

10.

Bouteraa

, Abdallah

I.B.

and Elmogy

A.M.

, Training of hand rehabilitation using low cost exoskeleton and vision-based game interface, Journal of Intelligent & Robotic Systems 96.1 (2019), 31–47.

11.

Bouteraa

and Abdallah

I.B.

, A gesture-based telemanipulation control for a robotic arm with biofeedback-based grasp, Industrial Robot: An International Journal, 2017.

12.

Bouteraa

Yassine

, Boucetta

Rahma

, Benabdallah

Ismail

and Rekik

, Kinect-based Computed Torque Control for lynxmotion robotic arm. In Proceedings of the International Conference on Modelling, Identification and Control (ICMIC), 2015.

13.

Abdallah

I.B.

, Bouteraa

and Rekik

, Kinect-based sliding mode control for lynxmotion robotic arm, Advances in Human-Computer Interaction 2016 (2016), 1–10.

14.

Ibañez

, Soria

, Teyseyre

and Campo

, Easy gesture recognition for kinect, Advances in Engineering Software 76 (2014), 171–180.

15.

Voon

, Silberstein

, Eranki

A.P.

, Phillips

M.S.

, Wood

F.M.

and Edgar

D.W.

, Xbox kinect based rehabilitation as a feasible adjunct for minor upper limb burns rehabilitation: A pilot rct, Burns: journal of the International Society for Burn Injuries 42 8 (2016), 1797–1804.

16.

Trbac

, Kocovic

, Markovic

and Popovic

, Microsoft kinect-based artificial perception system for control of functional electrical stimulation assisted grasping, BioMed research international, 2014, 2014.

17.

Chih-Hsiao

, Yin-Hao

, Chu

K.-C.

and Yen

J.-C.

, Development and evaluation of game-based learning system using the microsoft kinect sensor, International Journal of Distributed Sensor Networks 2015 (2015), 1–10, 07.

18.

Triandafilou

, Barry

, Stoykov

M.E.

, Roth

, Mussa-Ivaldi

F.A.

, Kamper

D.G.

, Ghassemi

and Ranganathan , Development of an emg-controlled serious game for rehabilitation, IEEE Trans Neural syst Rehabil Eng 27 (2) (2019), 283–292.

19.

Mian

A.S.

, Liu

, Li

B.Y.

and Krishna

, Using Kinect for Face Recognition under Varying Poses. In Proceedings of the IEEE workshop on applications of computer vision (WACV), pages 186–192, 2013.

20.

Liu

C.H.

, Chu

K.T.

, Mai

Y.C.

, Hsieh

P.C.

, Hsu

K.C.

, Lee

H.W.

and Tseng

H.C.

, Kinect who’s coming—applying kinect to human body height measurement to improve character recognition performance, Smart Science 3 (2) (2015), 117–121.

21.

Ding

and Chang

, An eigenspace-based method with a user adaptation scheme for human gesture recognition by using kinect 3d data, Applied Mathematical Modelling, Elsevier, 2015.

22.

Kooij

, Keemink

and Stienen

, Admittance control for physical human-robot interaction, International Journal of Robotics Research 37 (2018), 1421–1444.

23.

Castellini

, Vogel

and Smagt

P.V.

, EMG-Based Teleoperation and Manipulation with the DLR LWR-III. In Proceedings of the International Conference on Intelligent Robots and Systems, IEEE/RSJ, 2011.

24.

Zhang

, Meng

and Yang

, Virtual rehabilitation training system based on surface emg feature extraction and analysis, Journal of Medical Systems 43 (3) (2019), 48.

25.

Dipietro

, Ferraro

, Krol

, Rannekleiv

, Volpe

, Krebs

, Palazzolo

and Hogan

, Rehabilitation robotics: performance-based progressive robot-assisted therapy, Autonomous Robots 15 (2) (2003), 7–20.

26.

and Guan

, A review on emg-based motor intention prediction of continuous human upper limb motion for humanrobot collaboration, Biomedical Signal Processing and Control 51 (2009), 113–127.

27.

Bouteraa

, Abdallah

I.B.

and Elmogy

, Design and control of an exoskeleton robot with emg-driven electrical stimulation for upper limb rehabilitation, Industrial Robot: the international journal of Robotics Research and Application, 2020.

28.

Aabdallah

I.B.

, Bouteraa

and Rekik

, Design of smart robot for wrist rehabilitation, International Journal on Smart Sensing & Intelligent Systems 9 (2), 2016.

29.

Ayas

and Altas

, Fuzzy logic based adaptive admittance control of a redundantly actuated ankle rehabilitation robot, Control Engineering Practice 59 (2017), 44–54.

30.

Ahmed

S.F.

, Kadir

K.A.

, Joyo

M.K.

, Ali

and Yarooq

R.S.

, Fuzzy PID controller for upper limb rehabilitation robotic system. In Proceedings of the International Conference on Innovative Research and Development (ICIRD), pages 1–5, 2018.

31.

, Wang

, Hou

Z.-G.

, Lin

and Liang

, Dynamic model based fuzzy-impedance interaction control for rehabilitation robots. pages 1583–1588. 2017 IEEE International Conference on Robotics and Biomimetics (ROBIO). IEEE, 2017.

32.

Harischandra

, Harsha

A.M.

and Abeykoon

, Intelligent bimanual rehabilitation robot with fuzzy logic based adaptive assistance, International Journal of Intelligent Robotics and Applications 3 (1) (2019), 59–70.

33.

Matsui

, Higashi

, Iso

, Hachisuka

, Yamamoto

and Hachisuka

, Wrist rehabilitation robot system and its effectiveness for patients, Sensors and Materials 30 (8) (2018), 1825–1830.

34.

Telegenov

, Zeinullin

, Tursynbek

, Omarkulov

and Shintemirov

, Preliminary mechanical design of NU-Wrist: A 3-DOF self-aligning Wrist rehabilitation robot. In Proceedings of the IEEE International Conference on Biomedical Robotics and Biomechatronics (BioRob), pages 962–967, 2016.

35.

Kim

J.P.

, Kim

, Hwang

C.H.

, Phan

H.L.

and Koo

K.I.

, Wrist rehabilitation system using augmented reality for hemiplegic stroke patient rehabilitation: A feasibility study, Applied Sciences 9 (14) (2019), 2892.

36.

Merzouk

, Choquet

, Gayda

and Ahmaidi

, Assessment of skeletal muscle fatigue in men with coronary artery disease using surface electromyography during isometric contraction of quadriceps muscles, Archives of Physical Medicine and Rehabilitation 86 (2) (2005), 210–215.

37.

Bouteraa

I.B.

, Abdallah

, ElMogy

, Ibrahim

, Tariq

and Ahmad

, A fuzzy logic architecture for rehabilitation robotic systems, International Journal of Computers Communications Control, 15 (4), 2020.

38.

Ottaviano

, Tappeiner

and Husty

M.L.

, A cable-driven robot for upper limb rehabilitation inspired by the mirror therapy. In Proceedings of the 7th International Workshop on Computational Kinematics, pages 174–181, 2018.

39.

Rosen

, Perry

, and Upper-Limb powered exoskeleton design S. Burns. Upper-limb powered exoskeleton design, IEEE/ASME Transactions on Mechatronics 12 (4) (2007), 408–417.

40.

Zhao

, Song

, Yin

, Pan

and She

, A novel robotaided upper limb rehabilitation training system based on multimodal feedback, Frontiers in Robotics and AI 6 (2019), 102.

41.

Phukpattaranont

, Phinyomark

and Limsakul

, Feature reduction and selection for emg signal classification, Expert Systems with Applications 8 (2012), 7420–7431.