Kinect-based virtual rehabilitation and evaluation system for upper limb disorders: A case study

Abstract

OBJECTIVE:

To help patients with disabilities of the arm and shoulder recover the accuracy and stability of movements, a novel and simple virtual rehabilitation and evaluation system called the Kine-VRES system was developed using Microsoft Kinect.

METHODS:

First, several movements and virtual tasks were designed to increase the coordination, control and speed of the arm movements. The movements of the patients were then captured using the Kinect sensor, and kinematics-based interaction and real-time feedback were integrated into the system to enhance the motivation and self-confidence of the patient. Finally, a quantitative evaluation method of upper limb movements was provided using the recorded kinematics during hand-to-hand movement.

RESULTS:

A preliminary study of this rehabilitation system indicates that the shoulder movements of two participants with ataxia became smoother after three weeks of training (one hour per day).

CONCLUSION:

This case study demonstrated the effectiveness of the designed system, which could be promising for the rehabilitation of patients with upper limb disorders.

Keywords

Virtual reality patient rehabilitation human-computer interaction kinematics

1. Introduction

Upper limb disorder (ULD) is “an umbrella term used to describe a variety of related injuries to the muscles, nerves, tendons and other soft body tissues of the finger, hand, wrist, arm, shoulder and spine” [1]. It is estimated that more than 80% of stroke survivors are affected by an upper extremity disorder [2], and nearly 35% of all acute injuries seen in emergency rooms involve an upper extremity. Because 90% of our daily life relies on the function of our upper limbs, ULDs seriously affect the health and the quality of life of affected patients and place a heavy burden on the patient’s family and society. The rehabilitation of ULDs is thus an important issue in clinics.

Clinical studies have demonstrated that intense training involving active movements, such as peripheral limb manipulation and occupational therapy, in repetitive tasks and task-orientated activities can form an essential component of programmes that seek to modify neural organization [3] and improve the recovery of functional movement skills [4]. However, the rehabilitation costs are too high for many patients [5], and patients are also often not very active due to tedious training patterns [6].

To overcome these problems, virtual reality (VR) technology has been developed for neural rehabilitation. VR is an effective training tool that can provide real-time performance feedback, multimodal sensory information, a safe and entertaining training environment, and low-cost environments that can be duplicated [3, 5]. Together, these features help patients increase the duration and intensity of rehabilitation [7]. VR is therefore a useful tool for the rehabilitation of patients with disordered movement due to the neurological dysfunctions [8, 14, 15].

Many VR-based neural rehabilitation systems have been developed for upper-limb dysfunction since the late 1990s. VR-based neural rehabilitation systems can be divided into two classes according to differences in their interactive mode, i.e., wired and wireless virtual rehabilitation systems.

Wired virtual rehabilitation systems commonly use a data glove, motion-capture devices, a robot, or haptic devices to enable interaction with a virtual environment. For example, a head-mounted display and a data glove were used in the system designed by Crosbie et al. [8]; the CyberGlove and a Rutgers Master II-ND haptic glove were used in the hand rehabilitation system designed by Adamovich et al. [9]; the PHANTOM was used in the upper-limb rehabilitation system designed by Bardorfer et al. [10]; and the Polhemus Fastrak was used in the tele-rehabilitation system presented by Holden et al. [11]. These wired devices are not only expensive, but they are also inconvenient and uncomfortable during application, which limits their practicality for rehabilitation therapy for people with motor disabilities.

To improve the comfort and security of interaction during virtual rehabilitation, a growing body of research has focused on virtual rehabilitation systems with wireless devices. The Wii sensor is one of the typical wireless devices that have been used to rehabilitate upper limb function, and research has demonstrated that the Wii sensor can be used as an occupational therapy tool [12]. However, the Wii sensor is only usable for patients with relatively good hand function. Additionally, it is also possible to ‘cheat’ the system because movements other than those originally intended by the system can be made (and are seen as successful by the Wii system). A web camera or industry camera can also be used with a wireless device in some proposed virtual rehabilitation systems [13]. Although such cameras are low-cost, they can only identify planar motion in the existing systems, which restricts the range of their applications. Three-dimensional cameras are another class of typical wireless devices that are used in the IREX (Interactive Rehab Exercise) VR System [14] and OmniVR systems [15] and are comfortable and effective for the rehabilitation of patients with upper limb disorders. However, the high cost of the 3D Camera prohibits its widespread use.

Recently, the Kinect sensor has been used as a low-cost alternative for real-time motion capture and body tracking, and this device has significant potential for rehabilitation [16]. It provides full-body human 3D motion capture, video facial recognition and voice recognition capabilities. Furthermore, the Microsoft Company provides a non-commercial Kinect software development kit (SDK) for Windows that allows developers to build various applications with C++. These features make the Kinect sensor well suited for the rehabilitation of patients with motor dysfunction. Thus far, a few Kinect-based rehabilitation systems for upper limb training have been developed; e.g., Chang et al. [17] designed an upper limb rehabilitation system for people with cerebral palsy (CP) using a 3 degrees of freedom Kinect-based rehabilitation programme and demonstrated that the two participants significantly increased their motivation for upper limb rehabilitation; Bao et al. [18] designed a Kinect-based virtual reality training system for the functional motor recovery of upper limbs, and fMRI was used to evaluate the effects of the training; Yao et al. [19] proposed a system that aims to simplify instructions from therapists, increasing patients’ motivation to participate in the rehabilitation exercise; Voon et al. [20] assessed the Xbox Kinect ${}^{\text{TM}}$ as a tool for the rehabilitation of minor upper limb burns. It has been demonstrated that the Xbox Kinect ${}^{\text{TM}}$ is a useful tool for increasing rehabilitation exercise time and patient satisfaction compared with conventional physiotherapy, and there are no indications of concurrent negative effects on patient recovery. However, key problems, including personalized interaction, serious game development, reasonable training and evaluation methods for patients with different types of disorders, still require further research.

Figure 1.

Kine-VRES rehabilitation system. (a) The hardware architecture. (b) The software architecture.

Figure 2.

The movements defined in the Kine-VRES system. (a) Shoulder extension and flexion; (b) shoulder internal and external rotation; (c) shoulder horizontal abduction and adduction; (d) shoulder vertical abduction and adduction; and (e) elbow extension and flexion.

In this study, a Kinect-based virtual rehabilitation and evaluation system (Kine-VRES) was developed to increase the coordination, control, and speed in the use of the shoulder. In the Kine-VRES, patients can interact with virtual objects in a customized mode that is particularly useful for training in reaching and grasping movements. The Kine-VRES system applies the kinematic features captured by the Kinect sensor to generate multiple feedbacks and measure smoothness. We also addressed a case study of two ataxic patients who underwent three weeks of training using the Kine-VRES system.

2. Design of the neural rehabilitation system

The Kine-VRES system was designed to improve the coordination capacity and motor performance of the upper extremities. The purposes of the system are to generate virtual rehabilitation tasks that meet unique needs based on the capabilities of each patient and to provide interactive feedback and evaluation of the results for patient self-assessment.

2.1 Kine-VRES system

As illustrated in Fig. 1a, the hardware of the Kine- VRES system includes a personal computer (PC), a 19-in liquid crystal display (LCD), a Microsoft Kinect sensor, a printer, and a mobile table. The PC is utilized for all computational processing. The LCD is utilized for the audio and visual feedback streams. The Kinect sensor is used to simultaneously record the video, depth map, and skeleton data of the patient. The printer is used to output an evaluation report that includes a velocity profile and the jerk-based measures of movement smoothness. To meet the demands of different patients, the above devices were all mounted on a mobile table with a height and angle that can easily be adjusted. The system can therefore be easily adjusted for the correct fit whether the training occurs when the patient is standing, sitting or lying down.

Figure 1b presents the software architecture of the Kine-VRES system, which was mainly developed using Virtools 5.0 and includes (1) the interaction module, (2) the game module, (3) the evaluation module, and (4) a database.

The interaction module provides several interactive modes and a customized method of assisting the patient in making decisions related to the interactive mode. This variety of modes and customization means that the patient is able to choose one or more of the movements (as shown in Fig. 2), based on the suggestions of the therapist, with their left, right or both arms to interact with the virtual environment before the training. Meanwhile, the expected range of motion (ROM) and the speed of each motion can also be set up based on the exercise goals of each training session. In Fig. 2, we define several movements of the arm based on the muscle contractions of the shoulder or elbow, including forward bending (flexion in Fig. 2a and e), backward movement (extension in Fig. 2a and e), turning (internal or external rotation in Fig. 2b), and movements towards or away from the body (adduction or abduction in Fig. 2c and d).

Once the interaction mode is determined, the patient is able to play the game for the selected movement in the Kine-VRES system. To train the strength and the control ability of the upper limb, the patient can stand, sit or lying down with good posture, their arms at their sides, with the palms towards their body before the start of the movement. During the training, the patient must raise or lower their left, right or both arms to complete the selected movement with a gentle controlled motion until the defined ROM or speed is achieved. When a virtual task is finished, the patient can pause and hold the position for several seconds but cannot drop their arm(s).

In the process of interacting with the virtual environment, the kinematic features are extracted, and the selected movement is immediately identified. Here, the kinematic features are derived from the movement data captured by the Kinect sensor. These features play important roles in human-machine interaction and movement identification as well as driving the feedback in the game module and the computational assessment of the movements in the evaluation module. Details of the technique are presented in the following sections.

The game module contains a series of virtual tasks that correspond to the specific interaction mode. These virtual tasks are designed to train the range of joint motion, hand-eye coordination, trajectory accuracy and the speed of the movement. According to the selected interaction mode, the patient can manipulate objects and complete virtual tasks. Moreover, three types of feedback, including visual feedback, audio feedback, and the kinematic features, are provided in the virtual environment for the patient’s motivation or self-assessment. Both visual feedback and kinematic features provides colour videos, funny pictures and kinematic parameters when the training achieves a certain goal (additional details are presented in the virtual tasks and feedback section). These feedback stimuli help the patient to be aware of the training status. The audio feedback is triggered when the goal (such as the selected ROM or speed) of the virtual task is achieved. Additionally, kinematic features and music can be selected or audio feedback can be switched off.

During training, all of the data are recorded. After one or more weeks, the therapist or patient can evaluate the effectiveness of the rehabilitation using the evaluation module, which is a separate module that was developed with Visual Studio 2010. The evaluation module provides a computational assessment of the movement, and it provides the capability to browse the recorded video and a speed profile of a chosen joint. This module is also capable of creating a printout of the velocity profile and evaluation results.

2.2 Interaction with virtual environment

After setting the interactive mode, the following kinematic parameters can be derived from the movement data captured by the Kinect sensor [21]:

(a)
The 3D position of a joint $j$ at the current time $t_{i}$ :

$\displaystyle P(j,t_{i})=(x_{j}^{t_{i}},y_{j}^{t_{i}},z_{j}^{t_{i}}).$ (1)
(b)
Three velocity components of a joint $j$ at the current time $t_{i}$ :

$\displaystyle F_{pv}(j;t_{i-1},t_{i})=\left\{{\begin{array}[]{l}F_{\textit{pvx% }}(j;t_{i-1},t_{i})=\frac{x_{j}^{t_{i-1}}-x_{j}^{t_{i}}}{t_{i}-t_{i-1}}\\ \\ F_{\textit{pvy}}(j;t_{i-1},t_{i})=\frac{y_{j}^{t_{i-1}}-y_{j}^{t_{i}}}{t_{i}-t% _{i-1}}\\ \\ F_{\textit{pvz}}(j;t_{i-1},t_{i})=\frac{z_{j}^{t_{i-1}}-z_{j}^{t_{i}}}{t_{i}-t% _{i-1}}\\ \end{array}}\right..$ (2)
(c)
The velocity of a joint $j$ at the current time $t_{i}$ :

$\displaystyle F_{pv}(j;t_{i-1},t_{i})=.$ (3)
(d)
The joint angle between three joints $j_{1}$ , $j_{2}$ , and $j_{3}$ at the current time $t_{i}$ :

$\displaystyle F_{a}(j_{1},j_{2},j_{3};t_{i})=\arctan$ $\displaystyle\left({\left|}\right|\right).$

According to these kinematic parameters, our system interacts with the virtual environment in two ways:

(1)
The kinematic parameters are directly mapped to the motion parameters of the virtual object. For example, if the velocity of a joint $j$ is used to replace the speed of the virtual object, the virtual object will move at the same speed as the arm of the patient.
(2)
The kinematic parameters are used to first identify the upper limb movements defined in our system, and the identified movements are then used to trigger the virtual object motion. Taking the movement identification of the left shoulder (L-shoulder) as an example, if the velocity of the left wrist (L-wrist) joint is greater than a small incremental change $\varepsilon$ , namely, $F_{pv}(j_{L-\textit{wrist}};t_{i-1},\linebreak t_{i})>\varepsilon$ , we define the left arm as being in motion, otherwise the left arm is static. If $|F_{j}(j_{L-\textit{shoulder}}$ , $j_{L-\textit{elbow}},j_{L-\textit{wrist}};t_{i})-180|<T$ ( $T$ is a threshold that was set to 15 in this study), then the left upper and lower arm form a horizontal angle. In this case, the movements of shoulder flexion, shoulder extension, shoulder adduction or shoulder abduction (see Fig. 2a, c and d) will be identified as follows:

When the condition of $|F_{\textit{pvz}}(j_{L-\textit{elbow}};t_{i-1},\linebreak t_{i})|<\varepsilon$ is true, if $F_{\textit{pvx}}(j_{L-\textit{wrist}};t_{i-1},t_{i})>0$ , the movement is taken as a shoulder horizontal adduction; otherwise, the movement is taken as a shoulder horizontal abduction. When condition of $\left|{F_{\textit{pvx}}(j_{L-\textit{elbow}};t_{i-1},t_{i})}\right|\linebreak<\varepsilon$ is true, if $F_{\textit{pvz}}(j_{L-\textit{wrist}};t_{i-1},t_{i})>0$ , the movement is taken as a shoulder flexion; otherwise, the movement is taken as a shoulder extension. When the condition of $\left|{F_{\textit{pvy}}(j_{L-wr};t_{i-1},t_{i})}\right|<\varepsilon$ is true, if $F_{\textit{pvz}}(j_{L-\textit{wrist}};t_{i-1},t_{i})>0$ , the movement is taken as a shoulder vertical adduction; otherwise, the movement is taken as a shoulder vertical abduction.

If $\left|{F_{j}(j_{L-\textit{shoulder}},j_{L-\textit{elbow}},j_{L-\textit{wrist}}% ;t_{i})-90}\right|<T$ , the upper and lower arms are vertical. The movements of shoulder rotation, elbow flexion or extension (see Fig. 2b and e) are identified as follows:

When the condition of $|F_{j}(j_{L-\textit{elbow}},j_{L-\textit{shoulder}},\linebreak j_{\textit{hist% }};t_{i})-90|<T$ is true, if $F_{\textit{pvz}}(j_{L-\textit{wrist}};t_{i-1},t_{i})\linebreak>0$ , the movement is taken as a shoulder external rotation; otherwise, the movement is taken as a shoulder internal rotation. When the condition of $\left|{F_{j}(j_{L-\textit{elbow}},j_{L-\textit{shoulder}},j_{\textit{hist}};t_% {i})}\right|<2T$ is true, if $F_{\textit{pvz}}(j_{L-\textit{wrist}};t_{i-1},t_{i})>0$ , the movement is taken as an elbow flexion; otherwise, the movement is taken as an elbow extension.

2.3 The virtual tasks and feedback

According to the suggestions of therapists and the main principles of training, the rehabilitation games should have the ability to increase the duration, frequency and muscle strength of treatment and create the appropriate, personalized practice paradigm. The feedback is added into the designed games, and the different difficulty levels, varying from simple to complex, are available. Figure 3 shows a sequence of screen shots of the games designed in our system.

Figure 3.

(a) Range of motion exercise. (b) Coordination exercise. (c) Control exercise.

Game in Fig. 3a: This game was developed to train the stability, smoothness and control capability of the ROM of a selected movement. This game includes a virtual environment, a virtual ball, three difficulty levels, a scoreboard, and a small video window. In this game, the range of motion of the shoulder flexion/extension or shoulder vertical adduction or abduction is estimated in the interaction module and then directly mapped to the movements of the virtual ball. The virtual ball is controlled to reach the position of level 1, 2, or 3. Here, the original position of the virtual ball corresponds with the lowest value of the set range of the ROM, and the position of level 3 corresponds to the highest value. Generally, we ask the patient to raise their arm out from the body slowly, using a gentle, controlled motion at first while increasing the speed slowly. At each level, the patient is asked to pause and hold the position for 5 to 15 seconds and to repeat this exercise 10 times. Generally, the longer the time that they hold the position is, the more funny pictures and encouraging music and sounds will appear and the higher their score will be.

Game in Fig. 3b: This game was developed to train speed and coordination of the arm. As illustrated in Fig. 3b, the scene consists of several virtual balls, a virtual basket, a virtual table and a virtual hand. During the training process, the 3D position of the wrist joint, which depends on the shoulder’s horizontal adduction or abduction movement, is directly mapped to the virtual hand and is used to control the movements of the virtual hand. When the virtual hand is moved over the virtual ball, the virtual ball is grasped and moved at the same speed as the virtual hand until reaching the basket. In this game, the movement speed of the arm is set based on each patient’s ability. In the first level, the patient can only grasp the nearest virtual ball, and in the next level they can grasp the virtual ball at a small little distance. Generally, the less time the patient uses on each level, the higher their obtained score will be. As the patient completes more levels, she/he receives a higher score as a reward.

Game in Fig. 3c: This game was developed to train the control and the coordination capability of a selected movement through a reaching task. As illustrated in Fig. 3c, the patient is asked to push the virtual block into the cave on each level. In this game, four movements should be selected to trigger the motion of the virtual block forward, backward, in a left turn and in a right turn. In this study, we chose the internal/external rotation of the shoulder and elbow flexion/extension movements. Ten different difficulty levels are set in the game. From levels I to III, the block can be controlled and moved into the cave by the selected movements within a short time. From levels IV to X, different obstacles, such as bridges, red blocks and curved roads are added, and the patient is required to spend increasing amounts of time controlling the block. Generally, the more levels the patient passes and the less the time the patient uses on each level, the higher the score will be.

For the above games, visual, audio and the kinematic parameter feedback [21] is provided to prevent the patient from losing interest too quickly and to increase motivation of the patient towards participating in rehabilitation. The visual feedback includes funny pictures, a scoreboard and a colour video. Here, the funny pictures are from animals or flowers, which make the patient feel comfortable and happy. For example, in the picture in Fig. 3a, the bee will fly up in the virtual environment when the patient successfully passes the checkpoint. The audio feedback includes pleasant music and warm applause. The kinematic parameters used for feedback include the 3D position, the joint angle and the velocity of the selected joint. These feedback stimuli are embedded in the virtual environment, and the trigger conditions of the funny pictures and music are the expected kinematic parameters. For example, if the selected movement arrives at the expected joint angle in the game Fig. 3a, the encouraging music will be triggered. If the virtual ball is moved over the virtual basket, the pictures and music will appear.

2.4 Assessment of the movement

Two types of functions are provided in the evaluation module of the Kine-VRES system. These functions are a quantitative evaluation and a short questionnaire.

Based on previous studies, the movements made by patients recovering from strokes become smoother as recovery proceeds [22, 23], and jerk-based smoothness measures have been proven effective as measures of motor performance. Therefore, the quantitative evaluation indexes of the rehabilitation of patients with ataxia are estimated based on the following jerks in the proposed system.

(1)
Integrated squared jerk (ISJ) [24]:

$\displaystyle\textit{ISJ}=\int_{t_{1}}^{t_{2}}{\left|{\frac{d^{2}v_{j}}{dt^{2}% }}\right|}^{2}dt.$ (5)

where $v_{j}$ denotes the velocity profile of the wrist joint $j$ , and

$\displaystyle v_{j}^{t_{i}}=,(1<i\leqslant N-1).$ (6)
(2)
Integrated absolute jerk (IAJ) [25]:

$\displaystyle\textit{IAJ}=\int_{t_{1}}^{t_{2}}{\left|{\frac{d^{2}v_{j}}{dt^{2}% }}\right|}dt.$ (7)
(3)
Mean squared jerk (MSJ) [26]:

$\displaystyle\textit{MSJ}=\frac{1}{t_{2}-t_{1}}\int_{t_{1}}^{t_{2}}{\left|{% \frac{d^{2}v_{j}}{dt^{2}}}\right|}^{2}dt.$ (8)
(4)
Root mean square jerk (RMSJ) [27]:

$\displaystyle\textit{RMSJ}=-\sqrt{\frac{1}{t_{2}-t_{1}}\int_{t_{1}}^{t_{2}}{% \left|{\frac{d^{2}v_{j}}{dt^{2}}}\right|^{2}}dt}.$ (9)
(5)
Mean squared jerk normalized by peak speed (MSJP) [28]:

$\displaystyle DJ=\frac{1}{v_{\textit{peak}}\left({t_{2}-t_{1}}\right)}\int_{t_% {1}}^{t_{2}}{\left|{\frac{d^{2}v_{j}}{dt^{2}}}\right|}^{2}dt$ (10) $\displaystyle v_{\textit{peak}}=\mathop{\max}\limits_{t\in[t_{1},t_{2}]}v_{j}(% t).$ (11)

In Eqs (5), (7)–(10), the velocity profile $v_{j}$ of the wrist joint is obtained and recorded from the movement coordinate test, which can be the finger-to-finger test, the finger-to-nose test or any goal-directed movement. Here, the movement coordinate test must be performed at the beginning and end of one rehabilitation session. During the performance of the test, the patient should perform five to-and-fro pointing movements between a finger and the nose/finger/specific goal at a self-paced speed. The rehabilitation therapist will provide guidelines at the beginning and end of one rehabilitation session. Generally, lower scores for the jerk-based evaluation indexes indicate smoother paths with fewer sub-units. Therefore, the smaller the jerk measure value is, the smoother the velocity profile and the better the training effectiveness will be. In this manner, the patient will be able to monitor his/her progress.

The short questionnaire is designed to assess whether the selected interaction mode and virtual task in a rehabilitation session are suitable. This short questionnaire is about the subjective responses of the participants and the therapist. A 7-item questionnaire (as shown in the appendix) was used, and the first four items of the questionnaire were based on the longer Presence Questionnaire developed by Witmer and Singer [29]. The responses are rated on a scale of 1–5 on which 1 $=$ “not at all”, and 5 $=$ “very good”. In our system, the short questionnaire form is completed by the participants at the end of each rehabilitation session. The results of the questionnaire can help the therapist adjust the rehabilitation plan in a timely manner and help the participant select a more exciting virtual task for the next rehabilitation session.
3. A case study

3.1 Participants

To test our system, a case study was undertaken in the rehabilitation centre of the Qinhuangdao First People’s Hospital, and two male subjects with ataxia caused by stroke were selected to participate. The first subject was 48 years old (Patient A), and the second subject was 57 years old (Patient B). Before initiation of the rehabilitation training with our system, both participants presented with an inability to smoothly reach a target and both had lost the coordination of their upper limbs for over a year.

3.2 Training procedures

The participants were trained exclusively and independently with our system for approximately 1 hour/day, 5 days/week, for a total of nearly 3 weeks. A therapist was asked to give some instructions at the beginning and end of this training programme. The participants were asked to complete finger-to-finger movement tasks at the beginning of the rehabilitation session, and the 3D coordinates of the wrist were simultaneously recorded. Next, the participants performed the rehabilitation training using the selected virtual tasks and interact mode, which were determined by the attending rehabilitation therapist, for three weeks, and each patient was trained in the virtual tasks in 1-hour sections. Patient A used shoulder flexion and shoulder horizontal abduction movement to play Games 1 and 2 with his left shoulder. Patient B used elbow flexion/extension and shoulder internal/external rotation movement to play Game 3 with his right arm. The patients were allowed a ten-minute break after half an hour of training. On the last day, the patients were asked to perform the finger-to-finger tests again to record the 3D coordinates of the wrist. During the training period, the exercise database was filled, and the smoothness measures of the movements during the finger-to-finger tests were calculated to evaluate the effects of the rehabilitation.

Figure 4.

Velocity profiles of Patients A and B.

Figure 5.

Jerk-based smooth measures of Patients A and B.

3.3 Results

3.3.1 Quantitative evaluation result

To obtain quantitative evaluation results for the two patients with ataxia, the five evaluation indexes in the evaluation module of the Kine-VRES system were calculated.

Figure 4 illustrates the velocity profiles of the movement coordinate tests for Patients A and B, which were recorded at the beginning and end of the rehabilitation test. Here, the velocity profile of each patient was the average result of five to-and-fro movements, i.e. movements of a finger pointing to his/her nose and then returning to his/her side. We observed that the velocity profiles of Patients A and B were full of jerky movements on Day 1, and the jerky movements were markedly reduced on Day 15.

Figure 5 illustrates the jerk measure results from the velocity profiles in Fig. 4 for the first to the last day of the training. Here, the five jerk-based evaluation indexes, including the ISJ, IAJ, MSJ, RMSJ and MSJP, were calculated. The black blocks are the evaluation indexes recorded on the first day, and the open green dots are the evaluation indexes recorded on the last day. The two patients exhibited significant decreases in the five smoothness metrics, and Patient A achieved a larger difference value between the first and the last day than Patient B.

3.3.2 Short questionnaire about the system

The short questionnaire about the subjective responses of the participants was also completed to evaluate the effects of the selected virtual tasks and the interact mode. As presented in Table 1, only the items “control” and “sense of being in the environment” were rated a “4”, and other items were rated a “5”.

Table 1
Questionnaire results for subjects and therapists

Questions	Patient A	Patient B
Feeling of enjoyment	5	5
Difficulty of the virtual task	5	5
Success	5	5
Control	5	4
Feeling of interaction	5	5
Effectiveness of the games	5	5
Sense of being in the environment	4	4

4. Discussion

The main aim of this study was to investigate the feasibility of the Kine-VRES system to improve the use of an upper limb to complete goal-oriented movements.

The quantitative evaluation result demonstrated that the training tasks designed in the system were able to enhance the coordinate capacities and motor performances of two patients with ataxia. Specifically, the relative changes in the five jerk-based indexes for Patient A were greater than those for Patient B. These results indicated that the selected interaction modes and virtual tasks were more suitable for Patient A.

In addition to the measured quantitative outcomes, the participant feedback provided by the short questionnaire also indicated whether the selected interaction mode and virtual tasks were appropriate for two patients. We saw that the evaluation results of the fourth question and the last question were “4”, which meant that both Patient A and Patient B thought that the sense of being in the environment was not sufficient, and Patient B thought the “control” item of the selected virtual task was not “very good”. The reason for this latter result might be that the selected movements or the virtual task was a little too difficult for Patient B. Thus, it would be best for the therapist to adjust the training plan for Patient B in the next rehabilitation section.

Another finding was that the difficulty level of the virtual task had some influence on the training effect and quality. In this case study, the games illustrated in Fig. 3a and b contained easy tasks and an easy interaction mode. Thus, Patient A, who had mild cognitive impairment, felt the training goal was easy to achieve, and he became more motivated during the training. However, the game in Fig. 3c contained complex difficulty levels and four interaction modes that had to be selected during the training. So this game was not easy to control, and Patient B also felt that the goals of the virtual tasks were difficult to achieve. During the training, he was unable to pass any other game checkpoints after passing level III and thus quickly lost his confidence. Therefore, it is very important to determine whether the virtual task and interaction mode are appropriate for each specific patient during each rehabilitation section.

Compared with other studies [17] that have been based on the use of the Kinect for rehabilitation, the advantage of our system is that it proposes quantitative evaluation results, and it can help patients quickly perform self-assessments. The study of Chang et al. [17] used the number of correct movements to assess the effect of the system. However, it was difficult to calculate the number of correct movements during training, which was one of the reasons that their method is less suitable for self-assessment. Zannatha’s system [30] used EMG signals to evaluate the system, which is not convenient for home-based training. The system proposed by Yao et al. [19] allows the therapists to assess the current state of the patients according to the uploaded data. This assessment depends almost entirely on the therapist, which will increase the burden on the therapist, and makes it impossible for the patient to perform a self-assessment. In our system, the therapist was also involved but was only required to provide suggestions about movement selection at the beginning and end of one rehabilitation section, which can be realized by remote guidance. Overall, the evaluation method proposed in our system is more suitable for self-assessment and home-based rehabilitation.

However, the developed system still has some limitations. First, richer virtual tasks should be developed to meet the needs and interests of different patients with upper extremity disorders. Second, although the case study demonstrated that the system was helpful in terms of increasing the motivation for upper limb rehabilitation, additional research with different individuals is still needed to demonstrate the efficacy of the developed system. The current version of the system only focused on the wrist, elbow, and shoulder; future studies should integrate these foci with hand training, and more subjects should be selected to participate in the experiments.

5. Conclusion

In this study, a Kinect-based rehabilitation system was designed for patients with upper limb disorders. This system allows for the creation of personalized training movements and provides a quantitative evaluation tool through the kinematic features captured by Kinect. This case study indicates the feasibility of our developed system for the rehabilitation of ataxia patients. In addition, the proposed system is low-cost, movable and provides objective measures of the current status of a subject’s level of coordination as well as self-assessment indexes over the period of therapy. Therefore, the Kine-VRES system exhibits the potential to enable patients to rehabilitate in their home environments (E-health).

Footnotes

Acknowledgments

The authors would like to thank the study participants for contributing their time and sharing their viewpoints and experiences, the Rehabilitation Center of the Qin Huangdao First People’s Hospital for supporting this research, and Honghai Liu for helpful comments on this article.

Conflict of interest

None to report.

Appendix

The short questionnaire includes seven items. These items assess a participant’s (a) feeling of enjoyment, (b) evaluation of whether the difficulty of the virtual task is suitable, (c) success, (d) control, (e) feeling of interaction, (f) evaluation of whether playing the game is effective rehabilitation, and (g) sense of being in the environment.

References

Upper limb disorders [Online]. 2018. Available from: http://www.hse.gov.uk/msd/uld/.

Truelsen

PiechowskiJóźwiak

Bonita

Mathers

Bogousslavsky

, and Boysen

. Stroke incidence and prevalence in Europe: a review of available data. European Journal of Neurology. 2006; 13(6): 581-598.

Jack

Boian

Merians

Adamovich

Tremaine

Recce

, et al. A virtual reality-based exercise program for stroke rehabilitation. In Proceedings of the Fourth International ACM Conference on Assistive Technologies. 2000; pp. 56-63.

Nudo

Wise

SiFuentes

, and Milliken

. Neural substrates for the effects of rehabilitative training on motor recovery after ischemic infarct. Science. 1996; 272: 1791-1794.

Morrow

Docan

Burdea

, and Merians

. Low-cost virtual rehabilitation of the hand for patients post-stroke. 2006 International Workshop on Virtual Rehabilitation. 2006; pp. 6-10.

Betker

Szturm

, and Moussavi

. Development of an interactive motivating tool for rehabilitation movements. Conf Proc IEEE Eng Med Biol Soc. 2005; pp. 2341-2344.

Novak

Mihelj

Ziherl

Olensek

, and Munih

. Psychophysiological measurements in a biocooperative feedback loop for upper extremity rehabilitation. IEEE Transactions on Neural Systems and Rehabilitation Engineering. 2011; 19(4): 400-410.

Crosbie

Lennon

McNeill

, and McDonough

. Virtual reality in the rehabilitation of the upper limb after stroke: the user’s perspective. CyberPsychology & Behavior. 2006; 9(2): 137-141.

Adamovich

Merians

Boian

Tremaine

Burdea

Recce

, et al. A virtual reality based exercise system for hand rehabilitation post-stroke: transfer to function. In Engineering in Medicine and Biology Society. 26th Annual International Conference of the IEEE. 2004; pp. 4936-4939.

10.

Bardorfer

Munih

Zupan

, and Primozic

. Upper limb motion analysis using haptic interface. IEEE/ASME Transactions on Mechatronics. 2001; 6(3): 253-260.

11.

Holden

Dyar

, and Dayan

. Tele-rehabilitation using a virtual environment improves upper extremity function in patients with stroke. IEEE Transactions on Neural Systems and Rehabilitation Engineering. 2007; 15(1): 36-42.

12.

Halton

. Virtual rehabilitation with video games: A new frontier for occupational therapy. Occupational Therapy Now. 2008; 10(1): 12-14.

13.

Cameirão

Badia

Oller

, and Verschure

. Neurorehabilitation using the virtual reality based rehabilitation gaming system: methodology, design, psychometrics, usability and validation. Journal of Neuro Engineering and Rehabilitation. 2010; 7(1): 48.

14.

Rand

Kizony

, and Weiss

PTL

. The Sony PlayStation II EyeToy: low-cost virtual reality for use in rehabilitation. Journal of Neurologic Physical Therapy. 2008; 32(4): 155-163.

15.

Omnivr – The World’s First Virtual Rehabilitation System [Online]. 2018. Available from: http://www.hellotrade.com/acp-accelerated-care-plus-corporation/omnivr-the-world-s-first-virtual-rehabilitation-system.html.

16.

Obdrzalek

Kurillo

Ofli

Bajcsy

Seto

Jimison

, et al. Accuracy and robustness of Kinect pose estimation in the context of coaching of elderly population. In Engineering in Medicine and Biology Society (EMBC), 2012 Annual International Conference of the IEEE. 2012; pp. 1188-1193.

17.

Chang

Han

, and Tsai

. A Kinect-based upper limb rehabilitation system to assist people with cerebral palsy. Research in Developmental Disabilities. 2013; 34(11): 3654-3659.

18.

Bao

Mao

Lin

, et al. Mechanism of Kinect-based virtual reality training for motor functional recovery of upper limbs after subacute stroke. Neural Regen Res. 2013; 8(31): 2904-2913.

19.

Yao

, and Li

. Kinect-based rehabilitation exercises system: therapist involved approach. Bio-medical Materials and Engineering. 2014; 24(6): 2611-2618.

20.

Voon

Silbersteina

Erankia

, et al. Xbox KinectTM based rehabilitation as a feasible adjunct for minor upper limb burns rehabilitation: A pilot RCT. Burns. 2016; 42(8): 1797-1804.

21.

Duff

Chen

Attygalle

Herman

Sundaram

Qian

, et al. An adaptive mixed reality training system for stroke rehabilitation. IEEE Transactions on Neural Systems and Rehabilitation Engineering. 2010; 18(5): 531-541.

22.

Liebermann

Berman

, and Weiss

. Kinematics of reaching movements in a 2-D virtual environment in adults with and without stroke. IEEE Transactions on Neural Systems and Rehabilitation Engineering. 2012; 20(6): 778-787.

23.

Rohrer

Fasoli

Krebs

Hughes

Volpe

Frontera

, et al. Movement smoothness changes during stroke recovery. The Journal of Neuroscience. 2002; 22(18): 8297-8304.

24.

Platz

Denzler

Kaden

, and Mauritz

. Motor learning after recovery from hemiparesis. Neuropsychologia. 1994; 32(10): 1209-1223.

25.

Goldvasser

McGibbon

, and Krebs

. High curvature and jerk analyses of arm ataxia. Biological Cybernetics. 2001; 84(2): 85-90.

26.

Wininger

Kim

, and Craelius

. Spatial resolution of spontaneous accelerations in reaching tasks. Journal of Biomechanics. 2009; 42: 29-34.

27.

Young

, and Marteniuk

. Acquisition of a multi-articular kicking task: Jerk analysis demonstrates movements do not become smoother with learning. Human Movement Science. 1997; 16(5): 677-701.

28.

Hester

Hughes

Sherrill

Knorr

Akay

Stein

, et al. Using wearable sensors to measure motor abilities following stroke. International Workshop on Wearable and Implantable Body Sensor Networks. 2006; pp. 4-8.

29.

Witmer

, and Singer

. Measuring presence in virtual environments: A Presence Questionnaire. Teleoperators and Virtual Environments. 1998; 7(3), 225-240.

30.

Ibarra Zannatha

Tamayo

Sánchez

ÁD

, et al. Development of a system based on 3D vision, interactive virtual environments, ergonometric signals and a humanoid for stroke rehabilitation. Computer Methods and Programs in Biomedicine. 2013; 112(2): 239-249s.