Abstract
BACKGROUND:
There is a growing interest in the use of technology in the field of neurorehabilitation in order to quantify and generate knowledge about sensorimotor disorders after neurological diseases, understanding that the technology has a high potential for its use as therapeutic tools. Taking into account that the rehabilitative process of motor disorders should extend beyond the inpatient condition, it’s necessary to involve low-cost technology, in order to have technological solutions that can approach the outpatient period at home.
OBJECTIVE:
to present the virtual applications-based RehabHand prototype for the rehabilitation of manipulative skills of the upper limbs in patients with neurological conditions and to determine the target population with respect to spinal cord injured patients.
METHODS:
Seven virtual reality applications have been designed and developed with a therapeutic sense, manipulated by means of Leap Motion Controller. The target population was determined from a sample of 40 people, healthy and patients, analyzing hand movements and gestures.
RESULTS:
The hand movements and gestures were estimated with a fitting rate between the range 0.607–0.953, determining the target population by cervical levels and upper extremity motor score.
CONCLUSIONS:
Leap Motion is suitable for a determined sample of cervical patients with a rehabilitation purpose.
Keywords
Introduction
Currently, there is a growing interest in the use of technology in the field of neurorehabilitation with the aim of quantifying and generating knowledge about sensorimotor disorders after neurological diseases, understanding that the technology has a high potential for use as diagnostic and therapeutic tools (Shirota et al., 2019). However, these technologies, both robotics and sensor networks, are not currently implemented in daily clinical practice for several reasons. One of them, of high importance, is the high cost they have, which reduces their scope of application only to specialized rehabilitation centers.
On the other hand, neurological diseases produce in most cases disabling disorders that last throughout the patient’s life, affecting their quality of life, level of independence in carrying out activities of daily living (ADL) and social participation. One example is cervical spinal cord injury (SCI), which accounts for 50.9% of traumatic injuries, which lacks a complete healing process and in which the residual strength of partially paralyzed muscles is an important determinant of independence and functionality in the upper limbs (UL) (Yozbatiran et al., 2019). In this context, it is understood that the rehabilitation process must extend beyond the inpatient phase, and it is necessary to involve low-cost technology in the rehabilitation process, in order to have technological solutions that can be brought closer to the patient’s home. In addition, technology combined with the use of virtual applications would allow the execution of therapeutic exercises of the intensity and number of repetitions necessary to induce neuroplasticity, providing the necessary sources of feedback to motivate and ensure the patient’s adherence to the proposed functional tasks (Dimbwadyo-Terrer et al., 2016; Saposnik et al. 2011).
Focusing on low-cost technologies, Leap Motion Controller (LMC; Leap Motion Inc., San Francisco, CA) was developed with an entertainment purpose, allowing manipulate virtual applications through the performance of hand movements and gestures. These movements are the pinch and grasp ability, the flexion-extension of the wrist, the forearm pronation-supination and individual movement of the fingers. Until now, LMC have been applied in studies performed in healthy population (Fotopoulos et al., 2018; Smeraglioulo et al., 2016; Vamsikrishna et al., 2016) and neurological pathologies such as stroke (Fluet et al., 2019; Vanbellingen et al., 2017; Wang et al., 2017; Iosa et al., 2015), Parkinson disease (Cikajlo et al., 2019a; Cikajlo et al., 2019b; Fernández-González et al., 2019), Cerebral Palsy and (Tarakci et al., 2019). However, studies conducted in SCI patients are scarce.
With respect to the content of the virtual applications used in studies focused in neurological diseases, the Playground applications based on Leap Motion Controller involve the performance of functional tasks developed for an entertainment purpose, such as robot-assembly and petal-picking games. These applications, available in Leap Motion App Store©, have been used for the UL rehabilitation in subacute stroke patients, but without providing the users any feedback sources (Vanbellingen et al., 2017; Wang et al., 2017). However, feedback is essential because it allows patients to monitor their performance, promote errorless learning and avoid compensatory movements. Some authors have developed games for the rehabilitation of motor and cognitive aspects in patients who have suffered Parkinson Disease (Fernández-González et al., 2019). Other study proposed a similar enough version of the Box & Block Test, developing a virtual environment with 10 cubes to manipulate (Cikajlo et al., 2019b). However, although some studies reported that adverse effects no were observed in patients (Wang et al., 2017; Iosa et al., 2015), no studies have been found analyzing the target population in function of the neurological pathology studied. Therefore, the application of LMC into the clinical setting has begun to occur rapidly, making the validation of data provided by LMC an important a highly necessary and interesting research objective. Up to date, Smeragliuolo et al. have analysed kinematic data from hand movements in a sample of 16 neurologically healthy subjects (Smeragliuolo et al., 2016), but no studies have been found performed in neurological diseases.
In this study, the concept of serious game is used for a videogame with a rehabilitation purpose. The objective of the present study is to present the RehabHand prototype based on a set of serious games manipulated by means of the LMC for the rehabilitation of UL motor behaviour in SCI patients. These serious games have been designed with the collaborative participation of an interdisciplinary human equipment formed by engineers and clinical staff at the Hospital Nacional de Parapléjicos de Toledo (Spain), taking into account the clinical requirements and the rehabilitation objectives, providing the visual feedback necessary to enhance the motor learning and increase the motivation in patients. After an iterative development, the games versions shown in this work are those that SCI patients with UL affectation can manipulate in a more successful way. Moreover, as a novelty, a clinical study has been conducted to determine the target population in relation to the SCI disease in function of kinematic data provided by LMC during the performance of hand movements.
Methods
Leap motion controller
The LMC is a low-cost device that was designed to control applications by hand gestures and movements. It contains three infrared lights and two cameras. The device is small, rectangular and its weight is 45 g. The LMC is connected to the computer via a USB connection. Several versions of the SDK were available. However, taking into account clinical aspects within the serious games the version 2.3.1. was chosen.
In this study, the characteristics of the computer used were an i7 processor, 12 GB RAM memory and a graphic card NVIDIA for gaming. The operating system was Windows 10.
RehabHand prototype
This prototype has been designed and developed through the collaborative and coordinated participation of a multidisciplinary work team made up of doctors, occupational therapists and engineers. Based on the clinical requirements and patient's needs defined by the clinical staff, different virtual environments have been established and developed using the Unity 3D video game engine. The content of the applications is oriented to achieve complete functional tasks that require some upper limb movements specifically containing hand gestures with the ultimate rehabilitative goal of improving manual dexterity: Box and Block virtual: this serious game has been designed following the guidelines proposed in real Box and Block Test (Mathiowetz et al., 1985). The real scale contains 100 cubes. With the aim of facilitating the manipulation of the cubes in the virtual environment, only 3 cubes appear in the box partition at the same time. As the cubes are passed to the other box partition, other cubes appear randomly. Petal-picking game: the petal-picking within the Playground of LMC was designed to develop the pinch motor skill of the fingers. The objective is to pick petals in a virtual environment. In the present study, this application has been adapted in a serious game, involving visual feedback monitoring the patient’s progress. Several flower sizes are proposed and a random allocation in the screen. This serious game also improves the dexterity and coordination of the fingers. Laboratory game: In this serious game, the patient is a scientist doing an experiment. To achieve this goal, the user has to reach and mani-pulate several objects of different sizes, each one at a time: to put the pills and to pour the content of the cans into the central container Explorer game: solve a riddle that consists of forming a geometric figure with envelope shape, passing only once through each node. This application allows to analyse the hand trajectory. In this case, the order to reach each node is always the same. Bomb game: this serious game was designed to develop the bimanual coordination. In this application, the patient has to disable a bomb. To reach the objective, several objects should be reached and manipulated alternating both hands. In the case that the patient has a completely paralyzed hand, the serious game may be calibrated for the interaction with only one hand. Apocalypse game: This serious game was designed to develop the grasping strength. To accomplish the target, the patient has to make growing three seeds, following the states engine of the game. If the patient’s hand movements correctly adjust to the instructions provided for de game, the seed becomes a plant. If not, the plant dies. Magic cave game: This serious game was desig-ned to simulate the writing function proposing phrases similar to those included in clinical tests.
An example of these virtual applications is shown in Fig. 1. Each virtual application was implemented with the sources of visual and auditory feedback that patients need to appreciate their own improvement after consecutive experimental sessions. Those that consist of achieving the greatest possible number of objectives in a given time, such as the “Petal-picking” application shown in Fig. 1a, have a progression bar that reflects the performance relative to a reference pattern formed by a group of healthy people.

Virtual scenarios related to four serious games included in RehabHand prototype: a) “Petal-picking” serious game; b) “Bomb” serious game; c) “Explorer” serious game and d) “Laboratory” serious game.
The development of the prototype within the Hospital Nacional de Parapléjicos has allowed involving spinal cord injured patients from the project initial stages, allowing a development as part of an iterative process. In this way it has been possible to propose different difficulty levels for each serious game that represent a compromise between the difficulty of the proposed tasks and the functional capabilities of the patients.
Before conducting a functional validation study of the RehabHand prototype, it was necessary to determine the target population for its application to neurological patients. In the present study, specifically the spinal cord injury was selected.
In the study participated a total of 40 subjects, divided into two groups: a neurologically healthy group formed by 20 subjects and a group of 20 spinal cord injured patients, all of them with UL motor function impairments. All patients suffered a cervical SCI with metameric level between C3 and C8, and AIS grade between A and D as defined by the International Standards for the Neurological Classification of Spinal Cord injury (Kirshblum et al., 2014; Kirshblum et al., 2011), specifically the upper limb motor score of the dominant arm (UER if right arm /UEL if left arm) and the spinal cord injury independence measure scale (SCIM) self-care subscale (Catz et al., 1997). Patients who presented any vertebral deformity, joint restriction, surgery on any of the upper limbs, balance disorders, dysmetria due to associated neurologic or orthopedic disorders, visual acuity defects were excluded. The UL Motor Score (UER/UEL) and the self-care SCIM subscale was obtained, with the clinical staff assessment of the strength of five muscle groups of the dominant UL. Each muscle group can be evaluated between 0 (no function) and 5 (normal function), with a total of 25 points. Figure 2 summarizes the main characteristics of the spinal cord injuries of the recruited patients and the statistical comparisons with strength and function values between them and healthy subjects. All patients signed an informed consent form before the study. The guidelines of the declaration of Helsinki were followed in every case and the study design was approved by the local ethics committee. Subject demographics are provided in Table 1.

Main characteristics of the study participants. SCI patient’s series a) aetiology; b) injury level; and c) AIS grade of SCI severity. Comparisons between SCI patients and healthy controls regarding to d) upper limbs motor score, either UER or UEL; e) strength of the cervical key muscles; and finally, level functional independence analysis, namely f) self-care subscore and total SCIM score and g) the self-care subscore / total SCIM ratio. Data were expressed as mean±standard deviation. *p < 0.05. **p < 0.01.
Demographics characteristics of the sample analyzed
a(p < 0.01); *categorical variables are expressed as frequency and percentage; + continuous variables are expressed as mean and standard deviation.
The study is carried out in a single experimental session. The subject was seated in front of an adaptable table in height, and in front of him stands the computer and the LMC. The LMC device was placed between the participant and the computer at a standardized distance with the aim of avoiding trunk compensatory movements.
By means of the LMC device SDK (version 2.3.1), the data acquisition software was modified and programmed in C# for quantifying movements performed by all the participants. All the movements, frame to frame, were saved into a.txt file. Firstly, the hand static pose was captured. Then, all the movements were performed with the dominant hand opened and the palm facing the LMC. The pinching and grasping gestures and the flexion-extension of the wrist joint were analyzed in healthy and SCI patients during the execution of these analytical movements. In addition, the forearm pronation-supination movement was analyzed in the sample of neurologically healthy subjects during the execution of laboratory virtual application.
Finally, all the movements were validated in relation to the fitting rate between the hand image captured by LMC and the internal biomechanical model to LMC (Fig. 3) with the aim of determining the target population. This variable ranges between the values 0 and 1.

Adjustment between the real image of the hand captured by LMC and the internal biomechanical model of the hand.
For all the movements analyzed, the fitting rate variable was expressed as the mean and standard deviation (SD). In comparisons between healthy and spinal cord injured patients, the independent-samples t test was used to determine possible statistical differences. A significance level of 0.05 was used. Moreover, the effect sizes of the differences between groups were calculated by means of Cohen d test.
Results
Target population
As expected, the behavior of the upper limb clinical and functional variables (UER/UEL motor score, self-care subscore and total SCIM score) was quite similar, since as it has been well described previously, SCI patients’ functional level directly correlates to the preserved muscle strength (Oleson and Flanders, 2019). Interestingly, we have found that the functional independence is more dependent of the upper limbs force and function among SCI patients than among healthy people, although without statistically significance.
The results in relation to the UL functional requirements to manipulate LMC are shown in Fig. 4. Each figure contains data of the total sample analyzed, 20 SCI patients and 20 neurologically healthy subjects, showing the fitting rate in function of the upper extremity motor score (UER/UEL) measured over the dominant arm during a static posture in absence of movement (Fig. 4a) and during the execution of the pinching gesture (Fig. 4b). The minimum value of fitting rate was obtained for a UER/UEL of 4 corresponding to a SCI of the metameric level C4 AIS A. Data about healthy subjects are located for a UER/UEL of 25. A high correlation (r = 0.728, p < 0.01) was found between the fitting rate and the dominant arm UER/UEL for the static posture, whereas a moderate correlation (r = 0.617, p < 0.01) was obtained for pinching gesture.

Scatter diagrams between the fitting rate and the upper extremity motor score measured over the right arm during a static posture (a) and pinching gesture (b).
The fitting rate obtained for all the hand movements analyzed are shown in Table 2. The results have been obtained in both groups analyzed, the healthy people and SCI patients.
Fitting rate for all the hand movements analyzed
Fitting rate for all the hand movements analyzed
a(p < 0.01).
The fitting rate between the real images captured and the biomechanical model were always greater for the healthy group than for the patients group. However, the standard deviation was always higher for the patients group.
The fitting rate value was high for both groups in relation to the static pose and the pinching ges-ture. Although significant statistical differences were found between both groups for the static pose (p < 0.01), the effect size of this difference is small (d =0.15). So, although in this case the fitting rate is statistically greater in healthy subjects (0.953±0.080) than in patients (0.799±0.290), the difference is trivial. In fact, the effect size of the difference for each variable measured is small.
In the case of movements with a high variability, such as the flexion-extension of the wrist and the pronation-supination, the fitting rate obtained was moderate in every case. So, the movements estimated (black line) and the corresponding fitting rates measured (gray line) are shown in Fig. 5. In this figure the fitting rates are expressed as percentage. The figure on the left corresponds to the flexion-extension movement of the wrist joint executed as analytical movement. In this case, the minimum values of the fitting rate correspond to the maximum values of flexion and extension of the wrist, determining an effective range of motion between –40° and 50° of flexion to be estimated by means of LMC. The figure on the right represents the forearm pronation-supination about the performance of the laboratory virtual application. In this case, the minimum values for the fitting rate correspond to the supination displacement up to approximately 60° of this movement. The maximum value of pronation measured corresponds to a moderate fitting rate value greater than 60%.

Movement estimated (black line) versus fitting rate measured (gray line) for the flexion-extension of the wrist joint (left) and the forearm pronation-supination (right) for a healthy subject.
In the present study the RehabHand prototype has been presented, describing each virtual application with UL rehabilitative purposes. The target population in relation to SCI has been determined in function of kinematic data measured by means of LMC. To reach this objective, the variables with more influence in the estimations have been controlled, such as lighting conditions and the processor characteristics. Within this study, optimal results in terms of the fitting rate variable were obtained with a computer of these characteristics: i7 processor, 12 GB RAM memory and a graphic card NVIDIA for gaming. The operating system is Windows 10.
The application of LMC into the clinical setting has begun to occur rapidly, making the validation of data provided by LMC an important and interesting research objective. Smeragliuolo et al. have analysed kinematic data from hand movements in a sample of 16 neurologically healthy subjects (Smeragliuolo et al., 2016), but no studies have been found performed in neurological pathologies. The results of these authors have been taken into account and the hand movements have been performed with the opened hand as far as possible. Moreover, the fitting rate between the captured image of the hand and the internal hand biomechanical model at LMC has been analyzed.
Healthy subjects and cervical SCI patients have been recruited in the study in order to know the possible differences between healthy and pathological condition. The static hand pose and hand movements such as the pinch and grasping gestures, the flexion-extension of the wrist joint in an analytical way and the forearm pronation-supination during a functional task have been analyzed. Although the results in the adjustment rate have been higher in the healthy group than in the patients, no statistically significant differences were found, with the exception of the static hand pose. However, the effect size measured indicates that the difference, although statistically significant, is trivial.
More attention is needs to understand the results obtained for flexion-extension of the wrist and forearm pronation-supination. The fitting rate measured for these movements was moderate, indicating as acceptable the accuracy of the estimations. The effective estimated range of motion of for wrist flexion has been between –40° and 50°. Regarding to the pronation-supination movement, Smeragliuolo et al. do not recommend any accurate ranges of motion, because of the highly unstable nature of this movement (Smeragliuolo et al., 2016). Moreover, pronation-supination movements by definition imply forearm rotation executed in a tridimensional space, whereas the flexion-extension movements can be quite correctly represented two-dimensionally and for that, are easier to estimate. For those reasons, it is possible that LMC may be not suitable as a goniometry system in static conditions. However, in healthy subjects involved within a functional task, the estimation in terms of the fitting rate was moderate but appropriate. The minimum values of the fitting rate were obtained during the supination displacement for this degree of freedom. Previous studies have analyzed the functional range of motion of different joints (Namdari et al., 2012; Vasen et al., 1995; Ryu et al., 1991). Although attaining full motion is a reasonable goal of rehabilitative interventions, these studies indicate that to perform all sort of functional tasks involved in activities of daily living, less range of motion is required (Namdari et al., 2012). It is worth noting that in the present study, the results were always obtained during the execution of the laboratory virtual reality application within some UL functional tasks, in which pronation-supination movements are invariably required, at least in some degree.
Another interesting aspect is the lack of studies analyzing the target population according to particular neurological diseases. In a previous study, we applied the linear regression method to investigate the relation between the virtual Box and Block within the RehabHand prototype and the real Box and Block test (Alvarez-Rodríguez et al., 2020). So our purpose in the present study was to follow the same methodology for analyzing the relation between the fitting rate measured and the UER/UEL for the dominant arm (0–25 points). A high correlation between both variables was found for the static hand evaluation, and a moderate one was detected in the case of the pinching gesture. In both situations, the minimum value of the fitting rate was obtained for one particular patient with an upper motor index in the dominant arm of 4 points. This patient suffered a cervical SCI of the metameric level C4, classified as AIS scale grade A. This indicates that LMC is suitable for SCI patients with motor complete injuries of a metameric level below C4 or incomplete injuries with UER/UEL greater than 4.
Finally, this study has some limitations. Conducting experimental studies in populations with neurological pathology is always a challenge. Sme-ragliuolo et al. concluded his study emphasizing the need to validate this device in populations with pathologies that, among its consequences, produce UL function impairments. Smeragliuolo et al. concluded his study emphasizing the need to validate this device in populations with pathologies that, among its consequences, produce UL function impairments. In our case, we have studied patients with cervical SCI, who suffered more or less UL impairments depending on the level of cervical injury and its severity. The LMC device exhibited some problems while the registry of the UL pronation-supination movement in an analytical way. When subjects supinated the forearm until the hand was parallel with the device, LMC was unable for distinguishing the dorsal surface of the right hand and detected the palmar surface of the left hand. Fortunately, all the tasks analyzed in the present work require lower pronation-supination ranges of motion and, therefore, this problem does not appear.
Conclusions
The results obtained in this study allowed determine the target population suffering SCI enabled for using LMC. As conclusion, LMC is suitable for SCI patients with motor complete injuries of a metameric level below C4 or incomplete injuries with UER/UEL greater than 4. Our next step will involve the functional and usability study of the prototype, also carried out patients with spinal cord injured patients.
Conflict of interest
The authors have no conflict of interest to declare.
Funding
This research has been funded by grant from MINECO and cofunded from FEDER, National Plan for Scientific and Technological Research and Innovation by means of the project RehabHand (Plataforma de bajo coste para rehabilitación del miembro superior basado en Realidad Virtual, ref. DPI2016-77167-R).
