Abstract
Introduction
Long-distance information exchange allows the spread of knowledge independent of geographical barriers and provides a means for dialogue with a distant expert. Telemedicine concepts are approached when this exchange of knowledge is related to health promotion, disease prevention, and continuing education of health professionals. According to the World Health Organization, the four authentic elements of telemedicine are (1) to provide clinical support, (2) to overcome geographical barriers, connecting users who are not in the same physical location, (3) to use several types of information and communication technologies, and (4) to improve health outcomes. 1
Telerehabilitation is a branch within telemedicine intended to offer ways to increase motor and/or cognitive stimuli, aimed at faster and more effective recovery of given disabilities, and to measure kinematic data such as the improvement in range of motion (ROM). However, systems for ROM measurement such as OptoTrak, Motion Capture, Motion Analysis, Vicon, and Visual 3D are so expensive that they become impracticable in public health systems and even in private rehabilitation clinics.
In this work we present a virtual tool called RehabGesture, which allows healthcare specialists to remotely acquire and record ROM information of patients in rehabilitation using a tool based on the Kinect® (Microsoft, Redmond, WA) device (inexpensive) and technology for real-time data transmission optimized by the libGlass library (
This article is organized as follows. First, we present the computational tools used for the development of the virtual application, called RehabGesture, that allows the measurement of the motion range and records data in spreadsheets. Then we present a proposal for telerehabilitation using the libGlass library. Finally, we present a discussion comparing RehabGesture with other gesture recognition tools and draw a conclusion.
Materials and Methods
The RehabGesture application was developed in a multidisciplinary study in the Laboratory of Immersive, Interactive and Collaborative Visualization of the Computer Science Department of our institution, which is linked to the Graduate Program in Biotechnology and Computer Science. The study was approved by the Ethics Committee on Human Research of the institution, with application approval number CAAE 11319712.4.0000.5504, supported by the Center for Science and Technology of the Federal University of Sao Carlos.
In the development of the RehabGesture tool, we used the gesture recognition sensor Kinect 2 and the concepts of Natural User Interface (NUI) and Open Natural Interaction (OpenNI and NiTE). These concepts are described below.
NUI
NUI is the name used by computer designers and developers to refer to the interaction with the computer in an effectively invisible mode. Most computer interfaces use artificial control devices such as an alphanumeric keyboard. A NUI interface only requires the user to be able to interact with the environment through interactions previously known to him or her, for example, gestures and voice. 3 This type of interface also requires learning, but this is easier, given that the communication through gestures is something inherent to human beings.
OPENNI
OpenNI is a framework that provides an application programming interface for development of applications that use natural interaction. 4 This application programming interface covers the communication with low-level devices (vision sensors and audio) and high-level solutions (visual tracking using computer vision). The framework is written and distributed under the GNU Lesser General Public License, with source code freely distributed and available to the public.
Natural Interaction Middleware (NITE)
NiTE middleware is used by the OpenNI framework and was developed by PrimeSense. (The PrimeSense Company was acquired by Apple [Cupertino, CA] in 2013.) It is distributed as closed source but is free to use in the development of commercial applications. NiTE is responsible for processing user input obtained by the OpenNI framework, converting it to gestures. 5 The middleware provides two types of tracking: one for hands, capable of detecting gestures like a push, wave and circle; and one for the human body, which allows tracking of the entire body, providing information on the main body joints. The precision of gestures is limited by the sensor device, and the tracking by cameras is not as accurate as mechanical devices, such as mouse and keyboard.
Libglass Library
The development of RehabGesture has been proposed by means of the libGlass library.
6
libGlass is a library for distributed and cloud computing, focused on high performance and high-performance applications, especially virtual and augmented reality. The components used in the development of the RehabGesture collaborative module are as follows: • Event. This is responsible for sharing events between distributed applications, such as mouse clicks and keystrokes on the keyboard. • Shared variable. This allows primitive data types to be shared, such as int, float, strings, and others. • Barrier. This is designed to provide synchronization among collaborative environments (two or more).
Rehabgesture
The study of the angular movement of a body segment demands kinematic parameters, such as the ROM. The coordination of an individual allows for precise movement mainly through dynamic contraction (isotonic), where the force developed by the muscle is higher or lower than the resistance. This force allows the control of the movement against gravity (concentric contraction) or with gravity (eccentric muscle action). This part of the RehabGesture application was based on the open-source software KineticSpace, 7 but, unlike this open-source software, our application returns ROM measures of joints of the shoulders and elbows from the coronal plane.
Telerehabilitation
The rehabilitation process conducted in a hospital is the most desired activity for a person with a motor impediment; however, this cannot be always achieved. In these cases telerehabilitation can be used. One of the main objectives of telerehabilitation is to provide a continuation of the patient's rehabilitation process independently of the place where he or she might be, for example, at his or her home. Nowadays, communication technologies are capable of performing services such as transfer of audio/video and graphics data in real time with no big problems. 8
In order to allow users to carry out collaborative activity on the RehabGesture application, a collaborative architecture was used to develop a telepresence system module. The development model used in this work was proposed by Dias et al., 9 which consists of a connection architecture for three-dimensional collaborative virtual environments based on a client/server and peer-to-peer model, called the Dynamic-Adaptive Connection Model, which allows virtual environments to be used collaboratively, performing evaluations over connections in order to ensure quality of service over the connection among different sites.
Results
Rehabgesture
Figure 1 shows the RehabGesture interface, including the ROM values in degrees, labeled inside circles in the joints of the elbow and shoulder. The RehabGesture logo is represented by a figurative image (an angle of 90°) on the upper left corner of Figure 1. This has trademark registration at the Instituto Nacional de Propriedade Industrial of Brazil with approval number 909054541/2015 and software registration, also at the Instituto Nacional de Propriedade Industrial, with approval number BR 51 2015 000130 2.
Interface of the RehabGesture application.
The software allows the measurement of the ROM (in the coronal plane) from 0° extension to 145° flexion of the elbow joint and from 0° abduction to 180° abduction of the glenohumeral (shoulder) joint, leaving the standing position. The movement of abduction goes through three stages: first stage, 0–60° is performed solely by the shoulder joint; second stage, 60–120° is performed by the shoulder joint with participation of the scapulothoracic joint; and third stage, 120–180° combines trunk inclination with the other joints mentioned above.
Table 1 shows the ROM values recorded at 30 Hz (frames/s). Just a few frames are presented to illustrate the data recording feature of the tool. Data are shown for right shoulder angle, right elbow angle, left shoulder angle, and left elbow angle.
Data on Range of Motion Recorded in Spreadsheets and Separated by Joint (Shoulder/Elbow) and Side (Right/Left)
LEA, left elbow angle; LSA, left shoulder angle; REA, right elbow angle; RSA, right shoulder angle.
From the data acquired, it is possible to create charts (Figs. 2 and 3) to compare differences between movements for the same person (shoulder or elbow joint, right versus left) or among patients undergoing the same therapy, allowing health professionals (kinesiology experts) to create a database.
Shoulder abduction in the same individual.
Elbow flexion in the same individual.
Telerehabilitation
Figure 4 presents the connection model that provides collaboration between two different locations. Environment 1 runs an instance of RehabGesture, which is set as a PeerServer (patient), which provides updated values generated by RehabGesture to other environments. Environment 2 also runs an instance of RehabGesture, which is defined as a Client (health specialist). The environment presented in Figure 4 shows that everything done by the patient during his or her telerehabilitation (Environment 1) is monitored by a specialist using Environment 2. The type of message used in the development of RehabGesture was the synchronous one, so every time Environment 1 sends a message to Environment 2, it receives the updated data in the next frame.
Representation of the connection model server/client used in RehabGesture.
The initialization sequence required for both environments is presented in Figure 5. According to the Unified Modeling Language sequence diagram, the patient initializes the Environment 1 as a Server, so the RehabGesture Environment 1 initializes the tracking module and the Connection Module (as a Server). From this moment on, every interaction generated by the patient is sent to the other environments, in this case Environment 2. Environment 2 is initialized by the specialist (as a Client), who can then monitor the patient's activity.
Unified Modeling Language sequence diagram of RehabGesture (for telerehabilitation).
For environments based on the Internet, which is the case presented in this article, the solution proposed by Dias et al. 9 provides quality of service over the connection. Table 2 presents the values sent by Environment 1, as well as the ones received by Environment 2. RehabGesture renders the virtual environment in a 30 frames/s rate, so the message exchanging rate between the environments is high. Environment 1 sends updated messages at each frame, and it maintains the session data if the specialist desires (it is not default). In the same way, at each frame, Environment 2 requests updated data to Environment 1, which is guaranteed by means of sync barriers.
Message Exchange Values Between Environments for the RehabGesture Application
Data are in messages per second.
Environment 1 (Server instance) sends updated messages during all the execution time (60 messages/s) to Environment 2 and Environment 1 (Client instance). Regarding barriers, Environment 1 sends 60 messages/s. Given that even on the Server side a Client instance is run, a model of shared memory is applied; however, it is an internal behavior of libGlass, not necessary for the RehabGesture execution to worry about. Environment 2 never sends messages because it only makes some requests for synchronization and updated data. RehabGesture exchanges messages between different sites in real time. The lag time considered is the data network delay.
Regarding the messages exchange between Environments 1 and 2, communication solutions provided by libGlass were used. Three different communication protocols were evaluated: Stream Control Transmission Protocol, Transmission Control Protocol, and User Datagram Protocol. Figure 6 shows the results of the environments execution for the three communication protocols and different data network configurations, on which Setting 1 is the best and Setting 4 is the worst. Table 3 presents the three main adversities related to data networks, which are delay, jitter, and loss of package.
Data network adversities versus RehabGesture requirements. SCTP, Stream Control Transmission Protocol; TCP, Transmission Control Protocol; UDP User Datagram Protocol.
Adversities Settings
According to Figure 6, the Stream Control Transmission Protocol communication protocol proved to be the most appropriate choice for the RehabGesture requirements, for having characteristics favorable to real-time applications that require high data rate exchange.
Discussion and Conclusions
The Kinect device, launched in November 2010, has since been actively used to complement rehabilitation sessions, encouraging human–computer interaction in a physically active way. In this work we propose the integration of the Kinect device in a tool, RehabGesture, for physical assessment in general, considering the study of human movement, and providing a communication between the therapist and the patient in virtual reality environments. The Kinect device eliminates the use of markers for the measurement of body movements, representing a practical and low-cost solution.
RehabGesture can measure and record the ROM during rehabilitation sessions while the user interacts with the virtual reality environment. Thus, it allows the therapist to evaluate both the limitation of movements and the evolution of the ROM gain of the shoulder and elbow joints in the coronal plane. The proposed tool has application in the fields of training and physical evaluation of professional and amateur athletes in clubs and gyms and may have application in rehabilitation and physiotherapy clinics for patients with compromised motor abilities.
RehabGesture is not the only available tool for measuring ROM and providing feedback about the user gestures. For example, KineticSpace is a tool that allows recording and recognizing gestures from the Kinect sensor, providing visual feedback that allows the user to identify whether a gesture is the same movement that was previously standardized. 7 Despite this feature, KinectSpace does not present the ROM values in real time and does not allow remote interaction between patient and therapist in the same virtual environment. On the other hand, the Flexible Action and Articulated Skeleton Toolkit is a middleware to facilitate user interaction with virtual reality environments by means of a Kinect sensor. The user can implement motor gestures as input and determine keyboard keys or mouse buttons as output. This allows the user to control the computer from motor gestures, even if the user does not have programming skills. 10,11 Like KinectSpace, the Flexible Action and Articulated Skeleton Toolkit middleware does not show the ROM values in real time, does not allow remote interaction between the patient and the therapist on the same virtual environment, and also does not provide a comparison between a standardized movement and the movement executed by the patient. Table 4 presents a comparison between the aforementioned software tools and RehabGesture, emphasizing the innovations brought by RehabGesture.
Comparison Among Gesture Recognition Software Solutions
FAAST, Flexible Action and Articulated Skeleton Toolkit.
More studies are necessary to indicate the use of RehabGesture in the clinical environment. However, RehabGesture represents a low-cost solution to measure the movement of the upper limbs and to stimulate the process of teaching and learning in disciplines related to the study of human movement, such as kinesiology. The next research goals with RehabGesture are to expand data acquisition to other major joints of the human body, such as hip and knee, and to include the measurement of ROM in the sagittal plane to all the measured joints.
Footnotes
Acknowledgments
This research was supported by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior. The authors acknowledge the contribution of Matheus M. Ramos, a visual programmer at the Federal University of Sao Carlos, who helped elaborate the RehabGesture logo.
Disclosure Statement
No competing financial interests exist.