Objective assessments of human motor ability of the upper limb: A systematic review

2.1 Study selection

Titles and abstracts were screened to determine eligibility for inclusion in the systematic review. In cases where title and abstract were insufficient, the methodology, results and conclusions were read. All articles screened were recorded in a data extraction sheet detailing authors, title, year, keywords, whether to be considered or not, gross explanation of exclusion, health issue, sensors used, features extracted, statistics/machine learning techniques and conclusion.

OPEN IN VIEWER

For all outcomes, traditional assessments comparison studies or research related to neurodevelopmental, neurocognition, neurotoxicity and neuropsychology were excluded as well as review articles, conference abstracts, studies with animals, gait analysis studies or using magnetic resonance imaging (MRI). Subsequently, articles where the evaluation was based on traditional upper limb assessments, questionnaires or functional MRI (fMRI) instead of computer sensors were excluded too.

Another scholar read the information extracted on the data sheet to make sure that the selected studies met the inclusion criteria. As Fig. 1 describes, after applying the initial exclusion criteria, 75 articles were selected for an in-depth analysis, reading their objectives, methods, discussion and conclusions; which allowed the exclusion of 32 more.

Regardless of all the exclusion and inclusion criteria process, some articles remain because of the machine learning techniques applied, features engineering or sensors used. These studies are weighed as low. For instance, Taha et al. [23] extracted features such as shooting scores, vertical jump, static balance and hand grip strength to predict or classify performance of the archers through Support Vector Machines (SVM) techniques, nevertheless they captured data using a non-automated dynamometer and also through human observation, moving away from the main purpose of the authors which is to review objective assessments but using automated sensors.

On the other hand, Mentis et al. [24] used a Leap Motion device, an automated infrared sensor connected to the computer to track hand movements but with the purpose of letting Parkinson’s patients to co-interpret movement outcomes during the evaluation meeting through a hand open/close test, not extracting movement data nor using any machine learning algorithm on the task oriented measure neither looking for an automation of the assessment.

Also, Linkel et al. who work with inverse dynamics [25]is pondered as low due to our interest in kinematics rather than dynamics movement analysis. In Table 1, a first group of detail information about these three studies is extended along with others using electromyography [26, 27] as well as a study of Joint Laxity [28].

In a second group of studies weighed as low are those working in posture analysis. Tao et al. [29] study pose representation in Parkinson’s patients for detecting gait abnormalities and transitions between sitting and standing through Hidden Markov Models (HMM) and depth sensors, but since it is focus on pose and gait parameters rather than upper limb movement was classified as low as well as the study of Iwasaki and Hirotomi [30] about postural control but using accelerometers, gyroscopes and root mean squares values of acceleration and angular velocity as a quantitative measure to compare therapist’s ratings. [31, 32] were similar studies working with postures as shown in Table 2 with more detail information about the features examined and learning models chosen.

Finally, a third group of articles pondered as low, are those working in walk/gait analysis as shown in Table 3. For example, Adachi et al. [33] measured spatial and temporal parameters for a further application of multivariate linear regression; even though they used feature engineering and linear regression, the study was classified as low because it is related to walk, lower limb and gait analysis, along with [34], a gait and balance study to evaluate Tinetti score and fall risk in elders where the classification problem was solved with a linear model and the prediction of the Tinetti outcome with artificial neural network (ANN). [35, 36, 37] are other studies focusing on gait analysis therefore weighed as low too.

Many research questions are answered by inferential statistics rather than machine learning techniques. Those studies were categorized as Medium, such as [38], where Kinect was used in martial arts to record spatial coordinates of upper limb joints while the system analyze similarities between practice cycles through Pearson’s correlation coefficient (r) of 0.71 ∼ 0.84 for instructors and practitioners of 0.52.

In a motor coordination study with autism spectrum disorder children [39], data coming from a 9 degrees of freedom (DOF) sensor is normalized with the Shapiro-Wilk test, then 2-tailed t -Tests were performed to assess task differences and Pearson’s correlation to examine relationships between dependent variables and clinical characteristics.

Also, McKenzie et al. [40] scrutinize the validity of 5 robot-based assessments of upper limb through a cross sectional analysis using means, standard deviations and other descriptive statistics techniques, in addition to Spearman’s rho (inferential statistic), looking for correlations between standard assessments and robot based (HWARD device) assessments. A similar comparison between conventional and sensor-based way to assess arm, is made in [41] with a 10 DOF sensor through Pearson’s correlation measurement and Steiger’s Z-test.

On the other hand, Jayaraman et al. [42] used Shapiro-Wilk test and Kruskal Wallis ANOVA test to know the influence of sensor type, sensor location, population specific movement signatures and characteristics of activities on the validity of outcome estimates (step count, energy expenditure, among others) by standard proprietary algorithms.

Energy expenditure and time spent were also features analyzed in a study of both upper extremities using accelerometers [43] with Kolmogorov-Smirnov tests for normality and Spearman correlation to examine relationships between primary and secondary variables across all tasks. In [44], ANOVA was also used for analyzing upper limb kinematics captured through a robotic device. Information about these studies weighed as medium are detail in Table 4.

Main discussion is focused on studies that fit authors objectives and are weighed as high. For this discussion, [45] is included because of the type of exercises required by researchers, features extracted, data process made and above all for the similarity to the upper limb motion analysis, although it is about a knee issue. The discussion is centered in best practices that Halilaj et al. in [21] state; thus, analysis of these studies leads us to extract information about feature engineering; feature selection, data purity (e.g. noise, missing data) and performance evaluation. In addition, purpose, sensors placement and exercises or movements required by researchers were extracted too.

Considering impairment detection or improper movement detection as a classifier for being health or not healthy, 35% of the weighed as high studies are using machine learning algorithms to detect if a patient is healthy or not. On the other hand, 40% of the studies are looking for prediction of assessment scores and 30% for automation of traditional assessment scores. Alternatively, 15% intend to characterize movement and other 20% aim to measure disease or recovery progression.

Next, we grouped studies pondered as high according to the sensor used identifying their machine learning (ML) tasks such as classification (represented by C), clustering (represented by K) and regression (represented by R). Table 5 presents more information about these group of studies that uses depth sensors.

Position and location of sensors vary according to their nature. For instance, studies that uses depth cameras like Kinect or RealSense are commonly placed in front of the subjects and camera setup depends of depth sensing technology if they are using triangulation with structured light or time of flight due to their range vision [46, 47, 48]. Likewise, distance between camera and subjects depend on the activity that subject has to perform and his/her height, although optimal range is calculated between 0.7 meters to 2 meters [49], if the study is detecting the full human joints or if the activity is related to walk analysis the range could extend.

Most of the studies exploit the built-in body tracking algorithm such as [50, 51, 52]. In [53], they combined Kinect at 1.5 meters with a glove flex sensor wear on by subjects to capture data for better performance in grasping and wrist flexion/extension exercises. In [54] discard the built-in body tracking and choose depth pixels analysis to pursue their objectives.

Leap motion is a device that also uses infrared lasers but with different techniques that makes it not necessarily a depth sensor. Despite the fact that Table 5 is about studies using depth sensors, we included in this group to [47, 55] where through Leap Motion they track patient fingers between 25 to 600 millimeters above the center of the sensor.

Like the studies that used depth sensors, articles using accelerometers and/or gyroscopes, in Table 6, vary their location and position. For example, [56, 57] place sensor in the geometric center of the arms, 10 cm between forearm and dorsal stripes and 8 cm between upper arm and epicondyus lateralis humeri. In [58] sensors are placed in the dorsal side of forearm, proximal to the wrist while in [45] they set them up in the chest, thigh and shank; with the y-axis of the sensors well-aligned to the long axis of the body segment. Bennasar et al. [17, 59] for a healthy/not healthy classification problem locate their units on the wrist and in the center of the chest.

Following, in Table 7 is shown studies that uses other sensors rather than Kinect and Accelerometers such as the iRobot in [60], Phantom Robot Arm [61], specific robotic device in [62], and Armeo Spring in [63], where a fixed setup is in place to performed the experiments. Lee et al. [64] use a handgrip device for predicting Oswestry disability index in grasping exercises. Lastly, an electromagnetic tracking system, Polhemus, is set up with their own sensors in thorax, scapula, upper arm, forearm, hand, thumb and index finger in [65] to measured smoothness progression.

Related to exercises or movements, in [63] Vertical Capture is a task where a target appears on the monitor and subjects must place the cursor controlled by the endpoint position on the target. When a target is hit, it disappears and new one appears on the screen while in [65] the task consisted of 2 parts: a reach-to-grasp toward a block, followed by a displacement of the block toward a target location.

For articles weighed as low and medium a gross inspection of features extracted and analyzed was conducted. It is difficult to stablish a pattern among studies, no matter if they are using the same kind of sensors or working in the same health issue, due to the different research purposes and exercises that patients must perform, but most importantly because of the feature engineering process that is domain-specific and may make a project succeed or fail [66]. In this process, researchers use the data extracted from sensors to create or add new inputs using techniques such as principal component analysis, Fast Fourier transforms, wavelet decomposition, information theory and/or domain expertise of the field studied. When working with accelerometers and gyroscopes Zhang and Sawchuk in [67] suggest a framework for features related to time domain and frequency domain.

53% of the articles pondered as high, are using domain knowledge only approach for the feature engineering process, while 27% of them extract features from Time – Frequency domain approach and 20% through other approaches disclosed later in the review. Many studies apply more than one approach, for instance, time domain and domain knowledge combined, or fast Fourier transform with domain knowledge.

Feature process and data techniques of the high weighed articles are grouped according to the feature engineering approach applied, inspecting data purity process, dimensionality reduction and evaluation. In Table 8 are these details of the studies using domain knowledge for feature engineering. In [50, 51] Target Features (TFs) refer to targets to achieve in terms of angles and distances (e.g., maximum knee angle flexion during squatting), Target Velocity Features (TVFs) describe the targets in terms of movement speed, while Postural Features (PFs) are constraints, angles or distances among anatomical landmarks that have to be maintained during the exercise. In [64], for dimensionality reduction, authors stablished their own method discarding features with poor reliability in a test-retest procedure, applying Leave-one-out Cross-validation (LOOCV), K-Fold and sum of errors and divide them by sample size. In [61] they combined movement and demographic features selecting the more important features, through a detailed process, that are more likely to improve recovery. Ferraris et al. [47] set also as goal to select kinematic parameters which best let algorithms to replicate clinical results.

As expected, domain knowledge approach group articles vary in their features. Researchers define the best features to reach their objectives and then, it is suggested to apply additional dimensionality reduction methods such as forward/backward feature selection, PCA, Elastic Nets and Relief algorithm to improve model performance. On the contrary, in Table 9, studies that applied Time-Frequency domain feature engineering approach show similarities among the features analyzed and most of them use accelerometers and gyros for capturing raw data.

Finally, in Table 10 studies were applying other feature engineering approaches that sometimes are combined with Time-Frequency domain or domain knowledge. For instance, in [17, 59] feature engineering is completed through Recurrence Quantification Analysis, in [54] with Spatial Temporal Cube Features in [48] Convolutional Pose Machines are used along with domain knowledge, and in [45] through Fast Fourier Transforms. In [45], the best angle is obtained by subtraction between the median of largest and smallest value.

The inspection of the studies selected for an in-depth analysis was intended to describe techniques used in data process rather than to validate their performance and feasibility. As described before, although studies have same intention of predicting assessments scores, measure rehabilitation progression or classify subjects between healthy or impaired, movement characteristics examined vary due to the complexity of feature engineering process. Before that, it is suggested to apply filters to correct raw data and reduce noise from sensors through a process calle data purity. In this process, filters are more common in studies using accelerometers and gyroscopes but also in those using joints tracking to smooth or correct joint position and/or orientation detected by built-in algorithms.

Few studies include in this process techniques to fill missing data, however reviewers could not identify risk of bias evaluation in the scrutiny of high weighed papers. Experts also recommend the implementation of feature selection algorithms after feature engineering process [21], such as forward/backward feature selection or regression analysis techniques that serves for both selection and regularization such as Lasso. It is also relevant although not included but disclosed in some studies in the scrutiny, feature scaling or normalization process for rescaling data when implementing scale-variant algorithms. Z-score, and min-max scaling are common techniques for pursuing that matter.

Finally, evaluation of performance in reviewed papers was commonly conducted by LOOCV, k-fold cross validation and the area under the receiver operating characteristic curve. Other valid methods are bootstrapping, class imbalance and F-1 score. Sensitivity is another performance metric applied by some studies in the review.

According to [68] Deep Learning (DL) is a set of algorithmic models composed of multiple processing layers to embody data representations with multiple levels of abstraction.

DL differs from other Machine Learning algorithms due to particular aspects, such as specific demands about the amount of data required: DL needs larger amount of training data, and its algorithms usually run on higher-end machines, among others [69].

In the search criteria selected for the authors, few studies using Deep Learning or Neural Networks techniques were found. EBSCO was used because of its metasearch engine functionality: it retrieves citations from other search engines which, in turn, provide a dependable and adequate coverage both for medical and engineering sources.

Since the number of citations retrieved by EBSCO exceeded the defined threshold, there was no need to utilize other search engines, such as Google Scholar. Furthermore, some authors suggest that, due to the way Google Scholar indexes citations, it may not be the best tool for use alone in systematic reviews [70]. Nevertheless, considering the improvements that DL algorithms, such as Convolutional Neural Networks, are producing in terms of object detection and recognition in real time, an additional basic search was performed using Google Scholar, and the following articles were selected from the first 100 results, based on how close the reported application is to the original search equation. To do so, the search applied set [Human Motor Ability of the Upper Limb deep learning] as words to be found within articles dated between 2014 and 2018 where words occur anywhere in the article and without excluding words. For this new search, authors only reviewed articles that included Deep Learning or Convolutional Neural Network words in their titles, a total of 20 were selected. Twelve of them were about electrical activities analysis such as EEG, EMG or biological signals. Other 3 studies were related to humanoids or robots, 1 about cognitive assessment and other 1 was a survey.

The remaining studies were related to biomechanics such as [71], where the researchers reported that they collected data from wearable sensors for detection of bradykinesia, comparing traditional machine learning algorithms with DL based on CNN. As other referenced studies in this review which uses accelerometers, features applied in this research are related to Mean, Maximum, Minimum of the signal amplitude, energy, skewness, among others. In [72], researchers analyzed tremor, gait and bradykinesia as well on patients with Parkinson but used a smartphone app that allowed them to carry out novel motor performance tests and self-assessments consisting of 17 observations that included kinetic, postural, resting tremor and target finger tapping. Time and Frequency features were used, as well as Butterworth High and Low Pass and Fast Fourier Transform to filter waveform data to be sent to a supervised Feed Forward ANN employing a five hidden layers architecture called Deep Multilayer Perceptron. Lastly, in [73], researchers also used wearables for human activity recognition through time/frequency data on an architecture that combined Long Short-Term Memory network, CNN and Deep Feed Forward structures.

In terms of DL suitability for human movement analysis, in [21], authors discuss several underutilized approaches to generate new knowledge because, even though DL is very useful for reducing the need for feature engineering, it produces better results when large datasets are available. Large training datasets are sometimes difficult to obtain in experiments that involve patients’ movement.