Electromyogram-based hand gesture recognition robust to various arm postures

Abstract

In the recognition of electromyogram-based hand gestures, the recognition accuracy may be degraded during the actual stage of practical applications for various reasons such as electrode positioning bias and different subjects. Besides these, the change in electromyogram signals due to different arm postures even for identical hand gestures is also an important issue. We propose an electromyogram-based hand gesture recognition technique robust to diverse arm postures. The proposed method uses both the signals of the accelerometer and electromyogram simultaneously to recognize correct hand gestures even for various arm postures. For the recognition of hand gestures, the electromyogram signals are statistically modeled considering the arm postures. In the experiments, we compared the cases that took into account the arm postures with the cases that disregarded the arm postures for the recognition of hand gestures. In the cases in which varied arm postures were disregarded, the recognition accuracy for correct hand gestures was 54.1%, whereas the cases using the method proposed in this study showed an 85.7% average recognition accuracy for hand gestures, an improvement of more than 31.6%. In this study, accelerometer and electromyogram signals were used simultaneously, which compensated the effect of different arm postures on the electromyogram signals and therefore improved the recognition accuracy of hand gestures.

Keywords

Electromyogram accelerometer hand gesture arm posture pattern recognition

Introduction

Hand or arm gestures are types of nonverbal communication along with facial expressions. In particular, hand gestures are the most frequent type of nonverbal communication and represent diverse intentions. Numerous studies to identify the intentions of those gestures have been conducted. Among them, vision-based studies that employ camera sensors such as time-of-flight (ToF) or RGB cameras^1,2 were conducted and progressed to improve the recognition accuracy of hand gestures by incorporating the depth and color of images. However, those studies are vulnerable to surrounding brightness and have a limitation in recognizing hand gestures within the sight of the camera sensors. In addition, there have been studies that employed accelerometer, gyroscope, magnetoresistive, and flexure sensors embedded in data gloves to track hand motions or to recognize hand gestures.^3,4

Besides the employment of cameras and data gloves, electromyogram (EMG) signals were also used in studies to examine the recognition of hand gestures. The surface EMG signal is an electric signal generated in the muscles making intentional motions and is measurable from the surface of the skin. Studies employing the EMG signal to recognize the hand gestures are particularly popular in fields related to the control of prosthetic hands for patients.^5–11 Especially, in Atzori et al.,¹⁰ the first NINAPRO database with EMG of 27 intact subjects performing 10 successive repetitions of 52 hands, wrist, and forearm gestures was described in detail and characterized with global analyses and benchmark classification results. The results reported that the nonlinear support vector machine (SVM) and multi-layer perceptron (MLP) classifiers perform better than the linear SVM and linear discriminant analysis (LDA). Also, a versatile embedded platform for EMG acquisition and SVM-based gesture recognition was implemented for the real-time processing.¹¹ Recently, many studies have aimed to improve myoelectric control of prostheses.^12–14 It was reported that the displacement of the EMG electrodes and the positions of the arm during the use of the prosthesis affect the classification performance.^12,13 Also, the detailed information about clearing up commonly associated noises and artifacts from EMG signals were provided, and advantages and disadvantages of the various analytic methodologies were discussed to help researchers encounter the better method for analyzing EMG signals.¹⁴

Besides the prosthetic purpose to amputee subjects, another aim by EMG-based hand gesture recognition is to infer user intentions for the human–computer interface (HCI) and smart device interaction. Studies delving into active and intuitive recognition of hand gestures for normal and healthy people are in progress. Researches on hand gesture recognition based on EMG were initiated for the recognition of simple motions of the wrist of a subject and have evolved into the recognition of motions of individual fingers.^15,16 Furthermore, the works studied by Naik and Nguyen¹⁷ and Tsujimura et al.¹⁸ aimed at the recognition of multi-finger movements like those involved in playing rock–paper–scissors are currently in progress. In addition, the recognition of sign languages has been proposed by Kosmidou and Hadjileontiadis,¹⁹ Zhang et al.,²⁰ Li et al.,²¹ Cheng et al.,²² and Su et al.,²³ as well as those that employ EMG-based hand gesture recognition technologies to control the hand movement or driving motions of robots have been performed.^24,25 However, despite the high accuracy of gestures considering in many literatures, there still exist accuracy degradation in practical use. One of the major causes of such performance degradation is the change in EMG signals depending on the posture of the arm. The effect of arm posture on the performance of myoelectric control algorithm both on trans-radial amputees and able-bodied subjects was investigated.²⁶

For the information about arm postures, accelerometer signals have been jointly used with EMG signals in recognizing hand gestures. The use of accelerometer signals can be divided into two purposes. One is for recognizing complicate and numerous hand gestures such as sign languages and dexterous prosthetic hand control.^{19–21,23,27–29} The other is to prevent the performance degradation of the hand gesture recognition due to various arm postures, that is, even for the same hand gesture, the recognition accuracy decreases as arm postures change.³⁰

In this study, we aim to improve the robustness of hand gesture recognition to arm posture changes. Furthermore, we are targeting the interaction of able-bodied subjects such as HCI and smart device control. The EMG-based hand gesture interaction for healthy subjects in daily life has attracted more and more attention. But accelerometer and EMG sensor fusion techniques have not been widely applied to hand gesture interaction. Although the commercial product, MYO is a recently introduced forearm-band for EMG- and accelerometer-based gesture recognition for interactive applications, accelerometer, and EMG sensor fusion for hand gesture recognition is still in the initial stage. In addition, most researchers, as well as MYO (Armband, Thalmic Labs Inc., 2015), acquire EMG signals from the forearm. The reason is that most muscles related to hand gesture are anatomically placed in the forearm and thus good EMG signals bearing the gesture information can be obtained. But currently, most commercial wearable bands such as Fitbit and Jawbone are wrist-type bands and friendly acceptable to common users since they can be easily worn like fashion accessories. To integrate EMG-based interaction functions in wrist-type bands, we need to record the signals from wrist areas, not from the forearm. Here, we challenge this problem.

Our previous pilot study³¹ demonstrated that hand gesture recognition can be improved more than 20% for eight hand gestures when arm postures are considered. In the previous study, only EMG signals were used without accelerometer signals and we just investigated the effect of various arm postures on the performance degradation. Moreover, the EMG signals were recorded in the forearm. Also, although we used both accelerometer and EMG signals in Rhee et al.,²⁵ the study was mainly for the directional control of moving robot such as forward, backward, left, and right movements. This control was done by accelerometer signal only, and EMG signals were used for single hand gesture, fist. Inspired by but unlike to the previous study, in this study, we improve hand gesture recognition robust to arm posture changes with accelerometer and EMG signals acquired only from wrist areas.

We propose a hand gesture recognition method using EMG and accelerometer signals simultaneously to overcome the degradation of performance according to various arm postures. For the research purpose, we designed the sensing device to record accelerometer and EMG signals simultaneously. The accelerometer signals had distinct values corresponding to different arm postures. Thus, we can identify the arm postures using the accelerometer. Also, EMG signals for the gestures of different hands had different amplitudes and frequencies, and the recognition of the hand gestures was based on such distinct characteristics. Features that are typically used for accelerometer and EMG signals are the mean values²⁰ and the waveform lengths,¹⁰ respectively. For the classification, nine arm postures are recognized using the 1-nearest neighborhood based on the Euclidean distance and eight hand gestures are classified using the maximum likelihood estimation. EMG signals are statistically modeled using Gaussian models considering nine arm postures. In the experimental results, we compared the cases that took into account the arm postures with the cases that disregarded the arm postures for the recognition of hand gestures. Furthermore, the performance was evaluated using other existing classifiers such as linear/nonlinear SVM and k-nearest neighbor (kNN). To investigate the effect of training models, three different scenarios were established and demonstrated. These three models were made using data from each nine arm postures, one specific arm posture (local training model), and all arm postures (global training model).

Experiment environment

Figure 1(a) shows the device used to measure EMG and accelerometer signals simultaneously. The device consists of a four-channel EMG sensor, sensor connector, three-axial accelerometer sensor, battery, Bluetooth module, and micro-controller unit (MCU) configured with the dimensions of 12 cm in width and 6 cm in length. The EMG analog front end depicted in Figure 1(b) has four differential data acquisition channels. Each channel consists of an instrumentation amplifier, a low-pass filter, a high-pass filter, and a second amplifier. The instrumentation amplifier is chopper-stabilized to remove the flicker noise. The EMG signal of the instrument amplifier is then passed through a high-pass filter and a low-pass filter, which were designed for passing the frequencies ranging 10–400 Hz and then be amplified again. A digital notch filter was used to eliminate the power line noise (60 Hz) in the preprocessing. The power consumption is about 100 mW, which is very high compared with devices reported in literatures. Thus with four 1.2 V batteries, the working duration is about 1.5 h. Both EMG and accelerometer signals are sampled at 512 Hz using a 12-bit analog-to-digital (A/D) converter. Figure 1(c) shows the sensor position and muscle anatomy. Four EMG sensors are placed near wrist area. Bi-polar Ag-AgCl electrodes (Dual electrode #272, Noraxon USA Inc., USA) and snap electrodes (SECS-4, Laxtha Inc., Korea) were used. Four EMG sensors are placed near wrist area and the reference electrode is attached behind neck. The EMG and the accelerometer signals are acquired wearing the device near wrist area as shown in Figure 1(d).

Figure 1.

Experimental environment for acquiring both EMG and accelerometer signals: (a) acquisition device, (b) analog front end, (c) EMG sensor position, and (d) signal acquisition wearing the acquisition device.

Figure 2 illustrates the eight hand gestures and the nine arm postures. The hand gestures comprise flexion motions of five individual fingers, grab, pointing, and pinch. The arm postures are constructed by distinguishing between the angles of the elbow while the upper arm was anchored on the body and the angles during the supination and pronation of the wrist. The arm posture in $P_{1}$ consists of the elbow bent at $90^{\circ}$ and the palm facing the body. Arm postures $P_{2}$ and $P_{3}$ are thus defined with the motions of the wrist supinated or pronated by $90^{\circ}$ . The arm postures in $P_{4}$ and $P_{7}$ correspond to the angular motions of the elbow with the angle changed to $170^{\circ}$ and $30^{\circ}$ from the arm posture in $P_{1}$ . The arm postures in $P_{5}$ and $P_{6}$ and those in $P_{8}$ and $P_{9}$ are thus defined from the arm postures of $P_{4}$ and $P_{7}$ by supinating or pronating the wrist by $90^{\circ}$ , respectively.

Figure 2.

Model of (a) hand gestures and (b) arm postures.

Figure 3 presents the EMG and the accelerometer signals of hand gestures obtained from diverse arm postures. The arm postures were $P_{2}$ , $P_{6}$ , and $P_{7}$ and the hand gestures are $F_{1}$ and $F_{4}$ . We can observe that EMG signals and their waveform lengths are changing even for identical hand gestures according to various arm postures.

Figure 3.

Changes of raw EMGs and waveform lengths for different arm postures ( $F_{1}$ and $F_{4}$ gestures).

Proposed method

The proposed method in this study uses EMG and accelerometer signals simultaneously to prevent the degradation of hand gesture recognition due to varying arm postures. The signal processing consists of the feature extraction of the EMG and accelerometer signals, statistical modeling of hand gestures, and recognition.

Arm posture recognition using accelerometer

The accelerometer sensor can measure the three-dimensional motion of an object in the $x$ , $y$ , and $z$ axes. Figure 4 presents the three-axial accelerometer signals obtained by varying the arm postures from $P_{1}$ to $P_{9}$ .

Figure 4.

Accelerometer signals for nine arm postures.

Let $a^{i} [n] = [a_{x}^{i} [n], a_{y}^{i} [n], a_{z}^{i} [n]]$ be the accelerometer measured at the time instant, $n$ where $i$ denote an index of the arm posture, $P_{i}$ . We use the mean value of $a^{i} [n]$ as a feature

μ_{a}^{i} = \frac{1}{N} \sum_{n = 1}^{N} a^{i} [n]

(1)

Figure 5 shows the mean values of the accelerometer signals in the $x$ , $y$ , and $z$ axes. The feature distribution of the accelerometer signals forms a cluster for each arm posture and could be easily distinguished by calculating the nearest distance to each cluster. Therefore, the nearest neighborhood algorithm, which requires less computation, was used as a classifier. The Euclidean distance between the accelerometer sensor signal, $a [n] = [a_{x} [n], a_{y} [n], a_{z} [n]]$ , and the model $μ_{a}^{i}$ is calculated to classify the arm postures

{\hat{P}}_{i} = \underset{i}{\arg \min} ∥ a [n] - μ_{a}^{i} ∥

(2)

Figure 5.

Feature distribution of accelerometer signals.

Hand gesture recognition using EMG and accelerometer signals

The waveform length can quantify the dynamic complexity of EMG signals.^32,33 The waveform length is simply the cumulative difference of signals. Let the EMG signal be measured from channel $c$ for the hand gesture $F_{k}$ performed at the arm posture $P_{i}$ be $x_{i, k, c} [n]$ . Then, the waveform length $w_{i, k, c}$ can be expressed as in the following equation

w_{i, k, c} = \sum_{n = 1}^{N} | x_{i, k, c} [n - 1] - x_{i, k, c} [n] |

(3)

Then, we calculate the histogram, $h_{i, k, c} (m)$ of the waveform length, $w_{i, k, c}$ as in equation (4)

\begin{array}{l} I_{m} = {w_{i, k, c} [\cdot] | w_{\max} \frac{m - 1}{M} ≦ w_{i, k, c} < w_{\max} \frac{m}{M}} \\ m = 1, 2, \dots, M \end{array}

(4)

Figure 6 shows bins and the maximum value of the waveform length. Where $M$ determines the number of total bins and $w_{\max}$ indicates the maximum value of the waveform length.

Figure 6.

Bins and the maximum value of the waveform length.

Figure 7 shows the histograms of the waveform length obtained by repeatedly performing the eight hand gestures. The shape of the histogram reveals the placement of large values in the center while the rest of the values decrease bi-directionally along the lateral axis. Different distributions of the waveform lengths were observed even for the identical hand gestures depending on the arm postures. From the shape of the histograms, the underlying distribution is assumed to be Gaussian with the mean $μ_{i, k, c}$ and variance $σ_{i, k, c}^{2}$ . Using the values of the mean $μ_{i, k, c}$ and variance $σ_{i, k, c}^{2}$ , the Gaussian probability density function $f_{c} (w; P_{i}, F_{k})$ is generated as a probability model for the EMG for the hand gesture $F_{k}$ and the arm posture $P_{i}$

f_{c} (w; P_{i}, F_{k}) = \frac{1}{\sqrt{2 π σ_{i, k, c}^{2}}} \exp {- \frac{{(w - μ_{i, k, c})}^{2}}{2 σ_{i, k, c}^{2}}}

(5)

Figure 7.

Histograms of the waveform lengths of four-channel EMGs for eight hand gestures.

Figure 8 shows the box plot of the waveform length for hand gestures $F_{3}$ and $F_{7}$ according to nine arm postures. One-way analysis of variance (ANOVA) was used to compare waveform length for $F_{3}$ and $F_{7}$ . The waveform lengths were considered significantly different for p-values less than 0.05. The difference in the distributions of waveform length attributable to the varied arm postures negatively influences the correct discrimination of hand gestures. Therefore, we have constructed the statistical models of EMG incorporating arm postures, that is, $f_{c} (w; P_{i}, F_{k})$ instead of $f_{c}$ $(w; F_{k})$ , which is the conventionally used feature.

Figure 8.

Box plot of the waveform length for hand gestures: (a) $F_{3}$ and (b) $F_{7}$ according to nine arm postures $(P_{1} P_{9})$ .

For the discrimination of hand gesture, we employed maximum likelihood estimation (MLE), a parametric estimation method that maximizes the value of the likelihood function for the probability model of the observed samples and the observations. Assuming the EMG signals in different channels are independent, we use the likelihood function, $L (k)$ as

L (k) = Π_{c = 1}^{4} f_{c} (w; P_{i}, F_{k})

(6)

Then, the hand gesture is estimated by maximizing $L (k)$ as

{\hat{F}}_{k} = \underset{k}{\arg \min} L (k)

(7)

Experimental results

Eight hand gestures combined with nine arm postures were done by eight subjects in the experiments. The subjects do not have any physical problems in making hand gestures and are fully aware of the experimental process. The subjects perform 10 trials for single hand gesture and rest for 1 min to avoid the accumulation of physical fatigue. Each trial is executed in 1–2 s and the subjects have a rest for 5 s between trials. For 10 trials, it takes about 1 min. In the experiments, we informed subjects of the movement timing by beeping. Thus, in this experiments, the start timing is known. The segmentation is done for 1 s after the start of beeping. Thus, the classification delay is the almost same as 1 s. The results obtained from the experiment were classified into cases that solely employed EMG signals and those that simultaneously employed EMG and accelerometer signals for comparison. The parameters $N$ , $M$ , and $w_{\max}$ for the waveform length and the histogram were set to 512, 50, and 200, respectively. The eight hand gestures were repeated 50 times at each of nine arm postures to calculate the average recognition accuracy. Twenty-five data points randomly sampled a total of 50 trials were employed as the training set, and the remaining 25 data points were used as the test set for cross-validation, which was repeated 100 times to calculate the average recognition accuracy. The majority voting has not been applied to the training model.

In Figure 9, the hand gestures for $F_{1}$ , $F_{2}$ , and $F_{3}$ were recognized as the arm posture changes from $P_{4}$ to $P_{7}$ and $P_{8}$ sequentially and the recognized gestures are marked on the graph. In the case of the sole employment of EMG signals, the hand gestures $F_{2}$ and $F_{3}$ were incorrectly recognized as $F_{4}$ and $F_{5}$ , respectively. However, the recognition of hand gestures that took into account the corresponding arm postures correctly discriminated all hand gestures. The incorrectly recognized gestures are marked with rectangular boxes on the graph.

Figure 9.

Sampled results of hand gesture recognition.

The experimental results for the average recognition accuracy of eight hand gestures performed for each of nine arm postures are presented in Figure 10. In order to verify the performance of the proposed method, we compared it with the existing classifiers such as linear, nonlinear SVM, and kNN. The y-axis in Figure 10 denotes the actual arm postures and the x-axis indicates the inferred arm posture. The diagonal elements contained in the matrix are the average recognition accuracies results of hand gestures for correct arm posture recognition. The off-diagonal elements correspond to the average recognition accuracy of hand gestures.

Figure 10.

Comparison average of recognition accuracies over subjects and eight hand gestures for various classifiers for correct and incorrect arm posture recognition: (a) MLE, (b) kNN, (c) linear SVM, and (d) nonlinear SVM (average accuracy over eight hand gestures for correct arm posture recognition: 85.7 (MLE), 85.0 (kNN), 66.6 (linear SVM), and 80.4 (nonlinear SVM)).

The average recognition accuracies for hand gestures in cases in which the hand gestures were recognized by taking the corresponding arm postures into account and those recognized without taking the corresponding arm postures into account were calculated and are illustrated in Figure 11, and the averaged accuracy over all subjects, eight hand gestures, and nine arm postures and the performance improvement by considering the arm posture are summarized in Table 1. Figure 11 shows the average recognition accuracy for hand gesture and subject with and without considering arm postures. The white bar means the average accuracy without considering the arm postures. The improvement by taking the corresponding arm postures into account is represented by the black bar. The average recognition accuracies of hand gestures with the correct arm posture inference were 85.7%, which is 31.6% higher than without considering the corresponding arm postures.

Figure 11.

Comparing the average recognition accuracy of eight hand gestures using MLE, kNN, SVM with linear kernel and SVM with RBF kernel. The white bar means the average accuracy without considering the arm postures. The improvement by considering the arm postures is represented by the black bar: (a) average accuracy of eight hand gestures for each subject and (b) average accuracy of each hand gesture over subjects.

Table 1.

Average accuracy (mean ± SD, %).

	MLE	kNN	Linear SVM	Nonlinear SVM
Without arm posture	$54.1 \pm 7.8$	$56.4 \pm 8.2$	$45.1 \pm 6.4$	$51.8 \pm 7.3$
With arm posture	$85.7 \pm 5.4$	$85.0 \pm 4.9$	$66.6 \pm 5.7$	$80.5 \pm 4.3$
Performance improvement	$31.6$	$28.6$	$21.5$	$28.7$

SD: standard deviation; MLE: maximum likelihood estimation; kNN: k-nearest neighbor; SVM: support vector machine.

Although the degradation of the recognition performance of hand gestures can be overcome by considering the arm posture, we need various training models of hand gestures according to the arm postures. We investigated the effect of the training model on the performance from two scenarios building the models. The first scenario is that the training model is built using EMG signals in one specific arm posture and the recognition performance is evaluated for all other arm postures, that is, the training model is a local training model. The other is that the training model is made using EMG in nine arm postures and the performance is investigated for each posture, that is, the training model is a global model. The results of the two scenarios are shown in Figures 12 and 13, respectively. The recognition accuracies are decreased compared with ones in Figure 11, and the performance is better in the second scenario than in the first one except for kNN. Also, we can observe that the performance variation is not so high in MLE according to subjects or hand gestures compared with other three classifiers.

Figure 12.

Average recognition accuracy of hand gestures for the scenario building the training models: training with one posture a local training model and testing with all other postures (a) average accuracy of eight hand gestures for each subject and (b) average accuracy of each hand gesture over subjects.

Figure 13.

Average recognition accuracy of hand gestures for the scenario building the training models: training with all posture a global training model and testing with each postures: (a) average accuracy of eight hand gestures for each subject and (b) average accuracy of each hand gesture over subjects.

In addition, we have evaluated the effect of electrode position bias on the performance under the correct arm posture inference. The electrode position bias is generated by moving the center of EMG sensor arrays upward, downward, right, or left by 5 mm. The original center is set to that in Figure 1(c) and we define this position as $L_{5}$ . Thus, there are eight more different centers around $L_{5}$ . Figure 14(a) shows nine positions of the center. For example, $L_{1}$ is the position left and upward to $L_{5}$ and $L_{9}$ is the one right and downward to $L_{5}$ . In the results shown earlier than Figure 14, the $L_{5}$ position was used. The diagonal elements in Figure 14(b) mean the recognition accuracy when both training and testing are done in the same position and the off-diagonals correspond to the cases when the position bias occurs. By comparing the diagonals with the off-diagonals, we can easily see the accuracy without the bias is higher than with the bias. Also, among the off-diagonals, the accuracy generally decreases as the distance from the diagonals increases. This is because the hamming distance between two positions is proportional to the level of the position bias. For example, in the case when the training and testing positions are $L_{1}$ and $L_{9}$ , the hamming distance of the position bias is as high as 4 × 5(mm), and the accuracy is as low as 52.2% compared with the average of the diagonals, 87.0%. Note that the average recognition accuracy (87.0%) in Figure 14(b) is higher than that in Figures 10 and 11. Also, the reason why the performance in Figure 14(b) is higher than that in Figures 10 and 11 is that the results in Figure 14(b) are obtained from the subject 6, who shows the highest performance. The results in Figures 10 and 11 are the averaged performance for all eight subjects.

Figure 14.

Effects of electrode position bias on the recognition accuracies: (a) positions of electrode center and (b) recognition accuracies according to electrode position mismatch during the training and the test modes.

Controlling display devices using hand gestures

A human interactive model controlling display devices was established to evaluate the proposed method. Figure 15 shows the hand gestures and the corresponding commands for various display controls. The grab gesture $(F_{7})$ was used to control display on/off toggle. The selection and execution of the contents were performed using the index $(F_{2})$ , and the index and middle finger flexion $(F_{9})$ , respectively. The contents shift right/left on the selection screen were controlled by wrist flexion/extension gestures ( $F_{10}$ and $F_{11}$ ). The pinch grip $(F_{8})$ was allocated to hold the content item temporarily.

Figure 15.

Six hand gestures for control display and its commands: (a) display on/off, (b) contents selection, (c) contents execution, (d) contents shift right, (e) contents shift left, and (f) contents holding.

Seven males and one female were participated in experiments. The subjects performed the experiment in the order of Figure 15(a)–(f). The experimental results for the average accuracy of six hand gestures performed for each of nine arm postures are presented in Figure 16. The diagonal elements in the matrix are the average recognition accuracies results of hand gestures for correct arm posture. The off-diagonal elements correspond to the average recognition accuracy of hand gestures for different arm postures. Figure 17 shows the average accuracy for each hand gesture and each subject with and without considering arm postures and the performance improvement is summarized in Table 2. The white bar means the average accuracy without considering the arm postures. The improvement is represented by the black bar. The average recognition accuracies of hand gestures for display control with the correct arm posture was 90.3%, which is 28.7% higher than without considering the arm postures.

Figure 16.

Average of recognition accuracies of six hand gesture for various arm postures.

Figure 17.

Comparing the average recognition accuracy of six hand gestures: (a) average accuracy of six hand gestures and (b) average accuracy of various subjects.

Table 2.

Average accuracy of six hand gestures for control display (mean ± SD, %).

	MLE
Without arm posture	$61.5 \pm 9.9$
With arm posture	$90.3 \pm 5.2$
Performance improvement	$28.7$

SD: standard deviation; MLE: maximum likelihood estimation.

Discussion and conclusion

In this article, we have described a hand gesture recognition method using EMG and accelerometer signals simultaneously recorded from wrist area to overcome the degradation of performance according to various arm postures. Changes of EMG signals due to arm postures have been reported in various literatures, and many researchers have tried to solve the issue by using accelerometer signals. Peng²⁸ showed the relative movement of the muscle and sensor by the rotation of the arm, and Fougner³⁰ studied the gravitational and biomechanical effects of the limb position affect the EMG signal. Our approach also used accelerometer signals, and we have verified that experimentally the recognition performance is highly improved more than 31.6% for eight hand gestures. In this study, EMG signals were measured on the wrists differently from previous studies,^27–30 and our results indicated that gravity, biomechanics, and muscle rotation due to arm posture have a significant effect on the signals measured on the wrist.

The main contribution of the article is not simply the improvement of the recognition accuracy for hand gestures. Considering EMG-based interaction in wrist-type wearable bands, we have acquired both EMG and accelerometer signals from wrist area, not the forearm. Unlike to conventional studies, we have improved hand gesture recognition robust to arm posture changes with accelerometer and EMG signals acquired only from wrist areas. Although there are other issues to be solved to realize the EMG-based interaction in wrist-type bands such as the integration of EMG sensors in wearable band, the conductance improvement, the user-friendly design, and others, our research is the first attempt to recognize eight hand gestures robust to nine arm postures with weak EMG signals from wrist areas. However, the hand gestures we used are not as practical as the ones in the ARMYs standardized hand signals for close range engagement (CRE) operations and NINAPRO database. We will conduct research using practical hand gesture in the future.

Another contribution is the use of the MLE-based classifier, where EMG signals are statistically modeled using Gaussian models considering nine arm postures. MLE is known as the statistically optimum estimator when the probabilistic model is correct. Also, this classifier does not need additional parameters causing the performance dependency once the probabilistic model is obtained. In the case of Gaussian modeling, only the mean and the variance of features are needed. When updating the training model with new data, only Gaussian modeling is needed, that is, updating the mean and the variance. Some classifiers may need to recalculate decision boundaries with the updated whole data. Also, the MLE-based classifier is computationally efficient since most cost is in calculating the likelihood in equation (6). Although we have assumed EMG signals are independent for the mathematical convenience, the performance improvement may be expected if features are modeled using multi-dimensional Gaussian model. More importantly, to improve the robustness to the variation of arm postures, a method and experimental model need to be developed for continuous changes in arm posture, not just discrete changes.

Footnotes

Handling Editor: Gianluigi Ferrari

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was supported by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Science and ICT (No. 2017046246).

References

Garg

Aggarwal

Sofat

. Vision based hand gesture recognition. World Acad Sci Eng Technol 2009; 49: 972–977.

Van den Bergh

Van Gool

. Combining RGB and ToF cameras for real-time 3D hand gesture interaction. In: Proceedings of the 2011 IEEE workshop on applications of computer vision (WACV), Kona, HI, 5–7 January 2011, pp.66–72. New York: IEEE.

Amma

Gehrig

Schultz

Airwriting recognition using wearable motion sensors. In: Proceedings of the 1st augmented human international conference, Megve, 2–3 April 2010, p.10. New York: ACM.

Kim

J-H

Thang

Kim

T-S.

3-d hand motion tracking and gesture recognition using a data glove. In: Proceedings of the IEEE international symposium on industrial electronics, Seoul, South Korea, 5–8 July 2009, pp.1013–1018. New York: IEEE.

Matrone

Cipriani

Secco

et al . Principal components analysis based control of a multi-DoF underactuated prosthetic hand. J NeuroEng Rehabil 2010; 7: 16.

Schultz

Kuiken

. Quantifying pattern recognitionbased myoelectric control of multifunctional transradial prostheses. IEEE T Neural Syst Rehabil Eng 2010; 18: 185–192.

Zecca

Micera

Carrozza

et al . Control of multifunctional prosthetic hands by processing the electromyographic signal. Crit Rev Biomed Eng 2002; 30: 459–485.

Peerdeman

Boere

Witteveen

et al . Myoelectric forearm prostheses: state of the art from a user-centered perspective. J Rehabil Res Dev 2011; 48: 719–737.

Huang

Englehart

Hudgins

et al . A Gaussian mixture model based classification scheme for myoelectric control of powered upper limb prostheses. IEEE T Biomed Eng 2005; 52: 1801–1811.

10.

Atzori

Gijsberts

Kuzborskij

et al . Characterization of a benchmark database for myoelectric movement classification. IEEE T Neural Syst Rehabil Eng 2015; 23: 73–83.

11.

Benatti

Casamassima

Milosevic

et al . A versatile embedded platform for EMG acquisition and gesture recognition. IEEE T Biomed Circuits Syst 2015; 9: 620–630.

12.

Benatti

Farella

Gruppioni

et al . Analysis of robust implementation of an EMG pattern recognition based control. In: Proceedings of the international conference on bio-inspired systems and signal processing: Vol. 1: BIOSIGNALS, (BIOSTEC 2014), Angers, 3–6 March 2014, pp.45–54. Setbal: SciTePress.

13.

Young

Hargrove

Kuiken

. Improving myoelectric pattern recognition robustness to electrode shift by changing interelectrode distance and electrode configuration. IEEE T Biomed Eng 2012; 59: 645–652.

14.

Chowdhury

Reaz

Ali

MABM

et al . Surface electromyography signal processing and classification techniques. Sensors 2013; 13: 12431–12466.

15.

Ahsan

Ibrahimy

Khalifa

OO.

Electromygraphy (EMG) signal based hand gesture recognition using artificial neural network (ANN). In: Proceedings of the 4th international conference on mechatronics (ICOM), Kuala Lumpur, Malaysia, 17–19 May 2011, pp.1–6. New York: IEEE.

16.

K-J

Cha

K-M

Shin

H-C

. Maximum likelihood method for finger motion recognition from sEMG signals. In: Proceedings of the 13th international conference on biomedical engineering, Singapore, 3–6 December 2008, pp.452–455. Berlin: Springer.

17.

Naik

Nguyen

. Nonnegative matrix factorization for the identification of EMG finger movements: evaluation using matrix analysis. IEEE J Biomed Health Inf 2015; 19: 478–485.

18.

Tsujimura

Yamamoto

Izumi

. Finger gesture estimation based on forearm electromyogram signals. In: Proceedings of the 7th international symposium on image and signal processing and analysis (ISPA), Dubrovnik, 4–6 September 2011, pp.113–118. New York: IEEE.

19.

Kosmidou

Hadjileontiadis

. Sign language recognition using intrinsic-mode sample entropy on sEMG and accelerometer data. IEEE T Biomed Eng 2009; 56: 2879–2890.

20.

Zhang

Chen

et al . A framework for hand gesture recognition based on accelerometer and EMG sensors. IEEE T Syst Man Cy A 2011; 41: 1064–1076.

21.

Chen

Zhang

et al . A sign-component-based framework for Chinese sign language recognition using accelerometer and sEMG data. IEEE T Biomed Eng 2012; 59: 2695–2704.

22.

Cheng

Chen

Liu

et al . A novel phonology-and radical-coded Chinese sign language recognition framework using accelerometer and surface electromyography sensors. Sensors 2015; 15: 23303–23324.

23.

Chen

Cao

et al . Random forest-based recognition of isolated sign language subwords using data from accelerometers and surface electromyographic sensors. Sensors 2016; 16: 100.

24.

Zhao

Xie

Jiang

et al . EMG control for a five-fingered underactuated prosthetic hand based on wavelet transform and sample entropy. In: Proceedings of the IEEE/RSJ international conference on intelligent robots and systems, Beijing, China, 9–15 October 2006, pp.3215–3220. New York: IEEE.

25.

Rhee

You

K-J

Shin

H-C

. Intuitive robot control using surface EMG and accelerometer signals. Int J Elec Comput Eng 2013; 7: 1356–1359.

26.

Jiang

Muceli

Graimann

et al . Effect of arm position on the prediction of kinematics from EMG in amputees. Med Biol Eng Comput 2013; 51: 143–151.

27.

Gijsberts

Caputo

. Exploiting accelerometers to improve movement classification for prosthetics. In: Proceedings of the IEEE international conference on rehabilitation robotics (ICORR), Seattle, WA, 24–26 June 2013, pp.1–5. New York: IEEE.

28.

Peng

Hou

Chen

et al . Combined use of sEMG and accelerometer in hand motion classification considering forearm rotation. In: Proceedings of the 35th annual international conference of the IEEE engineering in medicine and biology society (EMBC), Osaka, Japan, 3–7 July 2013, pp.4227–4230. New York: IEEE.

29.

Atzori

Gijsberts

Mller

et al . Classification of hand movements in amputated subjects by sEMG and accelerometers. In: Proceedings of the 36th annual international conference of the IEEE engineering in medicine and biology society (EMBC), Chicago, IL, 26–30 August 2014, pp.3545–3549. New York: IEEE.

30.

Fougner

Scheme

Chan

et al . Resolving the limb position effect in myoelectric pattern recognition. IEEE T Neural Syst Rehabil Eng 2011; 19: 644–651.

31.

You

K-J

Rhee

K-W

Shin

H-C

. Finger motion decoding using EMG signals corresponding various arm postures. Exp Neurobiol 2010; 19: 54–61.

32.

Englehart

Hudgins

. A robust, real-time control scheme for multifunction myoelectric control. IEEE T Biomed Eng 2003; 50: 848–854.

33.

Powell

Kaliki

Thakor

. User training for pattern recognition-based myoelectric prostheses: improving phantom limb movement consistency and distinguishability. IEEE T Neural Syst Rehabil Eng 2014; 22: 522–532.