Research on recognition and application of hand gesture based on skin color and SVM

Abstract

Aiming at the problem that hand gesture recognition difficult issues in the real scene, a method of hand gesture recognition that combines skin color with SVM was proposed. The skin color area was separated by Otsu adaptive threshold algorithm in the YCbCr color space, and hand gesture was segmented by hand gesture area criterion. Hu moment features and finger number were extracted on the hand gesture contour as the feature vector. Six common static gestures were classified and recognized by SVM classifier. Experimental result showed that this method had good stability and real-time performance, average recognition rate could reach 94%. On the hand gesture recognition application, the hand gesture recognition results were converted into instructions, which achieved the real-time controlled simulation of the NAO robot in Webots simulation environment, and verified the feasibility of hand gesture recognition algorithm.

Keywords

Hand gesture recognition color segmentation Hu moments fingertip detection SVM classifier Webots

1. Introduction

With the rapid development of information technology, human-computer interaction technology occupies an increasingly important position in people’s life. Nowadays, in order to meet the needs of people’s life, hand gesture recognition as a kind of natural and humanized human-computer interaction way is increasingly adopted [1]. At present, many scholars have done a lot of work on hand gesture recognition technology. Such as Yang et al. [2] proposed a gesture recognition algorithm which was based on the main direction of gesture and Hausdorff-like distance template matching, and solved the problem of hand gesture recognition which was affected by hand gesture rotation, translation and scaling. But the experiment can only be carried out under the condition of the light stability, less noise and no human face interference. Dardas et al. [3] extracted the image scale invariant features and vectorization features, and then used bag-of-features and support vector ma-chine techniques to recognized hand gesture, and the recognition effect is better. However, due to the high computational complexity of SIFT algorithm, the recognition speed is slow and the real-time performance is poor. Tao et al. [4] trained image patch by the unsupervised sparse self-coding neural network, and extracted the edge features of hand gesture image as the input of the training classifier. Finally, they adjusted the parameters of the trained classifier to improve the accuracy. But it can be realized only in the limited background, which cannot be recognized under the realistic background.

In previous research, in order to improve the recognition rate, the hand gesture is usually recognized using a background-limited or a simple background method [5, 6]. It is impossible to exclude interferences such as face, illumination change, and similar skin color, etc. Therefore, it is not conducive to natural human-computer interaction. At present, there is no mature hand gesture recognition system that can be widely used in the real environment. Therefore, it is of great theoretical and practical value to improve the human-computer interaction.

2. Hand gesture recognition design process

In the real environment, hand gesture recognition is difficult due to complex background, light change and hand shape difference. In order to solve these problems, this paper proposes a method of hand gesture recognition based on skin color and SVM (Support Vector Machine), and this method is applied to the robot NAO control. The process of specific design is shown in Fig. 1.

Figure 1.

Hand gesture recognition and its application flow chart.

Firstly, the hand gesture image in real environment is collected through the camera. After pretreatment, the hand gesture is detected and segmented. Then the hand gesture features are extracted and the hand gesture recognition is performed by using the classifier. Finally, the hand gesture recognition results are transformed into instructions for robot real-time control simulation.

3. Hand gesture segmentation

3.1 Color space conversion

In hand gesture detection, there are three commonly used skin color spaces: RGB, HSV and YCbCr. Experiments show that the chrominance and luminance of YCbCr color space are separated from each other, and the $C b$ and $C r$ of skin color region have better clustering results under different luminance [7]. Therefore, we can detect two components of $C b$ and $C r$ , and select the threshold for these two components. The relationship between YCbCr and RGB color space is linear transformation. The amount of calculation is small and the effect is good, as shown in Eq. (1). Among them, $Y$ represents Luminance (grayscale value), $C r$ represents hue, and $C b$ represents saturation.

$\displaystyle\left[{\begin{array}[]{c}Y\\ Cb\\ Cr\\ \end{array}}\right]=\left[{\begin{array}[]{c}16\\ 128\\ 128\\ \end{array}}\right]+\left[{{\begin{array}[]{*{20}c}{65.481}&{128.553}&{24.966}% \\ {-37.797}&{-74.203}&{112.000}\\ {112.000}&{-93.786}&{-18.214}\\ \end{array}}}\right]\left[{\begin{array}[]{c}R\\ G\\ B\\ \end{array}}\right]$ (1)

3.2 Image binarization

According to the calculation statistics, the $C r$ and $C b$ chrominance components of the skin color show the two-dimensional normal distribution in the YCbCr color space. According to its two-dimensional normal distribution characteristics, Eq. (2) is used to analyze and calculate the skin color similarity of image pixels. The formula of similarity calculation is as follows:

$\displaystyle P(Cr,Cb)=\exp[-0.5(x-m)^{T}C^{-1}(x-m)]$ (2)

Among them, $m=E(x)$ is the mean of $C r$ and $C b$ chrominance components of the skin color sample, $x=({Cr},{Cb})^{T}$ is the chrominance component value of the skin color pixel $C r$ and $C b$ , $C={E[(x-m)(x-m)}^{T}]$ is the covariance matrix of skin color component.

After image normalization, the image is transformed into gray scale image with skin color similarity. Then Otsu dynamic adaptive threshold algorithm is used to segment the skin area. Otsu threshold method is a standard of measuring the biggest difference between target and background, namely, calculate the variance between them. When the variance reaches the maximum, the threshold is used as image segmentation threshold. When the between-class variance ( $\sigma_{B}^{2}$ ) reaches the maximum, its corresponding optimal threshold value as shown in Eq. (3).

$\displaystyle k^{*}=\textit{ArgMax}[\sigma_{B}^{2}]$ (3)

3.3 Hand gesture segmentation

Although the Otsu threshold segmentation algorithm can achieve accurate segmentation of the skin color region, the face and skin-like color objects still exist in the image, so we should try to remove the non-hand area. For the skin color area after binarization, set the hand gesture area determination condition as shown in Fig. 2.

Figure 2.

Hand gesture area determination flowchart.

(1)

In this paper, the hand shape of the closed outline is analyzed and recognized, and only the case of wearing long-sleeved clothes is considered. Therefore, in the image, there are only face and hand in the skin area of the human body. After image binaryzation processing, many experiments have shown that if some small area skin color areas (it may also be a skin-like color area) account for less than 0.02 of the whole image, then these areas are not human hands or faces, and they are eliminated. In Fig. 2, contArea represents small area skin color areas, and imgArea represents the whole image area.

(2)

The remaining skin color areas are only face and hand. Calculates the ratio (K) [8] of the height and width of the skin color area, and if it is within the range of [0.7, 3.0], it is the gesture area.

(3)

To recognize a gesture, the full gesture shape must appear in the window screen. If the skin color area of the image is connected to the collection window, no processing is done, whether it is a hand gesture or not, because the incomplete case will result in a misjudgment of the recognition result.

As shown in Fig. 3, the original image (a) is processed into an image (b) after binary processing, remove the non-gesture area and get the gesture image (c).

Figure 3.

Image binary processing.

4. Feature extraction

4.1 Hand gesture contour extraction

After obtaining the complete binary image, we need to find the appropriate and accurate features to describe the hand gesture, and at the same time, we should try to minimize the amount of computation. Therefore, this paper chooses the contour of the gesture area to be processed, that is, to extract the shape of the object.

In this paper, OpenCV function library is used in image processing. The gesture contour is retrieved from the binary image using the library function cvFindContours. When mode $=$ CV_RETR_EXTERNAL (retrieval mode), the outermost contour of the gesture image is retrieved. Then we use function cvDrawContours to draw hand gesture contour.

4.2 Feature extraction

In feature extraction, 7 Hu contour moments and fingertip numbers are extracted from hand gesture contour.

4.2.1 Hu contour moment

Hu invariant moment has the characteristics of image translation, rotation, and scale invariance. Compared with the commonly used silhouette moment [9], this paper only calculates the boundary moments of gestures, which can reduce the computation time and reduce the storage space.

4.2.2 Fingertip detection

Through multiple fingertip detection experiments, it is concluded that the fingertip number can be accurately obtained by combining fingertip curvature and gesture convex hull. The fingertip curvature is the position of the contour change larger [10], the outline convex hull is the smallest external convex polygon on all the vertices of the gesture contour [11], and its vertices are all the points on the contour.

Firstly, all the fingertip candidate points are determined by curvature algorithm, and convex hull algorithm is used to find the convex hull of the hand contour. Comparing the convex hull vertices with the candidate points, we get the finger points and count the fingertip numbers. The fingertip detection effect of gesture in video is shown in Fig. 4.

Figure 4.

Fingertip detection effect diagram.

4.3 Feature vector

According to the above analysis, the feature vectors for hand gesture recognition are composed of seven Hu moment features of Hu1 $\sim$ Hu7 and the number of fingertips Num, which are represented as (Hu1, Hu2, Hu3, Hu4, Hu5, Hu6, Hu7, Num). The specific steps of extracting feature vectors are as follows:

(1)
After the gesture is segmented, the contour is extracted from the gesture area.
(2)
Then use the Hu invariant moment formula to calculate the 7 Hu moments of the gesture contour.
(3)
The number of fingers is detected by gesture contour curvature and convex hull algorithm.
(4)
The 7 Hu invariant moments and the number of fingertips are combined to form a feature vector.

The recognition object uses 6 commonly used static gestures, as shown in Fig. 5.

Table 1
Feature vector of gesture 5

Hu1 Hu2 Hu3 Hu4 Hu5 Hu6 Hu7 Num

1 3.7042 11.3261 9.7546 12.6336 24.0362 19.8400 25.4955 5

2 3.4974 10.6760 10.5163 14.5491 27.6186 21.8451 27.7289 5

3 3.1998 10.8657 10.5376 13.6279 25.7144 18.3586 26.4586 5

4 3.6474 10.3261 9.8855 13.0783 25.6376 18.5704 24.5652 5

5 3.4484 10.8390 10.3234 13.0395 25.4635 19.0497 26.8066 5

6 3.1619 10.2385 9.7168 12.9864 25.1255 18.1472 24.6254 5

7 3.4669 10.7994 9.1626 12.9168 25.4983 18.8661 25.9213 5

8 3.3143 11.2345 9.7313 16.6977 28.8416 23.9020 30.5565 5

9 3.6322 10.5903 9.6593 13.2882 25.2549 20.2400 26.4936 5

10 3.3036 11.7526 10.9790 13.2670 24.4941 19.1554 25.6291 5

Figure 5.
Six kinds of gestures.

5. Hand gesture recognition

	Hu1	Hu2	Hu3	Hu4	Hu5	Hu6	Hu7	Num
1	3.7042	11.3261	9.7546	12.6336	24.0362	19.8400	25.4955	5
2	3.4974	10.6760	10.5163	14.5491	27.6186	21.8451	27.7289	5
3	3.1998	10.8657	10.5376	13.6279	25.7144	18.3586	26.4586	5
4	3.6474	10.3261	9.8855	13.0783	25.6376	18.5704	24.5652	5
5	3.4484	10.8390	10.3234	13.0395	25.4635	19.0497	26.8066	5
6	3.1619	10.2385	9.7168	12.9864	25.1255	18.1472	24.6254	5
7	3.4669	10.7994	9.1626	12.9168	25.4983	18.8661	25.9213	5
8	3.3143	11.2345	9.7313	16.6977	28.8416	23.9020	30.5565	5
9	3.6322	10.5903	9.6593	13.2882	25.2549	20.2400	26.4936	5
10	3.3036	11.7526	10.9790	13.2670	24.4941	19.1554	25.6291	5

In gesture recognition, the sample of hand gestures is limited, and a classifier for accurate classification of finite samples is needed [12]. Therefore, the support vector machine (SVM) proposed by Vapnik is used as a classifier for training samples and recognizing samples.

5.1 Database establishment

The database is built in 3 environments, with 5 experimenters, 6 common gestures, 10 times for each gesture, 900 samples, half as a training sample set and the other half as a test sample set. Each gesture collects 150 gesture images in various situations such as blurring, different backgrounds, angle rotation, size scaling and so on. Due to the length of this paper, the contour image and feature vector values of the gesture 5 under ten conditions are given, as shown in Fig. 6 and Table 1.

Figure 6.

Ten cases of gesture 5 and its contour image.

5.2 The use of LIBSVM software package

After establishing 6 gesture databases, SVM classifier is used to train and classify hand gesture samples respectively. In the classification process, the specific steps to use the LIBSVM software package are as follows:

(1)
First, the sample set data format is converted. The feature data of the 450 sets of training samples are stored in train_hand.txt, and the remaining 450 sets of feature data of the tested samples are stored in test_hand.txt. The svmscale function is used to scale the data, so that the range of feature value is unified.
(2)
Then, train and classify the samples. svmtrain function is used to train sample data to get the optimal parameter to train the classifier, and the training model is hand.modle. The test sample set (test.txt) is classified and recognized by svmpredict function and training model.

5.3 Analysis of experimental results

Support vector machine (SVM) and template matching method were used to recognize gesture, and 450 images in the experimental database were tested. By comparing the experimental results obtained by the two methods, the average recognition rate and average recognition time of 6 commonly used static gestures are obtained.

Table 2
6 kinds of hand gesture recognition rate

Gesture number	6 kinds of hand gesture recognition rate
	SVM	Template matching
1	0.95	0.86
2	0.92	0.82
3	0.93	0.87
4	0.88	0.81
5	0.97	0.89
6	0.96	0.91

Table 3

Experimental results of two methods

Recognition method	The total number of samples	Training set	Test set	Recognition rate	Recognition time
SVM	900	450	450	94%	58.7 (ms)
Template matching	900	450	450	86%	37.3 (ms)

From Table 2, we can see that the gesture recognition rate based on SVM is significantly higher than that of template matching method, and it can meet the requirement of the recognition rate. The recognition rate of gesture 4 is slightly lower. Through experimental analysis, most of the misjudged samples were recognized as gesture 3 and gesture 5. This is because in some cases, the Hu moment features of gesture 4 is similar to that of gesture 3 and gesture 5, difference between them is relatively small, resulting in misjudgment. In addition, the degree of specification of the gesture sample and the complexity of the gesture background are also the reasons for the misjudgment.

From Table 3, we can see that the hand gesture recognition algorithm based on SVM can ensure that the experimental results have a high recognition rate. Although the running efficiency is lower than that of template matching, it can also meet the real-time requirement and ensure the robustness and stability of gesture recognition system.

6. Robot control simulation based on hand gesture recognition

In the Webots robot simulation environment, hand gesture images are captured through the camera, and hand gesture recognition is performed using the algorithm mentioned above. Then the recognition results are transformed into control instructions and transmitted to the robot NAO, calling the relevant API functions of NAO and making corresponding actions. The gestures are used to control the robot in real time, enabling the robot to move forward, turn back, turn left, turn right, sit down and stand up.

In order to develop the algorithm in Webots environment with the characteristics of strong portability, short development cycle and high efficiency, the algorithm is developed based on third party computer vision library OpenCV. Transplant OpenCV to Webots environment and configure Webots compiler environment.

The simulation of the “world” is equivalent to the real world. It is required to set a certain number of steps when making “Move forward” gesture instruction (the maximum step is 16 steps in this paper), then automatically stop and execute the next instruction. When issuing the “Sit down” instruction, the next step is to issue “Stand up” instruction, and the system will automatically block other gesture commands. The time interval between the gestures is set to 3S, and the gesture image captured by the system will not be processed in 3S time. This can ensure that in the process of recognizing and executing instructions, even if the experimenter mistaken some gesture instructions, the system can shield the instructions and avoid interfering with the normal work of the robot. Instruction conversion is shown in Table 4.

Table 4
Translation table of hand gesture instruction

Gesture code	Instruction meaning	Gesture code	Instruction meaning
1	Move forward	4	Turn right
2	Turn back	5	Sit down
3	Turn left	6	Stand up

The simulation design of NAO robot control based on gesture recognition is as follows:

(1)

To control NAO in Webots, first open the controller, then edit the control program in the textbox, and initialize before designing the main program. The corresponding functions are as follows:

wb_robot_init();

(2)

Start the body and load the “action” file.

Drive body function:

Find_and_enable_devices();

Load the “action” file function:

load_motion_files();

(3)

When receiving the gesture code 1, call the forward loop function until the next gesture instruction is received, or completing the maximum step number (16 steps) and stop.

Forward loop function:

wbu_motion_set_loop(forwards,true);

wbu_motion_play(forwards);

(4)

When NAO receives the next gesture instruction, that is to change the current movement, then it needs to stop the current action and execute the new action.

Interrupt the current action:

wbu_motion_stop(currently_playing);

Execute new action: (motion represents the new action instruction)

wbu_motion_play(motion);

currently_playing $=$ motion;

(5)

In the NAO turn design, the angle parameter of left turn and right turn is set to 90 degrees, and the angle parameter of backward turn is set to 180 degrees. For example:

start_motion(turn_left_90);

start_motion(turn_right_90);

By calculating the response time of the robot after receiving instructions, the average response time of the system to six gesture instructions is about 65 ms. It has good real-time performance and can meet the requirement of the gesture control robot.

7. Conclusion

This paper focuses on the key technologies of hand gesture segmentation and recognition. The details include hand gesture segmentation, feature extraction, gesture recognition and gesture control robot. Finally, the stability of the algorithm and high recognition rate can be guaranteed in gesture recognition experiment, and the control simulation of the gesture to the robot also satisfies the good real-time performance.

The experiment completes the real-time control simulation of robot NAO by collecting gestures through the camera. In the next step, the algorithm will be implanted into the real robot NAO, and the robot “eyes” will replace the external camera, so that the robot can collect the gesture image and recognize it independently, then produce the corresponding action, and finally set the specific task to let the robot complete.

Footnotes

Acknowledgments

This work is supported by technology plan public project of Zhejiang Province in China (No. LGG18F040002), and supported by Natural Science Foundation of Zhejiang Province in China (No. LY19F020035). The authors thank the members of the Center for Advanced Life Cycle Engineering at the University of Maryland for their support of this work.

References

Chen

X.B.

and Xie

H.S.

, Hand gesture recognition based on bag of features, Computer Engineering & Design 34(3) (2013), 983–988.

Yang

X.W.

Feng

Z.Q.

Huang

Z.Z.

et al., Gesture recognition based on combining main direction of gesture and Hausdorff-like distance, Journal of Computer-Aided Design & Computer Graphics 28(1) (2016), 75–81.

Dardas

N.H.

and Georganas

N.D.

, Real-time hand gesture detection and recognition using bag-of-features and support vector ma-chine techniques, IEEE Transactions on Instrumentation and Measurement 60(11) (2011), 3592–3607.

Tao

M.P.

Huang

W.J.

et al., A gesture recognition based on unsupervised feature learning, Microelectronics & Computer 33(1) (2016), 100–103.

Sagayam

K.M.

and Hemanth

D.J.

, ABC algorithm based optimization of 1-D hidden Markov model for hand gesture recognition applications, Computers in Industry 99 (2018), 313–323.

Premaratne

and Nguyen

, Consumer electronics control system based on hand gesture moment invariants, IET Computer Vision 1(1) (2007), 35–41.

Xue

J.T.

Zong

Y.R.

and Yang

Z.L.

, Gesture recognition based on improved YCbCr space and multi-feature integration, Computer Applications and Software 33(1) (2016), 151–155.

Weng

H.L.

and Zhan

M.W.

, Vision-based hand gesture recognition with multiple cues, Computer Engineering & Science 34(2) (2012), 123–127.

Lee

D.L.

and You

W.S.

, Recognition of complex static hand gestures by using the wristband-based contour features, IET Image Processing 12(1) (2018), 80–87.

10.

Dai

Y.X.

T.B.

Yang

J.Y.

et al., An innovative method for fingertip and finger pointing detection based on contour features, Journal of Northeastern University (Natural Science) 39(7) (2018), 937–941.

11.

Kaehler

and Bradski

, Learning OpenCV3: Computer Vision in C++ with the OpenCV Library, O’Reilly Media, lnc, 2013.

12.

Leite

D.Q.

Duarte

J.C.

Neves

L.P.

et al., Hand gesture recognition from depth and infrared Kinect data for CAVE applications interaction, Multimedia Tools and Applications 76(20) (2017), 20423–20455.