In-situ identification and recognition of multi-hand gestures using optimized deep residual network

Abstract

The real-time perception of hand gestures in a deprived environment is a demanding machine vision task. The hand recognition operations are more strenuous with different illumination conditions and varying backgrounds. Robust recognition and classification are the vital steps to support effective human-machine interaction (HMI), virtual reality, etc. In this paper, the real-time hand action recognition is performed by using an optimized Deep Residual Network model. It incorporates a RetinaNet model for hand detection and a Depthwise Separable Convolutional (DSC) layer for precise hand gesture recognition. The proposed model overcomes the class imbalance problems encountered by the conventional single-stage hand action recognition algorithms. The integrated DSC layer reduces the computational parameters and enhances the recognition speed. The model utilizes a ResNet-101 CNN architecture as a Feature extractor. The model is trained and evaluated on the MITI-HD dataset and compared with the benchmark datasets (NUSHP-II, Senz-3D). The network achieved a higher Precision and Recall value for an IoU value of 0.5. It is realized that the RetinaNet-DSC model using ResNet-101 backbone network obtained higher Precision (99.21 %for AP_0.5, 96.80%for AP_0.75) for MITI-HD Dataset. Higher performance metrics are obtained for a value of γ= 2 and α= 0.25. The SGD with a momentum optimizer outperformed the other optimizers (Adam, RMSprop) for the datasets considered in the studies. The prediction time of the optimized deep residual network is 82 ms.

Keywords

Hand gesture recognition RetinaNet ResNet-101 CNN human machine interaction

1 Introduction

Human-Machine Interaction (HMI) facilitates a user-friendly interface system by directly incorporating the human’s natural communication and manipulation skills [1]. It is a non-verbal type of communication with symbolic meanings like hand actions or gestures. Hand gestures are a meaningful hand motion that is used to convey important information or to interact with the machine. Hand Gestures are made accessible due to the recent developments in the field of pattern recognition and computer vision [2]. Hand gesture control has a wide range of applications like human-computer interaction, sign language recognition, robotic device manipulation, automobile control, video surveillance system, and teleoperation system [3].

The real-time hand action recognition in an unconstrained environment is challenging and time-consuming. Self-occlusions, computational speed, rapid motion, illumination variations, uncertain environments, and high degrees of freedom (DOF) are the few factors that drastically affect the accuracy of the recognition systems. [4].

The conventional approach to accomplish a hand action recognition using the hand’s convex hull after the skin segmentation method has been reported. This method is used in controlling the robotic arm and is extremely sensitive to dynamic background interference. Furthermore, the hand must be entirely confronting the aperture, which is impossible to achieve with complex hand gestures.

In recent years, numerous research work has been reported on hand action recognition using machine learning architectures. Hidden Markov Models (HMM), Support Vector Machines (SVM), Recurrent Neural Network (RNN) Convolutional Neural Networks (CNN), etc., are the various algorithms used in training the recognition models [5]. CNN in deep learning architectures is used to extract the information from the visual data [6] and learn to discriminate between unique samples by training with a large number of samples. The speed of the computing device limits the functioning of CNN and tends to increase the computational time. The latest development in the field of semiconductor fabrication accelerates the computational speed of graphical processing units. Therefore, the hardware computing power bottleneck is minimized.

Convolutional Neural Networks are incorporated in localizing the objects (hands) in the input frame [7]. The computational speed and high precision are the two main parameters that determine the efficiency of the hand gesture recognition algorithm. Hence, a robust and efficient method is required to enhance the user experiences in a real-time Human-Machine Interaction [1].

The CNN based algorithms used in the hand gesture recognition are, the two-stage hand action recognition (Faster R-CNN Inception-V2 model [8]) system, the single-stage hand action recognition (SSD Inception-V2 [9], SSD Lite MobileNet-V2 [10], YOLO-V2 [11],) system. The two-stage recognition models are capable of achieving high precision values but the prediction time is on a larger scale. The single-stage recognition models identify the hand gestures with low computational time. But, the precision is lower than two-stage models. Also, the single-stage deep CNN models encounter a class imbalance problem leading to a large number of mispredictions.

In this paper, an optimized deep residual network is proposed in the context of a real-time hand action recognition system. The optimized deep residual network consists of a RetinaNet model (single-stage architecture for hand detection) and, Depthwise Separable Convolutional (DSC) layers (precise hand gesture recognition).

2 Related works

Huang et al. [1] proposed a high-speed and robust technique for a hand gesture recognition system. The author encountered the challenges such as unconstrained background, camera focus angles, and illumination variations. It resulted in a large number of mispredictions. The author identifies the hand region based on skin color. The contour generation and segmentation process are carried out to separate the hand portion in the image. The features are extracted from the hand region. Then, the features are provided as inputs to the deep learning architecture for recognizing the hand gestures. The method reported by the author achieved an accuracy of 98.41%on their dataset.

Tripathi et al. [4] proposed a hand detection and gesture recognition model for controlling the autonomous vehicle. A single-stage object detection algorithm is used for hand detection. The triplet loss function and majority voting model are utilized. The author achieved an Average Precision of 95.04%on the Auto-G database.

Nuzzi et al. [7] used Faster R-CNN architecture for the hand detection system. The custom prediction function (CPF) is used to evaluate the confidential probability of predicted hands. CPF discards all the predictions with a confidence score of less than 90%. The author obtained an Average Precision of 95.51%on their dataset with a computational speed of 130 ms. This hand gesture recognition system is utilized for interaction with industrial robots.

Gao et al. [12] proposed a method to utilize the hand feature and pose. Single Shot Multibox Detector with ResNet-Inception model extracts the hand feature information. The forward kinematic tree of the skeleton structure identifies the human body pose and then predicts the positions of the hand. The author adopted this model for the hand-gesture-based interaction between humans and robots. The authors achieved an average precision of 85.97 %on the SRSSL database.

Chaman et al. [13] proposed a real-time model for the gesture to text (G2T) conversion and text to gesture (T2G) conversion in the Hindi language. This model assists in the speech and hearing of physically impaired people. The 32 different gestures are identified using the feature extraction algorithms based on a 5-bit binary string. The reported gesture recognition algorithm achieved an accuracy of 97.65 %on Hindi Sign Language datasets.

Kopuklu et al. [14] proposed a Convolutional Neural Network (CNN) framework for hand gesture recognition. The lightweight CNN architecture, ResNet-10 is used as a gesture detector deep architecture, ResNeXt-101 CNN model is employed to classify the detected gestures. The author evaluated the architecture and obtained a classification accuracy of 94.04 %and 83.82 %on EgoGesture and NVIDIA datasets.

Avola et al. [15] demonstrated a Recurrent Neural Network (RNN) for hand gesture recognition. The leap motion controller is used to acquire the selected hand features. RNN is trained by using the hand features formed by incorporating the key points (joint angle, fingertip position) in human hands. The author achieved an accuracy of 96%by utilizing the American Sign Language dataset.

Neethu et al. [16] used a connected component analysis algorithm to segment the fingertip regions from the hand image. The segmented regions of a finger are given as the input to the CNN classification algorithm. The author achieved an accuracy and sensitivity of 96.2%and 98.1%, respectively.

Islam et al. [17] suggested a hand tracking method using kernelized correlation filter and median-flow algorithm. The author achieved an accuracy of 98.44%.

Xu et al. [18] utilized a fused feature model for Hand Gesture Recognition in Automotive interfaces. The ResNext network is used to extract the features for static and dynamic data. The static data features are followed by a Feature Embedding Branch. The features of dynamic and static data are fused with discriminant correlation analysis. Finally, the hand gesture patterns are recognized using a linear SVM classifier. The authors reported an accuracy of 98.35%.

Tan et al. [19] used a convolutional neural network combined with space pyramid pooling (CNN–SPP), the technique for hand gesture recognition. The author tested the CNN-SPP technique on the NUS hand posture (NUSHP) dataset. The author obtained an accuracy of 98.40%.

Rubin et al. [8, 9] proposed two different architectures, namely, Faster R-CNN Inception-V2 CNN model and SSD Inception-V2 CNN model for hand action recognition in an unconstrained background. Faster R-CNN is a two-stage detector and SSD is a one-stage detector. In these two models, Inception-V2-based CNN is used as a feature extractor. The Average Precision obtained for the Faster R-CNN Inception-V2 model is 99.10%and 99%for the SSD Inception-V2 model on the MITI-HD dataset. The prediction time is about 46 ms for the SSD model. It is lower compared to Faster R-CNN. But, it resulted in a large number of mispredictions, when it is exposed to real-time multi-hand gestures.

Mohammed et al. [20] proposed an architecture based on a single-stage object detection algorithm for hand detection and a lightweight CNN network for classification. The author utilized static gestures. The authors reported an Average Precision of 97.90%for hand detection. Also, an Average Precision of 91.36%is reported by the authors for the gesture classification process under complex environments.

Lin et al. [21] designed a dense object detector called RetinaNet. RetinaNet overpowers the other object detectors like two-stage Region-based Convolutional Neural Network (R-CNN) and Single Shot Detector (SSD) in terms of its accuracy and class imbalance, without any change in speed.

Based on the detailed survey, the various approaches employed for hand gesture recognition have their own pros and cons. The traditional deep approach [1 , 18] is used for contour generation, segmentation of hand regions, feature extraction of hand, classification of hand regions, and hand tracking. The computational time of the traditional algorithms is high. The RNN model [15] is utilized for hand gesture recognition. The various CNN algorithms [4 , 18–20] are used in the process of hand action recognition. The CNN algorithms perform well in dealing with spatial data. The RNN is good at temporal data and poor for spatial data processing. The various two-stage CNN algorithms (Faster R-CNN + CPF [7], Faster R-CNN Inception-V2 [8]) and single-stage CNN algorithms (SSD Inception-V2 [9], SSD Lite MobileNet-V2 [10], YOLO-V2 [11], and SSD ResNet-Inception [12]) are reported in the context of hand action recognition system. But, it resulted in a class imbalance problem leading to a large number of mispredictions.

2.1 Contribution

In this paper, an optimized deep residual network is proposed for a real-time hand action recognition system. It is used to mitigate the class imbalance problem encountered by the existing Single-stage models. The optimized residual network incorporates a RetinaNet-DSC model with ResNet-101 as a feature extractor. The RetinaNet model is utilized as a hand detector because of its high accuracy. The ResNet-101 CNN algorithm is used as a feature extractor because of its reliable performance and time efficiency. The innovation of adding a Depth-wise Separable Convolutional (DSC) layer enhances the precision of hand gesture recognition and reduces the computing parameters. DSC is a tiny version of the CNN model utilized in the classification of the detected hand gesture. The RetinaNet-DSC model with ResNet-101 backbone architecture is trained and evaluated by utilizing the MITI-HD (custom created) dataset. The performance metrics of the RetinaNet-DSC model using ResNet-101 as the backbone network is compared with the metrics of the ResNet-50 network. It helps to identify the hand actions in real-time more precisely. The model’s performance on the MITI-HD dataset is also compared with the benchmark datasets (NUSHP-II, Senz-3D).

3 Methodology

The block diagram representation of the optimum residual network is shown in Fig. 1. The model is trained and evaluated using two publicly available online datasets as NUS Hand posture dataset-II (NUSHP- II), Senz 3D dataset (Senz-3D), and one custom created dataset (MITI-HD).

Fig. 1

Block diagram representation of Optimum Residual Network Architecture.

3.1 Datasets

NUSHP-II [22, 23] is the hand posture dataset obtained from dynamic backgrounds with different hand sizes, shapes, genders, and various age groups. It consists of 10 classes and 200 samples per class.

Senz-3D [24, 25] samples are collected from 4 persons with 10 different postures. The color, confidence, and depth frames are part of this dataset. Each posture includes 120 samples (Total = 1200 samples).

MITI Hand Dataset (MITI-HD) [9] is a custom-developed hand posture dataset collected from diverse people with varying skin tones, dynamic backgrounds, disparate sizes, illumination conditions, geometry, rapid motions, and various age groups. The dataset consists of static gestures and it has about 10 classes and 750 samples per class. The sample frames of each class from MITI-HD are shown in Fig. 2.

Fig. 2

Sample Frames of MITI-HD dataset.

3.2 Pre-processing

The images in the dataset are resized using the Adaptive interpolation method into 300×300 pixels. The information points that lie outside the image territories are loaded with white pixels, by maintaining their aspect ratio. The process of resizing is followed by the process of selecting the region of interest (ROI) called Annotation. The annotation is performed by a graphical image annotation tool known as LabelImg. It is a machine learning process of nominating the proposed region on frames. Then, the feature extraction and training process are initiated. The training/testing data split ratio is in the order of 80:20.

3.3 RetinaNet-DSC Architecture

3.3.1 RetinaNet

RetinaNet [18] is an efficient single-stage object detection algorithm. It significantly reduces the class imbalance problem than its successors (SSD [9], YOLO [26]). In this paper, a RetinaNet architecture with focal loss function is used in the process of real-time hand gesture recognition process. The RetinaNet model utilizes a Residual Network (ResNet-101) as a backbone architecture. It is used in the process of generating feature maps. RetinaNet model can also be operated with other classical backbone architecture like VGGNet [27], DenseNet [28], or MobileNet [29]. In real-time hand action recognition, the ResNet architecture is preferred because of its low complexity and reduced computational time than its successors. The algorithm of the Single-Stage Hand Action Recognition system using RetinaNet-DSC is given in Algorithm 1.

Algorithm 1. Algorithm for training a RetinaNet-DSC model for hand action recognition.

Input: Input Frames f from Dataset D, Feature Pyramids fp, Classes = C₁, C₂ . . . . C_n, Bounding box coordinates = b_x, b_y, b_w, b_h, Probability of object = P_o, Probability of class = P_c, Anchors = a (Scales: 1, 1.26, 1.587 and AR: 1:2, 1:1, 2:1), Layers = l
Output: Predicted hand signs with b and C.
for each frame f in Dataset Ddo
generate fp at various scales
for each fpdo top-down pathway and lateral connections
for each l in fpdo
predict P_o at each spatial location on a and C
end for
end for
for each ground truth Odo
regresses the offset for b and a
end for
for each ldo
predict C using DSC
ifP_c = 1, then hand sign recognized with b coordinates
(b_x, b_y, b_w, b_h)
else ifP_c = 0, then the hand sign is not recognized in the f
end if
end for

The proposed system also consists of a feature pyramid network (FPN) [30], a classification subnet module, and a box regression subnet module as shown in Fig. 1. The subnet is a small fully convolutional network (FCN) appended to each level of FPN. The output of the backbone architecture is considered as an input to the feature pyramid network. The feature maps are categorized using the classification subnet module. At last, the box regression subnet module is used in predicting the bounding box coordinates.

RetinaNet addresses the two important problems encountered by its forerunners. In Faster R-CNN, the feature maps are generated after passing the input data through numerous convolutional layers. This resulted in a loss of low-level information. Hence, objects of small size are not detected by the network. A large number of anchor boxes are generated to discover the regions that contain the object. But, most of the boxes might not be the right fit for the object. This results in forming the classifier to be biased and focuses its attention on the background to minimize the loss. RetinaNet excels in minimizing the above-specified issue by using an FPN on top of the backbone architecture.

FPN is similar to the SSD architecture, where the feature maps at different scales are generated. SSD model utilizes the upper layers for generating the feature maps. Whereas, the FPN module generates the feature maps from both the lower and upper layers. Therefore, a top-down pathway and lateral connections are used to combine the low and high-level features. It results in increasing the performance of the RetinaNet model in detecting both the smaller and larger objects.

The imbalance between the foreground and background classes are addressed in RetinaNet by incorporating the Focal loss ((F_L (ρ_t)) function. The loss function is an improved version of the cross-entropy loss. It is determined by adding a modulating factor to the balanced cross-entropy loss, with an adjustable focusing parameter represented in Equation 1. $F_{L} (ρ_{t}) = - α_{t} {(1 - ρ_{t})}^{γ} \log (ρ_{t})$ (1) where α_t is the weighting factor to address the class imbalance problem, γ is the tunable focusing parameter γ ≥0, (1 - ρ_t) ^γ is the modulating factor and -log(ρ_t) is the cross-entropy loss function, where $ρ_{t} = {\begin{matrix} ρ if γ = 1 \\ 1 - ρ otherwise \end{matrix}$ (2)

ρ is the estimated class probability of the model.

3.3.2 Depth-wise separable convolution (DSC)

The depth-wise separable convolution (DSC) approach enabled the development of small image recognition models. This DSC replaces the CNN blocks used in the last three FCN’s. The small model of CNN is capable of classifying the identified hand motions into one of the pre-determined classes. This design offers a high classification efficiency and speed with minimal consumption of memory and low computational cost. It incorporates a depth-wise convolution, which is a spatial convolution executed separately on each channel of input, and then proceeds on to a point-wise convolution [29].

In the depth-wise convolution process, the convolution operation is performed on a single channel at a particular time instant, whereas in conventional CNN’s it is accomplished for all the M channels. The filters of size D_k×D_k×1 are used. The output size of depth-wise convolution is D_p×D_p×M. The mathematical expression of depth-wise convolution is expressed in Equation 3.

$\begin{matrix} Depth - wise Conv {(W, y)}_{(i, j)} \\ = \sum_{k, l}^{K, L} W_{(k, l)} ⊙ y_{(i + k, j + l)} \end{matrix}$ (3)

In point-wise convolution, a convolution operation of 1×1 is performed to the M channels. The filter size of 1×1×M is used for N filters. The output size of point-wise convolution is D_p×D_p×N. The mathematical expression of point-wise convolution is expressed in Equation 4. $Point - wise Conv {(W, y)}_{(i, j)} = \sum_{m}^{M} W_{m} . y_{(i, j, m)}$ (4)

3.4 Non maximum suppression

The non-max suppression method is utilized to select the single entity out of the multiple overlapping entities. The bounding box with the highest probability (IoU≥0.5) is selected and all other bounding boxes are suppressed.

4 Experimental setup

4.1 Implementation

The real-time hand action recognition framework is put into effect using the Deep Learning toolbox of Python, Tensorflow as the backend. The architectures are trained using a computer (Intel^®Core ^TM i7-4790 CPU @ 3.60 GHz, 64-bit processor, 20GB RAM, Windows 10 PRO operating system) and GPU (NVIDIA GeForce GTX TITAN X (PASCAL)). To provide parallel computations in a GPU, CUDA/CuDNN is used. The set of python modules like Numpy, Matplotlib, Cython, Pandas, Open-CV, Tensorflow are also utilized.

4.2 Training

The RetinaNet-DSC model is trained by using the fine-tuned weights of the COCO dataset. The models are trained and tested for three different datasets like NUSHP-II, Senz-3D, and MITI-HD. The gradient descent optimization techniques such as SGD with Momentum optimization [31, 32], ADAM optimization [31, 33] and RMSprop optimization [31, 34] are employed for training.

4.3 Testing and prediction

The RetinaNet-DSC architecture is evaluated using the test sample data. The performance metrics (Average Precision (AP), Average Recall (AR), F1-Score (F1), and Computation Time) of the CNN models are obtained by varying the values of Intersection over Union (IoU). The IoU value determines the precision of the object detection process. The error obtained by matching the predicted bounding box with the ground-truth box is used in the determination of IoU values. The zero error result in the IoU value of unity. The prediction is said to be accurate if the determined IoU value is above 0.5. The IoU value of 0.5 is considered to be a true prediction.

5 Results and discussion

The Average Precision and Average Recall metrics are standard metrics that evaluate the efficiency of the detector at various bounding box levels overlapping with the labeled box [35, 36]. The evaluation metrics of the hand action recognition algorithms (RetinaNet-DSC ResNet-101, RetinaNet-DSC ResNet-50) are estimated for various IoU values like 0.5, 0.75 and 0.5:0.95.

The Average Precision of the hand in a frame less than 32×32 pixels is termed as AP_small, the hand size greater than 32×32 pixels and lower than 96×96 pixels is AP_medium and greater than 96×96 pixels is AP_large. The overall Average Precision of the total recognized hands with all the above scales is AP_all.

Similarly, the Average Recall is also defined as AR_small, AR_medium, and AR_large. The Average Recall for 1, 10, and 100 number of detections are termed as AR₁, AR_10, and AR₁₀₀, respectively.

Precision is the ability of a hand action recognition model to recognize the relevant gestures. It refers to the proportion of positive predictions that are precise. The recall is the potential of a model to identify all ground-truth bounding boxes. It is the proportion of positive predictions that are accurate out of all the possible ground-truths.

5.1 Optimization of deep residual network

The proposed RetinaNet-DSC model is trained with various Stochastic Gradient Descent (SGD) Optimization techniques. The SGD optimization algorithms like Adam [31, 33], SGD with Momentum [31, 32] and RMSprop [31, 34] are incorporated in the training process. The optimization algorithm is synchronized with 1 GPU with a batch size of 4 images. The RetinaNet-DSC models using ResNet-101, and ResNet-50 backbone networks are trained for 35 k iterations with an initial learning rate of 0.04 for SGD with Momentum optimizer, 0.0002 for Adam optimizer, and 0.004 for RMSprop optimizer with Momentum of 0.9. The learning rates are subsequently reduced by a factor at 10 k and 20 k iterations.

The training loss is the sum of the focal loss and the standard smooth L1 loss used for box regression. Training time ranges between 5 to 6 hours for every model. Each model is trained on the MITI-HD dataset using the three optimization techniques. To identify the robustness of the custom (MITI-HD) dataset, the models are also trained and tested with other benchmark datasets like (NUSHP-II and Senz-3D). The MITI-HD 300 represents the data samples of MITI HD datasets with a dimension of 300×300 pixels. The model’s performance is evaluated on the various optimization algorithms, various scales of data samples. For all three datasets, the deep residual model using Momentum optimizer produces a state of the art results over the other optimization models.

The results of RetinaNet-DSC ResNet-101 using IoU = 0.5:0.95 are listed in Table 1. The Average Precision and Average Recall are calculated for the RetinaNet-DSC ResNet-101 model on datasets (MITI-HD, NUSHP-II, and Senz-3D) using various optimization techniques (SGD with momentum, Adam, and RMSprop). The Average Precision and Average Recall values on the MITI-HD dataset using SGD with Momentum optimizer yields better results than Adam and RMSprop optimizers. The same trend is followed on NUSHP-II and Senz-3D datasets. For the Senz-3D dataset, the smaller-sized object (hand region less than 32×32 pixel) in the image frame is not recognized. The RetinaNet-DSC model using ResNet-101 (trained on MITI-HD dataset) resulted in higher Average Precision and Average Recall for all ranges of detections. The hand actions in the input frames with a size less than 32×32 pixels are efficiently detected.

Table 1
Performance metrics of RetinaNet-DSC ResNet-101 model obtained using various optimizers, and datasets for IoU of 0.5:0.95

Dataset Optimizer Average Precision (AP) Average Recall (AR)

AP_all AP_large AP_med AP_small AR₁ AR₁₀ AR₁₀₀ AR_large AR_med AR_small

MITI-HD 300 Adam 0.757 0.783 0.707 0.689 0.800 0.804 0.804 0.823 0.759 0.702

RMSprop 0.733 0.753 0.703 0.650 0.794 0.802 0.802 0.818 0.767 0.668

Momentum 0.789 0.811 0.732 0.725 0.819 0.824 0.825 0.846 0.777 0.711

NUSHP-II Adam 0.766 0.833 0.771 0.391 0.801 0.804 0.804 0.840 0.800 0.435

RMSprop 0.706 0.746 0.709 0.284 0.744 0.749 0.750 0.755 0.751 0.433

Momentum 0.768 0.837 0.772 0.398 0.802 0.804 0.805 0.845 0.806 0.450

SENZ-3D Adam 0.810 0.826 0.751 – 0.838 0.838 0.838 0.851 0.773 –

RMSprop 0.784 0.790 0.744 – 0.813 0.813 0.813 0.815 0.762 –

Momentum 0.823 0.833 0.781 – 0.851 0.851 0.851 0.861 0.796 –

Dataset	Optimizer	Average Precision (AP)	Average Recall (AR)
MITI-HD 300	Adam	0.757	0.783	0.707	0.689	0.800	0.804	0.804	0.823	0.759	0.702
	RMSprop	0.733	0.753	0.703	0.650	0.794	0.802	0.802	0.818	0.767	0.668
	Momentum	0.789	0.811	0.732	0.725	0.819	0.824	0.825	0.846	0.777	0.711
NUSHP-II	Adam	0.766	0.833	0.771	0.391	0.801	0.804	0.804	0.840	0.800	0.435
	RMSprop	0.706	0.746	0.709	0.284	0.744	0.749	0.750	0.755	0.751	0.433
	Momentum	0.768	0.837	0.772	0.398	0.802	0.804	0.805	0.845	0.806	0.450
SENZ-3D	Adam	0.810	0.826	0.751	–	0.838	0.838	0.838	0.851	0.773	–
	RMSprop	0.784	0.790	0.744	–	0.813	0.813	0.813	0.815	0.762	–
	Momentum	0.823	0.833	0.781	–	0.851	0.851	0.851	0.861	0.796	–

Table 2 exhibits the performance metrics of RetinaNet-DSC ResNet-50 for IoU 0.5:0.95. The Average Precision and Average Recall for the RetinaNet-DSC model are calculated using several optimization strategies (SGD with momentum, Adam, and RMSprop) on various datasets (MITI-HD, NUSHP-II, and Senz-3D). The SGD with momentum optimizer produces a higher value of Average Precision and Average Recall than the RMSprop and Adam optimizer, irrespective of the dataset considered. For the Senz-3D dataset, the hand detection of size less than 32×32 pixels (small) is fully terminated. However, for all detection ranges of the region, the RetinaNet-DSC with ResNet-50 model (trained on MITI-HD dataset) obtained higher Average Precision and Average Recall values.

Table 2

Performance metrics of RetinaNet-DSC ResNet-50 model obtained using various optimizers, and datasets for IoU of 0.5:0.95

Dataset	Optimizer	Average Precision (AP)				Average Recall (AR)
		AP_all	AP_large	AP_med	AP_small	AR₁	AR₁₀	AR₁₀₀	AR_large	AR_med	AR_small
MITI-HD 300	Adam	0.812	0.833	0.763	0.702	0.847	0.850	0.851	0.868	0.806	0.695
	RMSprop	0.785	0.808	0.738	0.678	0.826	0.829	0.830	0.849	0.782	0.658
	Momentum	0.821	0.845	0.765	0.706	0.851	0.854	0.855	0.876	0.807	0.715
NUSHP-II	Adam	0.765	0.853	0.768	0.382	0.801	0.803	0.804	0.865	0.806	0.467
	RMSprop	0.465	0.771	0.470	0.304	0.550	0.604	0.608	0.805	0.607	0.317
	Momentum	0.781	0.858	0.783	0.486	0.813	0.814	0.814	0.870	0.815	0.533
SENZ-3D	Adam	0.808	0.821	0.747	–	0.840	0.843	0.843	0.852	0.783	–
	RMSprop	0.564	0.546	0.595	–	0.657	0.706	0.706	0.703	0.680	–
	Momentum	0.834	0.840	0.796	–	0.858	0.858	0.858	0.859	0.815	–

Table 3 shows the evaluation metrics of the RetinaNet-DSC model using ResNet-101 and ResNet-50 for the IoU ranges of 0.5:0.95, 0.5, and 0.75 on MITI-HD, NUSHP-II, and Senz-3D datasets. The ResNet-101 network with SGD momentum optimizer produces an Average Precision of 0.992 (IoU = 0.5), 0.968 (IoU = 0.75), and 0.789 (IoU = 0.5:0.95) on MITI-HD dataset. For ResNet-50, the MITI-HD dataset obtained an Average Precision, 0.991 (IoU = 0.5), 0.974 (IoU = 0.75), and 0.821 (IoU = 0.5:0.95). The obtained evaluation metrics for the network ResNet-101 and ResNet-50 using MITI-HD dataset is in correlation with the performance metrics obtained by using the benchmark datasets.

Table 3

Average Precision of RetinaNet-DSC model obtained using various optimizers, and datasets for IoU range of 0.5:0.95, 0.5, and 0.75

Backbone	Dataset	MITI-HD 300			NUSHP-II			Senz-3D
Network	Optimizer	Adam	RMSprop	Momentum	Adam	RMSprop	Momentum	Adam	RMSprop	Momentum
ResNet-101	AP_0.5 :0.95	0.757	0.733	0.789	0.766	0.706	0.768	0.810	0.784	0.823
	AP_0.50	0.975	0.955	0.992	0.986	0.988	0.996	0.988	0.997	0.999
	AP_0.75	0.934	0.917	0.968	0.947	0.888	0.955	0.980	0.979	0.990
ResNet-50	AP_0.5 :0.95	0.812	0.785	0.821	0.765	0.465	0.781	0.808	0.564	0.834
	AP_0.50	0.988	0.989	0.991	0.998	0.714	0.999	0.992	0.828	0.999
	AP_0.75	0.968	0.955	0.974	0.957	0.569	0.964	0.981	0.674	0.991

From Tables 1 , and 3, it is noticed that the performance metrics of RetinaNet-DSC ResNet-101 and RetinaNet-DSC ResNet-50 models on the MITI-HD dataset are highly correlated with the evaluation metrics of benchmark datasets. The higher Precision and higher Recall values obtained using the proposed model enhance the recognition accuracy of the hand action recognition framework.

5.2 Accuracy/Speed trade-off

Table 4 lists the accuracy speed trade-off of the RetinaNet-DSC model with ResNet-101 and ResNet-50 backbone architectures on different image scales. The performance of RetinaNet-DSC architectures using MITI-HD 300 (image size = 300×300 pixels) is high compared to MITI-HD 160 (image size = 160×160 pixels). The Average Precision is computed for different IoU ranges such as 0.5, 0.75, and 0.5:0.95. The Average Recall and F1-score values are obtained for the true prediction range (IoU = 0.5). The Average Precision, Average Recall, and F1-score obtained by utilizing the ResNet-101 model is 99.21 %, 96.99 %, and 98.10 %. Utilizing the ResNet-50 model resulted in 99.13 %, 96.93 %, and 98.01 %. The SGD with momentum optimizer resulted in obtaining higher performance metrics, irrespective of the backbone architectures.

Table 4
Accuracy/speed trade-off of RetinaNet-DSC model for various scales of images (MITI-HD 300, MITI-HD 160)

Dataset MITI-HD 300 MITI-HD 160

Backbone ResNet-101 ResNet-50 ResNet-101 ResNet-50

Optimizer Adam RMS prop Momentum Adam RMS prop Momentum Adam RMS prop Momentum Adam RMS prop Momentum

AP_0.5 :0.95 0.757 0.733 0.789 0.812 0.785 0.821 0.777 0.700 0.801 0.805 0.689 0.819

AP_0.50 0.975 0.955 0.992 0.988 0.989 0.991 0.987 0.939 0.992 0.990 0.924 0.991

AP_0.75 0.934 0.917 0.968 0.968 0.955 0.974 0.950 0.865 0.974 0.966 0.901 0.974

AR_0.50 0.946 0.934 0.969 0.946 0.942 0.969 0.935 0.921 0.967 0.943 0.917 0.965

F1-score_0.50 0.960 0.944 0.981 0.967 0.965 0.980 0.960 0.930 0.979 0.966 0.920 0.978

Prediction Time (ms) 82 82 82 64 64 64 80 80 80 60 60 60

Dataset	MITI-HD 300	MITI-HD 160
AP_0.5 :0.95	0.757	0.733	0.789	0.812	0.785	0.821	0.777	0.700	0.801	0.805	0.689	0.819
AP_0.50	0.975	0.955	0.992	0.988	0.989	0.991	0.987	0.939	0.992	0.990	0.924	0.991
AP_0.75	0.934	0.917	0.968	0.968	0.955	0.974	0.950	0.865	0.974	0.966	0.901	0.974
AR_0.50	0.946	0.934	0.969	0.946	0.942	0.969	0.935	0.921	0.967	0.943	0.917	0.965
F1-score_0.50	0.960	0.944	0.981	0.967	0.965	0.980	0.960	0.930	0.979	0.966	0.920	0.978
Prediction Time (ms)	82	82	82	64	64	64	80	80	80	60	60	60

The utilization of focal loss function in the proposed model resulted in reducing the number of misclassifications. It is an enhanced version of cross-entropy loss. It deals with the disparity of classes by focused training (adding more weights) on easily misclassified examples (hard or background objects) and non-focused training (reducing the weights) on easily classified examples (foreground objects). It resulted in the improvement of Average Recall values for the proposed model [35, 36]. The incorporation of a depth-wise separable convolutional layer enhances the accuracy of hand gesture recognition.

A good hand detector is expected to recognize all the ground-truth hand gestures (high Average Recall) and also to identify the appropriate hand gestures (high Average Precision). For a successful hand gesture recognition method, the Average Precision remains high as long as its Average Recall value is on a high scale. The Average Recall and Average Precision are retained as high, even if there is a change in the confidence threshold values [36]. The proposed RetinaNet-DSC model satisfies the conditions of a good hand gesture recognizer.

The prediction time of the RetinaNet-DSC model with ResNet-101 on MITI-HD 300 is 82 ms, and MITI-HD160 is 80 ms. The prediction time with ResNet-50 on MITI-HD 300 is 64 ms, and MITI-HD160 is 60 ms. The prediction time of a real-time hand gesture recognition system using the RetinaNet-DSC model with ResNet-50 is comparatively less than that of the ResNet-101 model.

The Average Precision of the individual classes of hand gestures for the RetinaNet-DSC model is shown in Fig. 3. The models imparted SGD with Momentum optimizer for training resulted in higher value for Average Precision in all the classes than its counterparts.

Fig. 3

Average Precision curves of RetinaNet-DSC models (γ= 2 and α= 0.25) for each class on the MITI-HD dataset.

Table 5 shows the performance metrics of the RetinaNet-DSC model using ResNet-101 and ResNet-50 by varying γ (tunable focusing parameter) for Focal Loss with optimal α (weighting factor). To address the class imbalance problem in the hand action recognition system, the tunable focusing parameter γ is varied with an optimal weighting factor α. By keeping the value of γ as fixed and varying α values (0.25, 0.5, and 0.75), the performance of the hand recognition models is noted. After a deep investigation, it is observed that the RetinaNet-DSC models with ResNet-101 and ResNet-50 architectures resulted in higher Average Precision value for γ= 2 and α= 0.25.

Table 5

Performance metrics of RetinaNet-DSC by varying γ for Focal Loss with optimal α

γ	α	ResNet-101						ResNet-50
		AP_all (%)	AP_0.50 (%)	AP_0.75 (%)	AP_small (%)	AP_medium (%)	AP_large (%)	AP_all (%)	AP_0.50 (%)	AP_0.75 (%)	AP_small (%)	AP_medium (%)	AP_large (%)
1	0.25	73.73	95.66	91.16	66.72	71.07	75.48	81.8	99.10	97.16	68.56	76.10	84.29
1	0.5	74.32	96.76	92.09	67.91	72.43	75.46	81.48	99.13	97.4	68.72	75.84	83.96
1	0.75	73.06	97.20	89.20	67.82	71.60	74.09	81.46	99.13	97.4	68.7	75.99	83.99
2	0.25	78.89	99.21	96.43	72.53	75.45	81.08	82.09	99.13	97.46	70.6	76.50	84.50
2	0.5	78.77	98.40	95.77	72.10	73.19	80.37	81.56	99.13	97.20	70.42	76.17	83.85
2	0.75	72.50	95.25	90.32	68.27	70.23	73.79	81.10	98.91	97.04	69.58	75.59	83.61
5	0.25	71.46	92.77	89.57	66.56	72.47	72.71	81.01	98.59	97.15	67.95	75.60	83.54
5	0.5	74.13	96.77	92.98	69.89	73.96	75.52	78.09	96.12	94.37	65.23	71.80	80.99
5	0.75	75.42	96.50	93.20	69.86	72.53	77.44	80.63	97.96	96.34	66.17	74.62	83.34

Initially, the proposed RetinaNet-DSC model is tested on the custom (MITI-HD 300) dataset and the performance metric results are compared with the two publicly available benchmark (NUSHP-II, Senz-3D) datasets. The performance metrics comparison of the proposed model on three datasets is illustrated in Table 6.

Table 6

Performance metrics comparison of the proposed RetinaNet-DSC model using MITI-HD, NUSHP-II, and Senz-3D datasets

Models	Backbone Network	Dataset	Classes	Train Samples/Test Samples	AP_0.5 :0.95 (%)	AP_0.75 (%)	AP_0.50 (%)
RetinaNet-DSC	ResNet-101	NUSHP-II	10	2200 / 550	76.80	95.50	99.60
		Senz-3D	10	960 / 240	82.30	99.00	99.90
		MITI-HD 300	10	5581 / 1424	78.90	96.80	99.20
	ResNet-50	NUSHP-II	10	2200 / 550	78.10	96.40	99.90
		Senz-3D	10	960 / 240	83.40	99.10	99.99
		MITI-HD 300	10	5581 / 1424	82.10	97.40	99.10

MITI-HD 300 dataset has more images compared to other benchmark datasets as visualized in Table 6. The complexity (variation in lighting condition, background, skin tone, shape, and size of hand geometry) of images in the MITI-HD 300 dataset is on a high scale when compared with other datasets. From Table 6, it is also observed that the performance metrics obtained by the RetinaNet-DSC model on MITI-HD 300 dataset are comparable with the results obtained by using NUSHP-II and Senz-3D datasets. The slight deviation in the performance metrics is related to the number of samples in the datasets and the complexity in input images.

Table 7 lists the performance metrics obtained by the RetinaNet-DSC model on MITI-HD dataset. The obtained results are compared with other methods such as SSD [9] and Faster R-CNN [8]. It is observed that the proposed RetinaNet-DSC model performed better in terms of improvement in performance metrics and minimizing the number of mispredictions. This is associated with the introduction of the Depth-wise Separable Convolution layer and Focal loss function in the architecture.

Table 7

Performance metrics comparison of the proposed RetinaNet-DSC model with the existing models on the MITI-HD Dataset

Models	Backbone	Dataset	AP_0.50 (%)	AR_0.50 (%)	F1-Score_0.50 (%)	Prediction Time (ms)	Mis-Predictions
RetinaNet-DSC	ResNet-101	MITI-HD 300	99.21	96.99	98.10	82	No
RetinaNet-DSC	ResNet-101	MITI-HD 160	99.20	96.70	97.90	80	No
RetinaNet-DSC	ResNet-50	MITI-HD 300	99.13	96.93	98.01	64	No
RetinaNet-DSC	ResNet-50	MITI-HD 160	99.10	96.50	97.80	60	No
SSD [9]	Inception-V2	MITI-HD 300	99.00	95.40	97.17	46	More
Faster R-CNN [8]	Inception-V2	MITI-HD 300	99.10	96.78	97.98	140	Less

The performance metrics obtained by the RetinaNet-DSC model on the MITI-HD dataset are compared with other models (RetinaNet-50 [20], Deep CNN [37], Faster R-CNN + CPF [7]) evaluated on different datasets as listed in Table 8. It is observed that the results obtained by the RetinaNet-DSC model are higher when compared with other reported methods. Since, the RetinaNet-DSC model resulted in the state-of-the-art result when evaluated on custom created MITI-HD dataset and Benchmark datasets (NUSHP-II, Senz-3D). The proposed model is also expected to perform better on other datasets such as 5-Signers [20], NUSHP [37], and Hand [7]. The comparison of the model’s performance metrics with the different datasets is carried out to illustrate the superiority of the proposed method. As a future endeavor, the proposed model will be evaluated on datasets such as 5-Signers, NUSHP and Hand.

Table 8

Performance metrics comparison of the proposed RetinaNet-DSC model with the existing models on different datasets

Models	Dataset	Train Samples/Test Samples	AP_0.50 (%)	AR_0.50 (%)	F1-Score_0.50 (%)	Prediction Time (ms)
RetinaNet-DSC ResNet-101	MITI-HD 300	5581 / 1424	99.21	96.99	98.10	82
RetinaNet-DSC ResNet-50	MITI-HD 300	5581 / 1424	99.13	96.93	98.01	64
RetinaNet-50 [20]	5-Signers	3935 / 2000	97.90	90.1	93.8	83
Deep CNN [37]	NUSHP	1600/400	94.96	94.85	94.26	–
Faster R–CNN + CPF [7]	Hand	1306 / 327	95.51	94.71	95.11	130

The total training loss function of RetinaNet-DSC models using residual backbone architectures (ResNet-101, ResNet-50) for the MITI-HD 300 and MITI-HD 160 datasets are shown in Fig. 4. (a), (b), (c), and (d), respectively. Based on the loss plot of ResNet-101 and ResNet-50 backbone architectures, the models trained using SGD with momentum optimizer obtained low loss than the other optimizer.

Fig. 4

Training Loss Curves of RetinaNet-DSC model (a) ResNet-50 MITI-HD 300×300 pixels (b) ResNet-50 MITI-HD 160×160 pixels (c) ResNet-101 MITI-HD 300×300 pixels (d) ResNet-101 MITI-HD 160×160 pixels.

5.3 Implementation of real-time hand action recognition system

The experimental setup for the real-time Single and Multi-hand actions recognition in an unpredictable environment is shown in Fig. 5. The models are trained using single-hand gestures. Data augmentation process like mirroring, flipping and rotation is carried out during the training process. The trained model is used to perform recognition for the right hand, left hand, and multi-hands in a frame. The hand gesture recognition system using RetinaNet-DSC is implemented using the hardware modules that are discussed in section 4.1. The web camera (Model No: Quantum QHM495LM, 25 MP) is used to capture the hand gestures in real-time. The detection and classification of hand gestures are recorded for the proposed RetinaNet-DSC model using ResNet-101 and ResNet-50 backbone architectures. The RetinaNet-DSC based models are efficient to recognize the hands under a dynamic environment. The predicted sample frames of real-time hand action recognition are shown in Figs. 6 , and 8, respectively. The sample frames are captured at various distances between the optical sensors and object, various skin tones, illumination, and complex background. Figure 6 (a), (b), and (c) shows the predicted samples of single-hand gestures at a distance of 100 cm, 150 cm, and 200 cm, respectively. Figure 7(a), (b), and (c) shows the predicted samples of two hands with gestures at a distance of 150 cm, 200 cm, and 250 cm, respectively. The real-time frames of multi-hand gesture recognition with a higher confidence score are shown in Fig. 8.

Fig. 5

RetinaNet-DSC based real-time hand action recognition framework.

Fig. 6

Sample frames were obtained by varying the distance between the optical sensor and object for The real-time Single Hand Action Recognition system (a) 100 cm (b) 150 cm (c) 200 cm.

Fig. 7

Sample frames were obtained by varying the distance between the optical sensor and object for the real-time Two-Hand Action Recognition system (a) 150 cm (b) 200 cm (c) 250 cm.

Fig. 8

A real-time Multi-Hand Action Recognition using RetinaNet-DSC CNN architectures.

Based on Fig. 9, it is observed that the proposed RetinaNet-DSC architectures using ResNet-101, ResNet-50 backbone networks identify the gestures correctly with lower prediction time when compared with the Faster R-CNN Inception-V2 model (140 ms). Even though the SSD model produces the results at a lower computation time (46 ms) when compared to the proposed RetinaNet-DSC model, it resulted in a large number of mispredictions. The improvement in the Average Recall value is modest for the proposed architecture compared to SSD and Faster R-CNN models, but the number of mispredictions is greatly reduced. The RetinaNet-DSC model with ResNet-101 and ResNet-50 backbone network produces similar performance metrics when it is evaluated on MITI-HD 300 dataset. Hence, any of the above-specified backbone networks can be utilized with the RetinaNet-DSC model for real-time implementation of hand gestures. In this work, the RetinaNet-DSC model with ResNet-101 backbone network is considered based on its prediction accuracy. However, for faster recognition of real-time hand gestures the RetinaNet-DSC model with ResNet-50 backbone network could be considered.

Fig. 9

Multi-Hand Action Recognition using RetinaNet-DSC, (a), (b) Output of existing models with mispredictions, (c), (d) Output of proposed models without mispredictions with varying confidence scores.

6 Conclusion

Real-time hand action recognition is vital to perform numerous human-machine interactions. The optimum Deep Residual Network (RetinaNet-DSC) architecture is used for real-time hand action recognition. Experiments are carried out using a custom (MITI-HD) dataset and on benchmark (NUSHP-II, Senz-3D) datasets to reveal the effectiveness and robustness of the proposed technique. The performance of the proposed deep Residual models on ResNet-101 architecture is analyzed with a learning rate of 0.0001 for SGD with momentum and Adam optimizers, and 0.004 for RMSprop, for 35,000 training steps. The performance is compared with the ResNet-50 CNN architecture. SGD with momentum optimizer outperformed for both the number of layers (101, 50) of the model. The performance metrics (Average Precision, Average Recall, F1-Score) obtained by the RetinaNet-DSC model are listed below. For ResNet-101 architecture, the obtained performance metrics on IoU = 0.5 are 99.21%, 96.99%and 98.10%. The values of 99.13%(AP), 96.93%(AR) and 98.01%(F1), are obtained by using the ResNet-50 architecture (IoU = 0.5). The prediction time of the RetinaNet-DSC model with 50 layers (64 ms on MITI-HD 300 and 60 ms on MITI-HD 160) is comparatively lesser than the 101 layers (82 ms on MITI-HD 300 and 80 ms on MITI-HD 160). The RetinaNet-DSC model performs better for γ= 2 and α= 0.25. The number of mispredictions is significantly reduced by utilizing the proposed model. Hence the proposed model is ideal for a real-time hand action recognition system. The reduction in computation time will further enhance the system performance, which is considered to be our future scope.

Footnotes

Acknowledgments

The authors are grateful to NVIDIA for providing the GPU (Model No: NVIDIA TitanX) under University Research Grant Initiative.

References

Huang

, Chong

, Nie

and Pan

, Hand Gesture Recognition with Skin Detection and Deep Learning Method, IOP: Journal of Physics: Conf. Series 1213 (2019), 022001, doi: 10.1088/1742-6596/1213/2/022001.

Abavisani

, Joze

H.R.V.

and Patel

V.M.

, Improving the performance of unimodal dynamic hand-gesture recognition with multimodal training, IEEE Explorer IEEE Conference on Computer Vision and Pattern Recognition (CVPR), (2019), 1165–1174.

Zengeler

, Kopinski

and Handmann

, Hand gesture recognition in automotive human–machine interaction using depth cameras, Sensors 19(1) (2019), 59.

Tripathi

, Keshari

, Ghosh

, Vatsa

and Singh

, AUTO-G: Gesture Recognition in the Crowd for Autonomous Vehicle, IEEE Explorer, International Conference on Image Processing (ICIP), Taipei, Taiwan, (2019), 3482–3486.

Chung

, Chung

and Tsai

, An efficient hand gesture recognition system based on deep CNN”, In Proceedings of the 2019 IEEE International Conference on Industrial Technology (ICIT), Melbourne, Australia 13–15 (2019), 853–858.

Islam

M.Z.

, Hossain

M.S.

, ul Islam

and Andersso

, Static Hand Gesture Recognition using Convolutional Neural Network with Data Augmentation, 2019 Joint 8th International Conference on Informatics, Electronics & Vision (ICIEV) and 2019 3rd International Conference on Imaging, Vision & Pattern Recognition (icIVPR), Spokane, WA, USA, (2019), 324–329.

Nuzzi

, Pasinetti

, Lancini

, Docchio

and Sansoni

, Deep Learning-Based Hand Gesture Recognition for Collaborative Robots, IEEE Instrumentation & Measurement Magazine 22(2) (2019), 44–51.

Rubin Bose

and Sathiesh Kumar

, Hand Gesture Recognition Using Faster R-CNN Inception V2 Model”, AIR 2019: Proceedings of the Advances in Robotics 2019, ACM digital library (2019), Article No.:19, 1–6.

Rubin Bose

and Sathiesh Kumar

, Efficient Inception V2 based Deep Convolutional Neural Network for Real-Time Hand Action Recognition, IET Image Processing (2020), DOI: 10.1049/iet-ipr.2019.0985.

10.

Sreekar

, Sindhu

V.S.

, Bhuvaneshwaran

, Rubin Bose

and Sathiesh Kumar

, Positioning the 5-DOF Robotic Arm using Single Stage Deep CNN Model, 2021 Seventh International conference on Bio Signals, Images, and Instrumentation (ICBSII), IEEE Explorer, (2021), 1–6, doi:10.1109/ICBSII51839.2021.9445124.

11.

Rubin Bose

and Sathiesh Kumar

, Precise Recognition of Vision-Based Multi-hand Signs Using Deep Single Stage Convolutional Neural Network. In: Singh S.K., Roy P., Raman B., Nagabhushan P. (eds), Computer Vision, and Image Processing. CVIP 2020. Communications in Computer and Information Science 1377 (2021).

12.

Gao

, Liu

, Ju

and Zhang

, Dual-hand detection for human–robot interaction by a parallel network based on hand detection and body pose estimation, IEEE Trans Ind Electron 66(12) (2019), 9663–9672.

13.

Chaman

, D’souza

, D’mello

, Bhavsar

and D’souza

, Real-Time Hand Gesture Communication System in Hindi for Speech and Hearing Impaired, Second International Conference on Intelligent Computing and Control Systems (ICICCS) Madurai, India, (2018), 1954–1958. doi:10.1109/ICCONS.2018.8663015.

14.

Kopuklu

, Gunduz

, Kose

and Rigoll

, Real-time hand gesture detection and classification using convolutional neural networks, arXiv preprint arXiv:1901.10323, (2019), https://arxiv.org/abs/1901.10323.

15.

Avola

, Bernardi

, Cinque

, Foresti

G.L.

and Massaroni

, Exploiting Recurrent Neural Networks and Leap Motion Controller for the Recognition of Sign Language and Semaphoric Hand Gestures, IEEE Transactions on Multimedia 21(1) (2019), 234–245.

16.

Neethu

P.S.

, Suguna

and Sathish

, An efficient method for human hand gesture detection and recognition using deep learning convolutional neural networks, Soft Computing 24 (2020), 15239–15248.

17.

Islam

M.M.

, Islam

M.R.

and Islam

M.S.

, An efficient human-computer interaction through hand gesture using deep convolutional neural network, SN Comput Sci Jun 1(4) (2020), 223.

18.

, Qin

, Sun

, Yan

, Jiang

and Zhang

, Feature Fusion Based Hand Gesture Recognition Method for Automotive Interfaces, Chinese Journal of Electronics-IET 29(6) (2020), 1153–1164.

19.

Tan

Y.S.

, Lim

K.M.

and Tee

, Convolutional neural network with spatial pyramid pooling for hand gesture recognition, Neural Comput & Applic (2020). https://doi.org/10.1007/s00521-020-05337-0.

20.

Mohammed

A.A.Q.

, Lv

and Islam

MD.S.

, A Deep Learning-Based End-to-End CompositeSystem for Hand Detection and Gesture Recognition, Sensors 19 (2019), 5282. doi: 10.3390/s19235282.

21.

Lin

T.Y.

, Goyal

, Girshick

, He

and Dollar

, Focal Loss for Dense Object Detection, IEEE International Conference on Computer Vision (ICCV) (2017), 2980–2988.

22.

Pisharady

P.K.

, Vadakkepat

and Loh

A.P.

, Attention based detection and recognition of hand postures against complex backgrounds, International Journal of Computer Vision 101(3) (2013), 403–419. https://doi.org/10.1007/s11263-012-0560-5.

23.

Kumar

, Vadakkepat

and Poh

L.A.

, The NUS hand posture datasets II. ScholarBank@NUS Repository. [Dataset]. (2017-11-03) https://doi.org/10.25540/AWJS-GMJB.

24.

Memo

, Minto

and Zanuttigh

, Exploiting Silhouette Descriptors and Synthetic Data for Hand Gesture Recognition, STAG: Smart Tools & Apps for Graphics (2015).

25.

Memo

and Zanuttigh

, Head-mounted gesture controlled interface for human-computer interaction, Multimedia Tools and Applications (2017).

26.

Redmon

and Farhadi

, Yolo9000: better, faster, stronger, arXiv preprint 1612 (2016).

27.

Simonyan

and Zisserman

, Very deep convolutional networks for large-scale image recognition, arXiv preprint arXiv:1409.1556, (2014).

28.

Huang

, Liu

and Weinberger

K.Q.

, Densely connected convolutional networks, CoRR, abs/1608.06993, (2016). https://arxiv.org/pdf/1608.06993v3.

29.

Howard

A.G.

, Zhu

, Chen

, Kalenichenko

, Wang

, Weyand

, Andreetto

M.M.

and Adam

, MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications, Google Inc., (2017).

30.

Lin

T.Y.

, Dollar

, GirshicMk

, He

, Hariharan

and Belongie

, Feature pyramid networks for object detection, In CVPR, (2017).

31.

Ruder

, An overview of gradient descent optimization algorithms, arXiv preprint arXiv:1609.04747, (2017). https://arxiv.org/abs/1609.04747.

32.

Qian

, On the Momentum term in gradient descent learning algorithms, Neural Networks 12(1) (1999), 145–151.

33.

Kingma

and Ba

J.A.

, A method for stochastic optimization, arXiv preprint arXiv:1412.6980, (2014). https://arxiv.org/abs/1412.6980.

34.

Tieleman

and Geoffrey

, Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude, Coursera: Neural Networks for Machine Learning 4(2) (2012).

35.

Kadish

, Risi

and Løvlie

A.S.

, Improving Object Detection in Art Images Using Only Style Transfer, arXiv:2102.06529v1 [cs.CV] 12 Feb 2021.

36.

Padilla

, Netto

S.L.

and da Silva

E.A.B.

, A Survey on Performance Metrics for Object-Detection Algorithms, 2020 International Conference on Systems, Signals and Image Processing (IWSSIP) (2020), 237–242. doi:10.1109/IWSSIP48289.2020.9145130.

37.

Adithya

and Rajesh

, A deep convolutional neural network approach for static hand gesture recognition, Procedia Comput Sci 171 (2020), 2353–2361.