A review of sign language recognition research

Abstract

Sign language is the primary way of communication between hard-of-hearing and hearing people. Sign language recognition helps promote the better integration of deaf and hard-of-hearing people into society. We reviewed 95 types of research on sign language recognition technology from 1993 to 2021, analyzing and comparing algorithms from three aspects of gesture, isolated word, and continuous sentence recognition, elaborating the evolution of sign language acquisition equipment and we summarized the datasets of sign language recognition research and evaluation criteria. Finally, the main technology trends are discussed, and future challenges are analyzed.

Keywords

Sign language recognition convolutional neural network encoder-decoder dataset

1 Introduction

Sign language is a visual gesture language of daily communication between hard-of-hearing and deaf people. It uses different body parts, such as fingers, hands, arms, head, body, and facial expressions, to convey information [1]. Every country and every region has its distinct sign language, which makes it impossible for people with hearing disabilities from different countries or regions to communicate.

China has the largest number of hard-of-hearing persons in the world. Moreover, the deaf and hard-of-hearing makes up an estimated population of 27.8 million in China alone, with one deaf person in every 100 people. Many people use spoken language to communicate in daily life. Almost no one understands and uses sign language. It becomes very difficult for communication between the hearing and the deaf community, with the result that deaf and hard-of-hearing people are isolated and find it difficult to integrate into society.

Sign language recognition involves research in various disciplines, such as spatial geometry, pattern recognition, probabilistic statistics, artificial intelligence, natural language processing, and image analysis and processing [1]. These subjects and technologies are applied in the research of sign language recognition. At the same time, the research results promote the development of the above fields. In sign language recognition, the results help improve the living, learning, and working environment of the deaf-mute. It can popularize standardized sign language better and enrich the teaching mode if applied in deaf-mute education. In addition, researchers have applied sign language recognition technology to other related fields, such as traffic police gesture recognition, intelligent home appliance control, film production, virtual games, etc. This research brings benefits to a wide range of social settings.

Sign language recognition has received more and more attention with computer technology development and intelligent human-machine interface research. This article summarizes and discusses the current status and development trend of sign language recognition research in recent years. The main contributions are as follows:

the current status of sign language recognition research in recent years, with reference to future research;

the datasets in the field of sign language recognition, so that researchers can easily access this information;

some innovative ideas for the research and development of sign language recognition in the future.

The article is organized as follows: Section 2 introduces sign language acquisition equipment. Section 3 summarizes the development of sign language recognition research technology from the three aspects of gestures, isolated words, and continuous sentences. Section 4 presents the sign language data sets and evaluation criteria in China and abroad. Section 5 gives the conclusions and future work.

2 Data acquisition equipment

The process of sign language recognition includes a number of general categories, namely, data acquisition, pre-processing, segmentation, feature extraction, and classification. Data acquisition is the primary and most important step. Its quality is related to the whole effect of sign language recognition. Throughout the history of sign language recognition research, data acquisition equipment can be divided into three categories: data sensors, Kinect sensors, and cameras.

Sensor-based recognition utilizes a data glove and position tracker to measure the angular information of each joint of the hand. It uses trajectory and spatiotemporal information of hand movements for sign language recognition. Firstly, the data glove feeds back data from each joint and then returns the hand’s 3D coordinates through a position tracker to measure the gesture’s position information in 3D space. The research on sensor-based sign language recognition started in 1982, and in 1983, Grime [2] of the American Telephone and Telegraph Company obtained the patent for the Digital Data Entry Glove Interface Device. He is considered the first person to research sign language recognition, This device can recognize 72 one-handed letters. Fels et al. [3] used VPL data gloves and Phloem’s position tracker as an input device to build a sign language recognition system. The system uses five neural network opponents’ movement trajectory, movement direction, offset, movement speed, and hand shape to extract features for recognition. Yu et al. [4] proposed a Chinese sign language recognition method based on Deep Belief Net, using two 6-D inertial sensors and eight sEMG sensors as input devices to collect datasets. The sensors were placed on both sides of the participants. On each side, one 6-D inertial sensor was placed on the back of the forearm, and four sEMG sensors were placed on the four target forearm muscles, which are the little finger extensor, palmaris longus, extensor carpi ulnaris, and extensor carpi radialis. The data collection was conducted in five different sessions, and a total of 3750 CSL words were collected. Literature [5] uses five flex sensors, an Inertial Measurement Unit (IMU), and three Force Sensing Resistors (FSR) to detect the degree of finger bending and hand movement information. The detected data are transmitted to the computer by an Arduino Micro. The data glove can directly obtain the coordinates of a human hand in the 3D space, the degree of bending of the fingers, and the direction and speed of movement. The disadvantage is that it is very inconvenient to wear and use and thus, it is still in the laboratory research stage.

Due to the high price of data gloves, researchers have been looking for more economical equipment. Microsoft launched a somatosensory device called Microsoft Kinect on June 14, 2010, primarily for video game players. However, researchers have widely used it as research equipment due to its ability to perceive the sound, gesture, and skeletal movement information and provide color and depth video streams. In the work using Kinect for sign language recognition [6], Chai et al. [7] placed the device in front of the body to obtain RGB and depth information and then fed it into the neural network for recognition. Feng et al. [8] conducted a palm segmentation study using the color and depth information collected by the Kinect sensor. Bencherif et al. [9] developed a novel Arabic Sign Language (ArSL) recognition system, which uses two Kinect cameras to collect sign language videos from different angles and input the hand and body key points selected from the sign language video frame into the parallel 1D-CNN for recognition. Although the Kinect device can provide more information, it is not convenient to carry. Microsoft decided to discontinue the Kinect device in 2014, so it has gradually faded from the field of sign language recognition research.

Recent research demonstrated that the addition of depth images does not bring about improvement in sign language recognition. In comparison, the RGB information of images is greater. Therefore, scholars gradually abandoned the device and instead use cameras that are simple and can easily obtain RGB information for image input. Sign language recognition input data is in the form of gesture images, which can be easily captured by the camera. Some researchers use more advanced active technologies such as standard cameras, webcams, stereo cameras, thermal imagers, and other image acquisition equipment to avoid problems when using data gloves and somatosensory equipment. For this reason, scholars have gradually turned to the camera or camcorders that can easily obtain RGB images. References [10] use the camera to capture the image and then perform the image analysis and gesture recognition. It’s easy and fast to capture information with a single camera, but only from one viewpoint, thus many researchers use multiple cameras to capture multi-directional information. In 2007, Ten et al. [11] collected 121 Dutch sign language words with cameras. Each word extracted 6-dimensional features and presented an algorithm for Dynamic Time Warping (DTW) on multi-dimensional time series (MD-DTW). Similarly, Kishore et al. [12] used four cameras to obtain multi-azimuth image information from the center to the positions of –20°, –10°, 10°, and 20°, respectively. In 2021, researchers [13] used a high-definition web camera to collect gestures of different positions, angles, and directions that volunteers made freehand in front of the image capture camera. And Park et al. [14] proposed a system based on depth video for translating sign language into text using the front camera of a smartphone. Most sign language recognition research is camera-based because of the low cost of acquiring data, clear images, and portability.

The analysis table of data acquisition equipment is shown in Table 1, summarizing the changes of sign language data collection equipment in the past 20 years. It can be seen that although it is very inconvenient to use data gloves and Kinect equipment to collect sign language, there are still researchers who continue to use them. This is because the data glove and the Kinect device can sample multi-modal information, and the complementarity between multi-modal information in sign language recognition has a certain effect on the recognition of sign language. However, from the perspective of the development and application of sign language recognition in the future, cameras are more suitable and convenient for communication and the use of deaf people in daily social life.

Table 1
Data acquisition equipment analysis table

Research Year Equipment Image information

Joint RGB Depth

[2] 1983 Data Glove √

[3] 1998 Data Glove √

[4] 2019 Data Sensors √

[5] 2021 Data Sensors √

[7] 2017 Kinect √

[8] 2018 Kinect √ √

[9] 2021 Kinect √ √

[11] 2007 Camera √

[12] 2015 Camera √

[10] 2016 Camera √

[13] 2021 Camera √

[14] 2021 Camera √

Research	Year	Equipment	Image information
[2]	1983	Data Glove	√
[3]	1998	Data Glove	√
[4]	2019	Data Sensors	√
[5]	2021	Data Sensors	√
[7]	2017	Kinect		√
[8]	2018	Kinect		√	√
[9]	2021	Kinect		√	√
[11]	2007	Camera		√
[12]	2015	Camera		√
[10]	2016	Camera		√
[13]	2021	Camera		√
[14]	2021	Camera		√

3 Sign language recognition technology

Sign language can be divided into gesture recognition, isolated word recognition, and continuous sentence recognition. The following explains these three aspects.

3.1 Gesture recognition

The object of gesture recognition is the shape of the hand in a single picture. Compared with isolated words, the semantics of gestures are simple and clear. Recognition mainly focuses on accurately recognizing the meaning of gestures and improving the recognition rate. Different gesture models determine the diversity of gesture recognition methods. The research methods mainly include three main frameworks: Dynamic Time Warping (DTW), Hidden Markov Model (HMM), and Convolutional Neural Networks (CNN) based methods. The specific classification is shown in Fig. 1.

Fig. 1

Gesture recognition classification.

3.1.1 Dynamic time warping

Dynamic Time Warping (DTW) algorithm is an early and classical algorithm based on dynamic programming and performs normalization and pattern recognition for non-linear time. It uses the non-linear warping function to eliminate the non-linear fluctuation in time. Thus, eliminating the difference between the spatiotemporal representation modes in different time axes.

Assume that a₁, a₂, …, a_m, …, a_M are the feature sequences in the reference template, b₁, b₂, …, b_n, …, b_N, M ≠ N are the feature vectors of the input module, the DTW algorithm finds a non-linear mapping relationship, called the time normalization function: $\begin{matrix} m = w (n) \end{matrix}$ (1)

This function can clearly express the mapping of the time axis (m and n) between the reference sample and the input sample, and meet the following conditions: $\begin{matrix} D = \min \sum d [n, w (n)] \end{matrix}$ (2)

Where d [n, w (n)] is the distance between the feature vector of the n-th frame of the input sequence and the feature vector of the mth frame of the reference sequence, D is the total distance between the input sample and the reference sample in the optimal time normalization state.

In gesture research Konecny et al. [15] added a histogram of oriented gradients (HOG) of hand and a histogram of optical flow (HOF) descriptor in the DTW to improve the accuracy of gesture recognition. Since DTW needs to perform matching calculations on the paths and all nodes, the amount of calculation is too large. Huang et al. [16] proposed a Multi-dimensional input dynamic time warping (MD-DTW) method for gesture recognition. It imposes boundary constraints on globally planned paths and reduces the number of computations required for template matching, saving unnecessary regularization time, and improving the algorithm’s robustness. Because it has bounded the regular path, it will cause a lack of features and bring the wrong prediction when matching. To a more comprehensive expression of the feature of gestures, Zhou et al. [17] proposed an algorithm based on Global Template Dynamic Time Warping (GTDTW). The global template obtained by this method can fully express the characteristics of gestures. Simultaneously, it reduces time consumption and ensures real-time gesture recognition. The global template length is also smaller than that of the conventional template of dynamic time warping.

3.1.2 Hidden markov model

The DTW algorithm improves the rate of gesture recognition, but it does not perform well in recognizing large amounts of data, complex gestures, and combined gestures. Therefore, the Hidden Markov Model (HMM)—a method based on pattern recognition—came into being. HMM is a commonly used state-space-based method and a mature mathematical method for matching time-varying data. Generally, HMM is expressed as a triplet μ = (A, B, π), where π is the probability distribution of the initial state, A is the state transition probability, and B is the symbol emission probability. The state sequence in HMM is invisible, so the expectation-maximization method (EM) is used, which can be used to estimate the maximum likelihood of the parameters of a statistical model with hidden variables. The calculation steps are as follows:

E-step: calculate the expected ɛ_t (i, j) and γ_t (i) from the model μ_i according to Eqs. (3) and (4): $\begin{matrix} ɛ_{t} (i, j) {= P (q}_{t} {= s}_{i} {, q}_{t + 1} {= s}_{j}, O | μ) \end{matrix}$ (3) $\begin{matrix} γ_{t} (i) = \sum_{j = 1}^{N} ɛ_{t} (i, j) \end{matrix}$ (4)

M-step: Using the expected value obtained in E-step, re-estimate the values of the parameters π_i, a_ij, b_j (k) according to Eqs. (5), (6), and (7) to obtain μ_i+1: $\begin{matrix} π_{i}^{'} = P (q_{1} {= s}_{i} | O, μ) {= γ}_{1} (i) \end{matrix}$ (5) $\begin{matrix} a_{ij}^{'} = \frac{\sum_{t = 1}^{T - 1} ɛ_{t} (i, j)}{\sum_{t = 1}^{T - 1} γ_{t} (i)} \end{matrix}$ (6) $\begin{matrix} b_{j}^{'} (k) = \frac{\sum_{t = 1}^{T} γ_{t} (j) {* δ (O}_{t} {, v}_{k})}{\sum_{t = 1}^{T} γ_{t} (j)} \end{matrix}$ (7)

Calculate in a loop, let i = i + 1. Repeat the EM calculation until π_i, a_ij, b_j (k) converge.

HMM was mainly used in the field of speech recognition in the past. In gesture recognition, it has developed rapidly in recent years, and many representative research methods have emerged. Yan et al. [18] adopted the SWAB algorithm for automatic endpoint detection, HMM for gesture instructions modeling, and adopted the K-means algorithm for vector quantization of gesture feature sequence. Although non-verbal communication mainly uses hand and finger movements, it is not limited to these. In [19], the author extends the gesture-based framework to full-body posture and selects an innovative feature descriptor based on active differential features to pass the HMM model on behalf of traditional geometric features to obtain a satisfactory recognition rate. With in-depth research, people found that the basic HMM model has been unable to learn more features, and its generalization ability is limited. Yang et al. [20] proposed a frag-HMM method, which obtains the similarity of gestures from a typical and independent video stream without accurate segmentation. With the rapid development of machine learning, scholars began to try to combine traditional HMM with machine learning models to obtain greater accuracy. In [21] Koller et al. combined the CNN classifier trained by GoogLeNet’s Inception-VL model with HMM. In another work, [22] proposed a combination of HMM and FNN for dynamic gesture recognition. It deconstructed the gesture image into three feature sequences: handshape change, hand position change on the two-dimensional plane, and movement in the Z-axis direction, And then an HMM model was established for each of them, and fuzzy inference is used to connect the FNN to determine the semantics of the gesture. The model can quickly and effectively recognize complex dynamic gestures. In addition, Deep Belief Networks are applied in gesture recognition. Chen et al. [23], presented a combined HMM with Deep Belief Networks. Meanwhile, the embedded time-level dictionary into the hidden Markov model introduced relative entropy to measure the dictionary’s information richness. It verified the validity of the proposed method on two benchmark gesture data sets.

3.1.3 Convolutional neural networks

HMM needs to pass a large number of training samples to obtain statistical characteristics. With more training samples, the performance of HMM is better, but the amount of calculation is huge, which affects the real-time performance of the system. With the rapid development of deep learning, neural networks have shown outstanding learning capabilities in various fields. Convolutional Neural Networks (CNN) are usually composed of a convolutional layer, a pooling layer, a normalization layer and a fully connected layer. The network structure is expressed as: Enter -> [Conv ->ReLU]*N -> [Pooling]*M -> [FC ->ReLU]*K ->FC, the formula is as follows: $\begin{matrix} F = FC (Pooling ((Conv {x_{t}}_{t = 1}^{T}) * N) * M) * K \end{matrix}$ (8)

Where Conv is the convolutional layer, ReLU is the normalization layer (due to space issues, ReLU is omitted in formula (8)), Pooling is the pooling layer, FC is the fully connected layer, and * is the number of repetitions. N> =0, M> =0, K> =0, usually K < 3. There are different settings for different research methods with different specific values.

After neural networks were introduced into the field of gesture recognition, methods based on CNN showed the most advanced performance in human gesture/action recognition. Kang et al. [24] used CNN to extract features for deep image sign language recognition. Karpathy et al. [25] extended CNN connectivity in the temporal domain and designed a multi-resolution centralized architecture using local spatiotemporal information to expedite the training. Liang et al. [26] projected the point cloud of the hand to different view planes, used CNN to extract the features in the view, and SVM for gesture recognition training. The experiment proved that the method has higher robustness but a poor generalization. Although CNN has strong feature extraction capabilities, it is limited to extracting time information and cannot extract spatial information. Compared with CNN, 3D-CNN further applies the convolution kernel to the space field. The spatiotemporal characteristics of the convolution kernel can solve the disadvantages of CNN. The network based on C3D [27] was successfully used in the 2016 ChaLearn LAP Large-scale Isolated Gesture Recognition Challenge [28 –30], showing the most advanced performance, and since then it has been widely used by scholars in gesture research. In [31] the authors used the 3D-CNN model to learn depth and intensity data separately and combined the information of multiple spatial scales for gesture prediction. Similarly, Li et al. [29] proposed 3D-CNN-based network architecture, introduced RGB image data and depth image data, and achieved good results. [32] proposed a ResC3D network model based on multi-modal data for gesture recognition. Molchanov et al. [33] combined 3D-CNN and RNN to detect and classify multi-modal data for dynamic gestures, which enhanced the generalization of the network model. Zhu et al. [34] used 3D-CNN and convolutional LSTM networks to learn Spatio-temporal features end-to-end and used multi-modal data for fine-tuning to further optimize the network model. Researchers in [35] used two streams of 3D-CNN for learning the fine-grained features of the hand shape and the coarse-grained features of the global body configuration. The experimentally tested results show that the proposed system outperforms state-of-the-art approaches, demonstrating its effectiveness. Yu et al. [36] utilize shallow two- stream CNNs to capture the original video frame’s low-level features and corresponding optical flow. In addition, they utilized an attentive feature fusion module to selectively combine useful information from the previous two streams based on the attention mechanism. This method achieves an accuracy of 95.77% on the Jester dataset.

Table 2 shows that the main technical framework lists gesture recognition technology and representative work. As for the representative work using the same framework, there is no comparison between them due to different data sets. Therefore, we list the representative work using the same framework in chronological order. At present, gesture evaluation is based on accuracy.

Table 2

Gesture recognition analysis table

Research	Year	Technical Classification	Dataset	Acc
[16]	2017	DTW	Custom(Six gestures)	98.00%
[17]	2018	DTW	Custom	98.30%
[18]	2012	HMM	Custom(Eight gestures)	97.25%
[19]	2017	HMM	Custom	94.70%
[20]	2006	HMM	Custom	93.80%
[21]	2016	HMM	SIGNUM	92.4%
[22]	2017	HMM+FNN	ASL	91.00%
[23]	2018	HMM+DBN	Chalearn IsoGD	82.00%
[25]	2014	CNN	Sports-1M dataset	82.40%
[24]	2015	CNN	Custom	85.49%
[26]	2016	CNN	Custom(Ten gestures)	99.88%
[31]	2015	3D-CNN	VIVA	77.50%
[29]	2016	3D-CNN	Chalearn IsoGD	49.20%
[32]	2017	3D-CNN+ResNet	Chalearn IsoGD	67.71%
[33]	2016	3D-CNN+RNN	Montalbano / SKIG	98.20% / 98.6%
[34]	2017	3D-CNN+LSTM	Chalearn IsoGD / SKIG	51.02% / 98.89%
[35]	2020	Two 3D-CNN	Custom(Forty gestures)	98.75%
[36]	2020	CNN+Attention	Jester	95.77%

3.2 Isolated word recognition

Unlike gestures, isolated words are sequences of time and space which are more structured gestures with precise meanings in sign language books. The traditional sign language recognition method has inadequate learning ability and insufficient adaptability. In contrast, the deep-learning sign language recognition method for isolated words has strong fault tolerance and adaptability. Therefore, this section will be described in detail based on the framework of convolutional neural networks (CNN) and Recurrent Neural Networks (RNN). The specific classification is shown in Fig. 2.

Fig. 2

Isolated word recognition classification.

3.2.1 Convolutional neural networks (CNN)

CNN is a locally activated feedforward neural network, which has three basic characteristics: perceptual field, weight sharing, and downsampling. And it has been widely used in speech recognition, license plate recognition, face recognition, and other fields. Its high-efficiency recognition accuracy and speed can also promote sign language recognition for isolated words. Research [37] and [38] both use CNN to obtain image features: the former uses multi-scale but all levels of image features, while the latter focuses more on hand changes. Besides hand features, information such as motion features and facial features can play a more important role in isolated word recognition. Kopuklu et al. [39] proposed a data-level fusion strategy for fusing motion information into static images at the CVPR2018 conference and sent the fused spatiotemporal features to the CNN network for subsequent classification and achieved good recognition results.

CNN has achieved great results in isolated sign language recognition, but it focuses only on each frame’s features without considering the inter-frame motion information. Scholars have introduced the 3D-CNN network to solve this problem, capturing the video’s temporal and spatial feature information. The breakthroughs of sign language recognition based on 3D-CNN are different fusion models and spatiotemporal attention mechanisms. Huang et al. [40] proposed a novel 3D-CNN model for isolated word recognition. To boost performance, multi-channel video streams, including color information, depth clue, and the body’s joint positions, are used as input to the 3D-CNN to integrate color, depth, and trajectory information. In a similar work, Liang et al. [41] proposed a 3D-CNN network based on multi-modal data input and convolution fusion for various data, which verified its effectiveness on large-scale data sets. However, its generalization is not high on sign language data sets composed mostly of RGB data. In another work, Wu et al. [42] proposed a 3D-CNN based on multi-channel data fusion for isolated word recognition. By incorporating RGB and depth information stacking into the 3D-CNN, Wu introduced the HMM model for short-time series modeling, verifying this method’s effectiveness on the ChaLearn data set. Adding the spatiotemporal attention mechanism into 3D-CNN can capture the most critical sign language movements in the sign language video and obtain superior recognition accuracy.

In 2018, Huang et al. [43] proposed an attention-based 3D-CNN model, which benefits from the learning ability of spatiotemporal characteristics of 3D-CNN, and simulates human visual mechanism processing by finding and paying attention to regions of interest. The results show that this method’s recognition rate is higher than that based on C3D in [40], which indicates the effectiveness of spatial and temporal attention mechanisms. Paper [44] further explored the ability of cascaded 3D-CNN to capture sign language information. With the continuous evolution of 3D-CNN and the proposal of C3D-Ret, sign language recognition has been further improved. For ordinary networks, as the depth of the network increases, gradient diffusion or gradient explosion will occur, but the residual network uses its internal residual block to make jump connections, which alleviates the disappearance of gradients caused by increasing depth in the deep neural network. Miao et al. [45] proposed a multi-modal gesture recognition method based on the ResC3D network in ICCV2017. This method’s key idea is to find a compact and effective video sequence representation and propose a feature fusion method based on regular correlation analysis for feature extraction, which won first place on the ChaLearn dataset. Liao et al. [46] combined ResC3D and bidirectional LSTM networks for sign language recognition. The method used Faster R-CNN to locate and identifies key hand part points to obtain the most advanced identification accuracy. Since only the characteristic information of the hand is concerned, it neglects the other useful information. Hand joints and whole-body skeleton data play an auxiliary role in the extraction of sign language video features. Razieh et al. [47] use the estimated 3D key points and midpoints of the RGB image to construct a hand skeleton and obtain different representations from key points of the hand. Then the skeleton was projected onto three surfaces and converted into an image format, and finally fed to 3DCNN to obtain different local spatiotemporal complementary feature representations. Jiang et al. [48] proposed a perceptual multi-modal sign language recognition framework that extracts whole-body skeleton data, optical flow data, and depth information from RGB sign language video sequences. The fusion of multi-modal information enables the framework to learn complementary global information. Meng et al. [49] extract skeleton information from RGB images, use graph convolutional networks (GCN) for multi-scale feature extraction and propose a keyframe extraction algorithm that improves efficiency at the expense of some accuracy.

3.2.2 Recurrent neural network

The above CNN structure has achieved remarkable results, but CNN can only extract the short-term spatiotemporal features due to its constraints. To improve isolated word recognition accuracy, researchers began to use RNN to extract long-term spatiotemporal features. The RNN model calculation is divided into two steps. The first step is to calculate the hidden layer h_t at the t-th time step; the second step is to calculate the predicted value y_t at the t-step: $\begin{matrix} h_{t} {= σ (W}_{x} x_{t} {+ W}_{h} h_{t - 1} + b) \end{matrix}$ (9) $\begin{matrix} y_{t} {= σ (W}_{y} h_{t} + b) \end{matrix}$ (10)

Where, W_xW_handW_y are weights, and b is the bias value. ${{y}_{t}}_{t = 1}^{T}$ is the output of RNN.

Research based on RNN has: Chai et al. [50] proposed a two-stream RNN network for isolated word recognition. One input of the network is RGB data. Another network input is the feature fusion of extracted skeleton data and a gradient histogram. Finally they used the two-stream network fusion for the final score. This method ranked first in the ChaLearn Gesture Recognition Challenge. In isolated word recognition, preserving spatially related information will obtain more meaningful spatiotemporal features. Therefore, many kinds of research have emerged that combine CNN networks with RNN networks. Ye et al. [51] proposed a 3D-RNN network model by combining 3D-CNN and RNN to capture sign language recognition’s temporal and spatial information. 3D-CNN learned multi-modal features from RGB, motion, and depth channels. FC-RNN captured the temporal information between the divided short video segments from the original video. The 3D-RCNN can recognize and locate the semantic information of different video lengths, improving recognition accuracy. In another work, to solve redundant information that affects the recognition accuracy, Huang et al. [52] proposed a sequence-to-sequence learning method based on keyframes for a sign language recognition algorithm. The multi-modal data stream’s keyframes were embedded in CNN, DMM (in essence, RNN), and the network used to extract trajectory information, allowing different attention to the input data. The experiment has obtained a remarkable effect on the dataset of 310 Chinese sign language words. Moreover, in another work, Lin et al. [53] combined the Res-C3D network with mask and LSTM network for skeleton data modeling and they combined segmentation algorithm and classification network to study isolated words. This method improves the recognition accuracy, but the early masking process is time-consuming. Ameur et al. [54] proposed a Hybrid Bidirectional Unidirectional LSTM (HBU-LSTM) framework that combined the LSTM network and BLSTM network. The model is capable of efficiently classifying the input data. The experiment obtained a remarkable result on the LeapGestureDB dataset. Santos et al. [55] proposed an approach called star RGB representation, which can describe and condense a video clip containing a dynamic gesture on one RGB image, then used ResNet convolutional neural network for feature learning. The method achieves an accuracy of 98% in the GRIT dataset, which proves its effectiveness. Literature [56] developed an SLR method based on a boundary adaptive encoder combined with window attention. The key idea is to use a boundary detection unit (BDU) to learn the time boundary of the input symbol sequence and embed the boundary detection unit BDU into BLSTM; the length layer of the coding block can be adaptively controlled according to the input information and the hidden state of the coding. In 2021, Rodri et al. [57] proposed a Common Spatial Patterns (CSP) algorithm for feature extraction, using multiple classifiers for classification, reaching 97.95% in Argentinian Sign Language. The representative work analysis table of isolated word recognition technology is shown in Table 3.

Table 3

Isolated word recognition analysis table

Research	Year	Technical Classification	Dataset	Acc
[37]	2014	CNN	Montalbano	78.90%
[38]	2016	CNN	RWTH-PHOENIX-Weather	WER: 46.60% / 45.10%
[39]	2018	CNN	ChaLearn IsoGD / NVIDIA	57.40% / 84.70%
[40]	2015	3D-CNN	Custom	94.20%
[42]	2016	3D-CNN	ChaLearn IsoGD	81.00%
[41]	2018	3D-CNN	ChaLearn IsoGD	83.60%
[43]	2019	3D-CNN	Montalbano	95.30%
[44]	2021	3D-CNN	Boston ASL	96.30%
[45]	2017	ResC3D	Chalearn IsoGD	67.71%
[46]	2019	ResC3D+BLSTM	DEVISIGN_D / SLR_Dataset	89.80% / 86.90%
[47]	2020	3D-CNN+Heatmap	RKS-PERSIANSIGN	95.36%
[48]	2021	3D-CNN+GCN+TCN	AUTSL	98.53%
[49]	2021	GCN+TCN	CSL-500	98.08%
[50]	2016	RNN	Chalearn IsoGD	28.69%
[51]	2018	3D-CNN+RNN	ASL	69.20%
[52]	2018	3D-CNN+RNN	Custom	91.18%
[53]	2018	ResC3D+LSTM	Chalearn IsoGD	68.42%
[54]	2020	LSTM+BLSTM	LeapGestureDB	94.00%
[55]	2020	ResNet+LSTM	GRIT	98.00%
[56]	2021	BDU+BLSTM	CSL-500	81.60%
[57]	2021	PCA+BLSTM	LSA	97.95%

Table 3 shows that the main technical framework lists isolated word recognition technology and representative work. As for the representative work using the same framework, there is no comparison between them due to different data sets. Therefore, we list the representative work using the same framework in chronological order. As can be seen from Table 3, 3D-CNN and RNN network architectures have been more accurate in isolated word recognition research in recent years. Combining these two technologies retains the spatial features extracted by CNN, learns long-term temporal information through the RNN network, and finally obtains more meaningful spatio-temporal features, which greatly improves the recognition performance of gestures and isolated words. At present, both gesture evaluation and isolated word evaluation are based on accuracy.

3.3 Continuous sign language recognition

Unlike isolated word sign language, continuous sign language recognition includes transitional frames (Transition frame refers to a frame that is used to connect adjacent sign language words but has no actual meaning). The purpose of CSLR is to identify annotation in a sign language sequence. It is different from isolated word recognition, in which each symbol is independently segmented and annotated. It is also different from sign language translation (SLT) [58]. Sign language translation involves an extra step, that is, translating recognized annotation into a grammatical sentence. For the trained model, the core problem of sign language recognition, speech recognition, and machine translation is in transforming (input) sequences from one domain into (output) sequences in another. They can be abstractly transformed into variable-length sequence learning problems, however, due to the unfixed sequence length and no correspondence between input and output sequence length, traditional neural network models (DNN, CNN, and RNN) cannot directly solve such problems of modeling and learning end-to-end manner. RNN can solve the modeling problem of long-time series well. The common methods are LSTM, BLSTM, GRU, etc.

The main sign language recognition methods include Hidden Markov Model(HMM) recognition, Connectionist Temporal Classification(CTC) recognition, Seq2Seq (encoder/decoder) recognition, and other recent research. The specific classification is shown in Fig. 3.

Fig. 3

Continuous sign language recognition classification.

3.3.1 Hidden markov model

HMM is also widely used in continuous sign language recognition. In related work, Koller et al. of Aachen University of Technology, conducted continuous sign language recognition research based on German sign language data set from 2009 to 2012. In 2016 [59], they proposed a continuous sign language hybrid model based on CNN and HMM. The model combines deep learning with traditional methods and has achieved good recognition results on two publicly available large-scale benchmark sign language data sets. It is a milestone study in continuous sign language recognition research. In 2016, Koller et al. [60] embedded BLSTM based on the original framework and proposed an iterative realignment algorithm that embeds the CNN-BLSTM network into HMM. The algorithm used multiple realignments to improve the recognition performance of the model. Since the HMM method is a mathematical method based on matching time-varying data, it only achieves better results in the case of large samples. However, as the samples increase, the amount of calculation increases geometrically, which seriously affects real-time performance. It has been gradually abandoned in continuous sign language recognition in recent years.

3.3.2 Connectionist Temporal Classification(CTC)

CTC is a way to avoid input and output, allowing RNN to directly learn the sequence data without marking the mapping relationship between the input sequence and the output sequence in advance training data. The training process of the CTC-based network framework is usually shown in Fig. 4.

Fig. 4

CTC-based network framework.

The CTC loss calculation process is: Let ${Π = {π}_{n}}_{n = 1}^{N}$ be a CTC alignment, where π_n ∈ Ω ∪ {*}, Ω is the sign language vocabulary and {*} is a blank label. Let $k_{t} {= {k}_{n}^{c}}_{c = 1}^{C}$ , where $k_{n}^{c}$ is the probability of label c in vector k_t. The probability of π is represented by the product of all probabilities, as shown in Eq. (11): $\begin{matrix} Q_{π} = \prod_{n = 1}^{N} K (π_{n}) = \prod_{n = 1}^{N} K_{n}^{π_{n}} \end{matrix}$ (11) CTC loss is: $\begin{matrix} L_{ctc} = - \log \sum_{{π = B}^{- 1} (y)} Q (π) \end{matrix}$ (12)

Where B^-1 (y) is the set of all alignments.

Using a CTC training network can realize end-to-end continuous sign language recognition and research methods for this activity can generally be divided into using single CTC loss and using multiple CTC loss. In recent years, related work using single CTC losses has been Pu et al. [61] from the University of Science and Technology of China who proposed a continuous sign language recognition framework based on 3D-ResNet dilated convolutional network in 2018. They used an optimization strategy based on the CTC algorithm to fine-tune the feature extractor with pseudo-tags and verified its effectiveness and superiority over the German benchmark dataset. In 2018, Guo et al. [62] proposed a network structure (DenseTCN) based on 3D-CNN and stacked time convolution. They used CTC to learn feature classification and generate translated sentences. These studies are based on video frame single-mode data as input, which is simple and fast, but the information is thinner than multi-modal data, so a batch of studies using multi-modal data as input has emerged. Camgoz et al. [63] proposed a SubUNets framework at the ICCV2017 conference and used CTC to solve the alignment and recognition problems. This method combines the modal information of both the full-frame image and the hand image. When focusing on the full-frame image information, attention can be paid to key gesture information. In another work, Cui et al. [64] proposed a continuous sign language recognition system with a recurrent convolutional neural network for optical flow images and RGB frame multi-modal data. They used CTC’s alignment suggestions as weak supervision to fine-tune the feature extraction module and used iteration strategies to improve performance. They verify the algorithm on two publicly available benchmark datasets.

CTC loss can be calculated to adjust the network, but a single CTC will be prone to overfitting. To solve this problem, scholars use multiple losses to jointly adjust the network. In related work, Cui et al. [65] designed a three-stage optimization processed to train the network: CTC loss for end-to-end training, KL-div loss for feature learning, improved features for sequence learning, and demonstrated the effectiveness of this optimization strategy in a challenging benchmark. In another work, Wang et al. [66] propose a deep hybrid architecture that consists of a temporal convolution module (TCOV), a bidirectional gated recurrent unit module (BGRU), and a fusion layer module (FL) to address the CSLR problem. The proposed joint CTC loss optimization and deep classification score-based decoding fusion strategy to boost performance. This method creates a new benchmark in both the RWTH-PHOENIX-Weather dataset and the CSL dataset, but the network architecture was relatively complex and required high hardware requirements. Furthermore, in another similar work, [67] proposed to use a parallel time encoder (PTEnc) network to simultaneously learn the local and global time relationship of video. They designed a reconstruction loss to measure the distance between the original visual and reconstruction features. They combined it with the CTC loss to realize end-to-end optimization. Similarly, Yang et al. [68] proposed a structured feature network (SF-Net). The proposed model extracts feature in a structured manner and gradually encodes information at the frame level, the gloss level, and the sentence level into the feature representation. It used a KL-div loss based on gloss-level tags and a CTC loss based on sentence-level to jointly optimize the network. Test results on two large-scale public SLR datasets showed that the proposed SF-Net outperforms previous sequence-level supervision-based methods in terms of accuracy and adaptability. Aiming at the insufficient training of feature extractors due to the easy overfitting of CTC, Min et al. [69] proposed a visual alignment constraint (VAC) to enhance the feature extractor through more alignment supervision, where VAC includes two parts of the loss. One part is used to predict the visual features of opportunity, and the other is used to align short-term verse features and long-term context features. Its effectiveness has been verified on two public datasets. Gao et al. [70] proposed an alignment network from visual level to vocabulary sequence and introduced an RNN translator in the field of continuous sign language recognition for the first time to learn the best alignment between sign language video and sentence level tags.

3.3.3 Seq2Seq(Encoder-Decoder)

Since CTC assumes that the original sequence and the target sequence share the same order and assumes conditional independence within the target sequence, the network cannot learn the implicit language model. Therefore, the seq2seq (Encoder-Decoder) model structure that can simultaneously focus on the source sequence and the target sequence has also been extensively developed. The main idea of the encoder-decoder network is to map two sequences with intermediate latent space. In other words, it must encode the source sequence into a vector of fixed size and then decode the target sequence. However, there is a problem due to the source sequence’s encoding as a fixed size vector and the long-term dependence between the source sequence and the target sequence. To solve this problem, Bahdanau et al. [71] proposed using the attention mechanism to pass other information to the decoder so that it could find the most relevant original sequence information at every moment of decoding. Since then scholars have widely used attention mechanisms in encoder/decoder networks. The most influential work is that Camgoz et al. proposed a framework model combining CNN and attention-based encoder-decoder at the 2018 CVPR conference [58] (the framework diagram is shown in Fig. 5). It uses CNN as the spatial embedding layer and hybrid RNN+HMM as the tokenization layer and then uses the attention-based encoder-decoder network to train the network framework in an end-to-end manner. This work has achieved good results on public data sets. As a result, the code of the algorithm is published, which provides a reference for new researchers in this area and further promotes the development of continuous sign language recognition research. Guo et al. conducted a series of continuous sign language recognition research based on the seq2seq model. Guo et al. [72] proposed a continuous sign language recognition framework based on asymmetric multi-layer LSTM in 2018. The attention-based weighting mechanism proposed by them balances the semantic relationship in the feature learning process while focusing on redundant information. Aiming at the problem that redundant information affects the recognition accuracy, this research proposes a feature mining method and pooling strategy for key blocks with edge length, which effectively improves the learning efficiency of the model for sign language change patterns and the accuracy of sign language translation. In the same year, they proposed a hierarchical HLSTM codec model using a time-weighted attention-weighting mechanism at the AAAI conference [73], which handles sign language recognition of different granularities through spatio-temporal conversion between frames, clips, and viseme units. The algorithm not only retains the original video features but also obtains advanced features. It has achieved good recognition results in the Chinese Sign Language dataset, but the model is too complex, and the recognition speed of multi-layer HLSTM is slow. In order to make full use of the complementarity between short-term spatiotemporal features and long-term spatiotemporal features, Guo et al. [74] proposed a temporal convolutional pyramid module to learn short-term temporal correlation from 2D-CNN features and then complement the original 3D-CNN features, learning and embedding dynamic programming into the decoding scheme. To solve the problem of slow recognition speed, they proposed a hierarchical deep recursive fusion (HRF) framework in 2020 [75], which uses an encoder composed of adaptive clip summary (ACS) and LSTM to explore RGB view position and bone information, and uses LSTM and a query adaptive decoding fusion to translate the target sentence. Pu et al. [76] proposed a network that uses CTC loss and cross-entropy loss to jointly train an encoder-decoder under the constraint of soft Dynamic Time Warping (soft-DTW). The algorithm uses strong supervision to achieve better performance on the German sign language dataset and the Chinese sign language dataset. Huang et al. [77] proposed a latent space hierarchical attention network (LS-HAN) for recognizing continuous sign language. It uses LSTM’s extended hierarchical attention network (HAN) to eliminate the pre-processing of time segmentation and reduce the loss of information, but due to the excessive number of frames, gradient explosion is likely to occur during LSTM calculation, and the calculation speed is slow. In response to this problem, Cheng et al. [78] performed end-to-end sign language recognition based on a fully convolutional network and introduced Gloss Feature Enhancement (GFE) loss to perform additional corrections to monitor the effectiveness of the method in the German sign language dataset.

Fig. 5

NSLT Model [58].

In 2020, [79] proposed a novel encoder-decoder architecture based on the Transformer that jointly learns Continuous Sign Language Recognition and Translation while being trainable in an end-to-end manner. This method utilized CTC loss to inject gloss-level supervision into the Transformer encoder, training it to sign language recognition while learning meaningful sign language translation representations. They are the first to apply the transform structure in NLP to sign language recognition and achieve the most advanced PHOENIX14T dataset results. Subsequently, Necati et al. [80] introduced gestures, mouth shapes, and bone information for multi-modal interactive learning on this basis and used Transformer to simultaneously model intra-modal and inter-modal contextual relationships, which not only maintained various specific information of the modal, while learning the complementary information between the models.

Zhou et al. [81] proposed a self-attention (SAFI) network in 2020, introduced full inception with different receptive fields to extract dynamic segment-level features, and used Aggregated Cross-Entropy (ACE) loss and CTC loss combined. The global sequence feature learning is trained, and the proposed model is optimized. Recently, a research paper [82] introduced a new method for context-aware continuous sign language recognition using a generative adversarial network architecture. This method includes a generator that recognizes sign language vocabulary by extracting spatial and temporal features from video sequences and a discriminator that evaluates the quality of the generator’s predictions by modeling text information at the sentence and vocabulary level. By competing with each other, both the generator and the discriminator have been improved, resulting in more accurate and robust SLR results. Continuous sign language recognition technology and representative work are shown in Table 4.

Table 4

Continuous sign language recognition technology and representative work analysis

Research	Year	Classification	Dataset	Evaluation Criteria (WER%)
[59]	2016	HMM+CNN	PHOENIX-2014	Dev: 38.8% / Test: 38.8%
			SIGNUM	7.40%
[60]	2017	HMM+CNN+BLSTM	PHOENIX-2014	Dev: 27.1% / Test: 26.8%
			SIGNUM	7.80%
[61]	2018	CTC+3D-ResNet+Dilated Convolution	PHOENIX-2014	Dev: 38.0% / Test: 37.3%
[62]	2019	CTC+3D-CNN+DenseTCN	PHOENIX-2014	Dev: 35.9% / Test: 36.5%
			CSL	Split I: 14.3% / Split II: 44.7%
[63]	2017	CTC+CNN+BLSTM	PHOENIX-2014	Dev: 43.1% / Test: 42.1%
[64]	2019	CTC+CNN+BLSTM (Multimodal Data)	PHOENIX-2014	Dev: 43.1% / Test: 42.1%
			SIGNUM	2.80%
[65]	2017	CTC+CNN+TCN+BLSTM (Multi-loss)	PHOENIX-2014	Dev: 39.4% / Test: 38.7%
[66]	2018	CTC+TCOV+BGRU (Multi-loss)	PHOENIX-2014	Dev: 37.9% / Test: 37.8%
[67]	2019	CTC+CNN+BLSTM+LSTM (Multi-loss)	PHOENIX-2014	Dev: 38.1% / Test: 38.3%
[68]	2019	CTC+3D-CNN+LSTM+BLSTM (Multi-loss)	PHOENIX-2014	Dev: 35.6% / Test: 34.9%
[69]	2021	2D-CNN+1D-CNN+BLSTM	PHOENIX-2014	Dev: 22.1% / Test: 23.0%
			CSL	Split I: 1.6%
[70]	2021	RNN-Transducer	CSL	Split I: 6.1% / Split II: 49.31%
[72]	2018	Encoder-Decoder (Multi-layer LSTM+Attention Mechanism)	PHOENIX-2014	Dev: 43.1% / Test: 42.1%
[73]	2018	Encoder-Decoder (3D-CNN+HLSTM+Attention Mechanism)	CSL	Split I (Precision): 92.9% / Split II: 64.1%
[74]	2019	2D-CNN+1D-CNN+3D-CNN	PHOENIX-2014	Dev: 38.9% / Test: 38.7%
			CSL	Split II: 61.9%
[75]	2020	Encoder-Decoder (3D-CNN+LSTM+ACL)	CSL	Split I (Precision): 99.1% / Split II: 67.2%
[76]	2019	Encoder-Decoder (3D-ResNet+	PHOENIX-2014	Dev: 37.1% / Test: 36.7%
		BLSTM+LSTM+Soft-DTW) Multi-loss	CSL	Split I (Precision):93.9% / Split II: 32.7%
[77]	2018	3D-CNN+HAN	CSL	Split I: 17.3%
[78]	2020	2D-CNN+1D-CNN	PHOENIX-2014	Dev: 23.7% / Test: 23.9%
[58]	2018	Encoder-Decoder (CNN+RNN+HMM)	PHOENIX-2014	ROUGE: Dev: 44.14% / Test: 43.8%
[79]	2020	Transform (CNN+Self-Attention)	PHOENIX-2014	Dev: 24.9% / Test: 26.2%
[80]	2020	Transformer (Mutli-modal)	PHOENIX-2014	ROUGE: Dev: 45.9% / Test: 43.7%
[81]	2020	Full-inception+Self-attention	PHOENIX-2014	Dev: 31.7% / Test: 31.3%
[82]	2021	CNN+GAN	PHOENIX-2014	Dev: 23.7% / Test: 23.4%
			CSL	Split I: 2.1%

From Table 4, it can be seen that the application of the Transformer structure to sign language recognition/ translation achieved the best results. Because the transform structure mainly adopts the self-attention mechanism instead of the previous RNN based, which avoids the gradient disappearance/explosion problem and solves the information bottleneck problem in an encoder-decoder structure. Therefore, the future research direction of sign language recognition should continue to develop along the direction based on Transform.

4 Sign language dataset and evaluation criteria

4.1 Dataset

With the development of sign language recognition research, the demand for sign language datasets is also expanding, and sign language datasets of various sizes and characteristics across the world have been launched one after another. This article summarizes the datasets that have been widely used in the field of sign language recognition over the years. The detailed classification of sign language datasets is shown in Table 5. From the research on sign language recognition over the years, most work is based on the SIGNUM dataset, RWTH-Phoenix-Weather dataset, CSL dataset, Montalbano dataset, and Chalearn IsoGD dataset. At the same time, CVPR 2021 organizes the ChaLearn LAP evaluation using the AUTSL dataset. The independent and isolated SLR challenge for large-scale signers attracted a large number of participants. Therefore, the information about these six datasets will be described in detail below. Figure 6 shows some sample datasets.

Table 5
A summary of sign language datasets

Dataset Country Data Type Collection Equipment Year Signers Vocabulary Sample

GSL-20 [83] Greece Isolated word Color gloves 2007 6 20 840

SIGNUM [84] Germany Continuous sentence RGB Camera 2008 3 455 2340

DGS Kinect 40 [85] Germany Isolated word Kinect 2012 15 40 3000

Boston ASLLVD [86] America Isolated word RGB Camera 2012 6 2742 9794

PSL Kinect [87] Poland Isolated word Kinect 2013 1 30 300

DEVISIGN-L [88] China Isolated word Kinect 2014 8 2000 24000

DEVISIGN-G [88] China Isolated word Kinect 2014 8 36 432

DEVISIGN-D [88] China Isolated word Kinect 2014 8 500 6000

RWTH-Phoenix-Weather [89] Germany Continuous sentence RGB Camera 2014 9 1200 6861

RWTH-Phoenix-Weather 2014T [58] Germany Continuous sentence RGB Camera 2018 9 1066 8257

CSL-500 [90] China Isolated word Kinect 2016 50 500 12500

CSL [90] China Continuous sentence Kinect 2016 20 178 2500

GSL [91] Greece Isolated word Kinect 2019 7 310 40785

Continuous sentence Kinect 2019 7 310 10295

Montalbano [92] / Gesture Kinect 2014 16 20 13858

Chalearn IsoGD [93] / Gesture Kinect 2016 21 249 47933

1-miohands [21] / Gesture Kinect 2016 23 60 1236680

EgoGesture [94] / Gesture Kinect 2018 50 83 24161

AUTSL [95] Turkey Gesture Kinect 2020 43 226 38336

Dataset	Country	Data Type	Collection Equipment	Year	Signers	Vocabulary	Sample
GSL-20 [83]	Greece	Isolated word	Color gloves	2007	6	20	840
SIGNUM [84]	Germany	Continuous sentence	RGB Camera	2008	3	455	2340
DGS Kinect 40 [85]	Germany	Isolated word	Kinect	2012	15	40	3000
Boston ASLLVD [86]	America	Isolated word	RGB Camera	2012	6	2742	9794
PSL Kinect [87]	Poland	Isolated word	Kinect	2013	1	30	300
DEVISIGN-L [88]	China	Isolated word	Kinect	2014	8	2000	24000
DEVISIGN-G [88]	China	Isolated word	Kinect	2014	8	36	432
DEVISIGN-D [88]	China	Isolated word	Kinect	2014	8	500	6000
RWTH-Phoenix-Weather [89]	Germany	Continuous sentence	RGB Camera	2014	9	1200	6861
RWTH-Phoenix-Weather 2014T [58]	Germany	Continuous sentence	RGB Camera	2018	9	1066	8257
CSL-500 [90]	China	Isolated word	Kinect	2016	50	500	12500
CSL [90]	China	Continuous sentence	Kinect	2016	20	178	2500
GSL [91]	Greece	Isolated word	Kinect	2019	7	310	40785
		Continuous sentence	Kinect	2019	7	310	10295
Montalbano [92]	/	Gesture	Kinect	2014	16	20	13858
Chalearn IsoGD [93]	/	Gesture	Kinect	2016	21	249	47933
1-miohands [21]	/	Gesture	Kinect	2016	23	60	1236680
EgoGesture [94]	/	Gesture	Kinect	2018	50	83	24161
AUTSL [95]	Turkey	Gesture	Kinect	2020	43	226	38336

Fig. 6

Some samples of sign language datasets.

4.1.1 Signum dataset

The SIGNUM database [84] contains isolated words and continuous sentences of various signers. To quickly and randomly access a single frame, it stores each video clip as a series of images. The database includes 455 basic symbols in German Sign Language, representing different word types. A total of 780 sentences are constructed based on this term. The length of each sentence ranges from 2 to 11. The entire corpus has 455 basic symbols and 780 sentences. It was performed by 25 signers of different genders and ages.

4.1.2 Rwth-phoenix-weather dataset

RWTH-Phoenix-Weather (2012) [89] is a daily news and weather forecast recorded by German Phoenix Public Television for three years (2009–2011). It consists of 386 subsets, all videos at 25 frames per second, the frame size is 210×260 pixels, and performed by 7 different signers. The entire corpus consists of 5,356 sentences, 45,760 running glosses, about 600k frames, and has a vocabulary of about 1,200 signs.

4.1.3 CSL dataset

Chinese Sign Language (CSL) Dataset [90] has been collected by the University of Science and Technology of China (USTC) with Kinect devices since 2015. It contains 25K video examples, captured by 50 operators, and has RGB, depth, and skeleton joint data. There are 500 kinds of words, 100 sentences, and a total of 5,000 videos. Each sentence contains an average of 4–8 words. A professional Chinese Sign Language teacher identifies each video example.

4.1.4 Montalbano dataset

Different from the above three standard sign language data, Montalbano [92] is an isolated gesture dataset composed of a series of continuous dynamic gestures. The dataset contains 13,858 RGB-D gesture video samples. Each RGB-D video only represents one gesture, totaling 20. A gesture tag is sampled and executed by 20 operators.

4.1.5 Chalearn Isogd dataset

ChaLearn LAP IsoGD [93] is an extended version of the Montalbano dataset. The total data length of about 24 hours, including 13,858 gesture instances. Due to technical methods’ generality, ChaLearn LAP IsoGD is a test dataset for sign language recognition research. It consists of 47,933 videos, including RGB and depth data. Each RGB-D video represents only one gesture and was performed by 21 operators.

In the recent sign language recognition research, gesture and isolated word recognition research mainly use the ChaLearn IsoGD data set. In contrast, the continuous sign language recognition research mainly uses the RWTH-Phoenix-Weather dataset, CSL dataset, SIGNUM Dataset, etc.

4.1.6 AUTSL dataset

AUTSL [95, 96] is a large multi-modal Turkish sign language data set proposed on CVPR in 2021. The dataset consists of 226 signs performed by 43 different signers and 38,336 isolated sign video samples in total. Samples contain a wide variety of backgrounds recorded in indoor and outdoor environments. Moreover, spatial positions and the postures of signers also vary in the recordings. Each sample is recorded with Microsoft Kinect v2 and contains color image (RGB), depth, and skeleton data modalities.

4.2 Evaluation criteria

4.2.1 The evaluation criteria of gesture, isolated word

Currently, the recognition benchmark of gestures and isolated words is accuracy. Accuracy rate refers to the proportion of correctly identified samples in all samples, as shown in Formula (13). $\begin{matrix} Acc = \frac{TP + TN}{P + N} \end{matrix}$ (13)

TP is the number correctly divided into positive examples. TN is the number correctly divided into negative examples; P is the number of positive samples, and N is the number of negative samples. Generally, the higher the accuracy, the better the recognition effect.

4.2.2 The evaluation criteria of Continuous Sign Language Recognition

The labels of the dataset are complex and varied. Some words are replaced, deleted, or inserted to keep the consistency between sign language sequence and label sequence. Related evaluation indicators for continuous sign language recognition include word error rate (WER), BLEU, METEOR, Rouge, Precision, CIDEr, Acc, etc. Table 6 shows a summary of the evaluation criteria used in the continuous sign language recognition paper.

Table 6
Analysis table of evaluation criteria for continuous Sign language recognition papers

Research Year Evaluation Criteria

WER BLEU CIDEr METEOR Rouge-L ACC Precision

[59] 2016 √

[60] 2017 √

[61] 2018 √

[62] 2019 √

[63] 2017 √

[64] 2019 √

[65] 2017 √

[66] 2018 √

[67] 2019 √

[68] 2019 √

[70] 2018 √ √ √ √ √ √ √

[72] 2020 √ √ √ √ √ √ √

[73] 2018 √ √ √ √ √ √ √

[75] 2020 √ √ √ √ √ √ √

[76] 2019 √ √ √ √ √ √ √

[77] 2018 √

[78] 2020 √

[58] 2018 √ √ √

[79] 2020 √ √

[80] 2020 √ √

[81] 2020 √

[82] 2021 √ √

Research	Year	Evaluation Criteria
[59]	2016	√
[60]	2017	√
[61]	2018	√
[62]	2019	√
[63]	2017	√
[64]	2019	√
[65]	2017	√
[66]	2018	√
[67]	2019	√
[68]	2019	√
[70]	2018	√	√	√	√	√	√	√
[72]	2020	√	√	√	√	√	√	√
[73]	2018	√	√	√	√	√	√	√
[75]	2020	√	√	√	√	√	√	√
[76]	2019	√	√	√	√	√	√	√
[77]	2018						√
[78]	2020	√
[58]	2018	√	√			√
[79]	2020	√	√
[80]	2020		√			√
[81]	2020	√
[82]	2021	√	√

a) Word Error Rate (WER): It evaluates the similarity between the predicted sequence and the reference sequence. It measures the minimum number of edits (addition, deletion, replacement) from the target sequence to the reference sequence, $\begin{matrix} WER = \frac{I + D + S}{L} \end{matrix}$ (14)

L is the total number of standard sequence words. I, D, and S represent the total number of inserted, deleted, and replaced words. Generally, the smaller the WER, the better the recognition effect.

b) BLEU: It analyzes the degree of co-occurrence of n-tuples in the candidate sequence and the reference sequence to determine the similarity between the two sentences, $\begin{matrix} BLUE = BP * \exp (\sum_{n = 1}^{N} W_{n} {logP}_{n}) \end{matrix}$ (15)

Where, $\begin{matrix} BP = {\begin{matrix} {1 if l}_{c} {> l}_{s} \\ e^{1 - \frac{l_{s}}{l_{c}}} {if l}_{c} ⩽ l_{s} \end{matrix} \end{matrix}$ (16)

BP is the penalty factor. Generally speaking, the higher the BLEU, the better the recognition rate.

c) METEOR: Lavir proposed after discovering the recall rate’s significance in the evaluation index in 2004. The goal is to address some of the flaws inherent in the BLEU standard. Unlike BLEU, METEOR also considers the accuracy and recall rate based on the entire corpus and finally gets a measurement. The calculation process is as follows: $\begin{matrix} Pen = γ {(\frac{ch}{m})}^{θ} \end{matrix}$ (17) $\begin{matrix} F_{mean} = \frac{P_{m} R_{m}}{{α P}_{m} + (1 - α) R_{m}} \end{matrix}$ (18) $\begin{matrix} P_{m} = \frac{| m |}{\sum_{k} h_{k} (c_{i})} \end{matrix}$ (19) $\begin{matrix} R_{m} = \frac{| m |}{\sum_{k} h_{k} (s_{i})} \end{matrix}$ (20) $\begin{matrix} METEOR = (1 - Pen) F_{mean} \end{matrix}$ (21)

Where α, β, and γ are the default parameters for evaluation, Pen is the penalty parameter. Generally, the higher the METEOR score, the better.

d) Precision: It refers to the ratio of correct sentences to the total number of sentences. $\begin{matrix} Precision = \frac{TP}{TP + FP} \end{matrix}$ (22)

TP is the number correctly divided into positive examples; FP is the number incorrectly divided into positive examples. Generally, the higher the accuracy, the better the recognition effect.

e) Rouge: This metric represents matching the longest common sequence of two text units. It is a similarity measurement method based on the recall rate, which focuses on the reference’s adequacy and loyalty rather than its fluency. It is almost the same as BLEU’s calculation method, but the N-gram phrase is derived from the reference translation. Rouge is divided into four types: ROUGE—Co-occurrence statistics based on N-gram; ROUGE-L—Statistics of co-occurrence accuracy and recall rate based on the longest common clause; ROUGE-W—Statistics on the co-occurrence accuracy and recall rate of the longest shared clause with weights; ROUGE-S—Statistics based on discontinuous binary co-occurrence accuracy and recall rate.

f) CIDEr: it is a metric for image summary proposed by Vedantm at the 2015 Computer Vision and Pattern Recognition Conference to evaluate the similarity between computer-generated sentences and manual descriptive sentences. It calculates the cosine distance of the TF-IDF necklace between each reference sentence and the sentence. The calculation formula is as (23): $\begin{matrix} {CIDEr}_{n} (c_{i} {, s}_{i}) = \frac{1}{M} * \sum_{j = 1}^{M} \frac{g^{n} (c_{i}) {* g}^{n} (s_{ij})}{g^{n} (c_{i}) {* g}^{n} (s_{ij})} \end{matrix}$ (23)

Where, $\begin{matrix} g_{k} (c_{i}) = TF (k) * IDF (k) \end{matrix}$ (24)

K is the K-th N-gram in a sentence, ci is the machine translation corresponding to the i-th picture, s_ij is the j-th translation in the manual translation corresponding to the i-th picture (multiple), and M is the number of human translations corresponding to each picture.

As shown in Table 6, from the continuous sign language recognition research of recent years, we can find that the most widely used evaluation standard is WER. However, in the last two years, the BLEU evaluation standard has also been widely used. Evaluating standards such as METEOR, Rouge, Precision, CIDEr, and Acc have also come into scholars’ view to jointly evaluate the quality of a model.

5 Conclusion and prospect

This article reviews the main directions and research status of sign language research around the world in recent years, introducing some representative work of sign language recognition, the process of sign language recognition, and listing the current more popular sign language databases. It can be seen from this article that the research on sign language recognition has achieved initial results, and many effective algorithms have been proposed over time. However, there are still some shortcomings, such as the single type at the input stage, the inability to focus on chiral and achiral features at the same time, and the lack of key information for feature extraction. The direction of future research will be to propose a model with higher accuracy and real-time application in real life is sign language recognition.

Sign language recognition has extensive application potential in computer vision, human-computer interaction, and other fields and has always been a research hotspot. The following aspects will be worthy of the attention of researchers going forward:

Pay attention to cross-disciplinary and cross-domain sign language research: The sign language recognition task should be combined with different fields to build a more lightweight, accurate, and real-time human-computer interaction method. At the same time, the application of sample-based cross-domain knowledge transfer and model-based transfer methods in sign language research is also one of the directions that can be explored and extended.

Sign language recognition in complex backgrounds: sign language datasets used by researchers are recorded in a fixed environment using relatively fixed equipment. However, sign language is usually complicated, such as too bright or too dark, different distances from the collection equipment, more non-sign language, etc. Complex background factors will lead to greater accuracy, therefore, there should be an improvement in the robustness of sign language recognition systems. Improving applicability while ensuring accuracy is a more challenging issue.

Sign language has excellent flexibility and details: Sign language uses different parts of the body, such as fingers, hands, arms, head, body and facial expressions, etc., which have an important influence on the semantic expression of sign language. Therefore, sign language recognition is still focused on using more abundant information about recognizing sign language and improving recognition accuracy.

Detection of sign language word boundaries: Sign language’s word segmentation research of continuous sign sentences is also a significant research trend.

Improvement to recognition algorithms: For continuous sentence recognition with an intense time sequence, the hybrid network structure will be the mainstream network algorithm in the future. The success of some algorithms in related fields, such as NMT, will also promote the further development of continuous sign language recognition.

Footnotes

Acknowledgments

This work was supported by the Collaborative education project of industry university cooperation of the Ministry of Education under Grant 202102594001, the Key teaching reform projects of Tianjin University of Technol-ogy under Grant ZD21-17, National Natural Science Foundation of China (Grant No. 61806071), the Open Projects Program of National Laboratory of Pattern Recognition (Grant No. 201900043), the Sci-tech Research Projects of Higher Education of Hebei Province, China (Grant No. QN2019207), and the Tianjin Sci-tech development strategy research planning Projects (Grant No. 18ZLZXZF00660).

References

Zhang

S.J.

, Zhang

and Li

, Review of Sign Language Recognition Based on Deep Learning, Journal of Electronics and Information Technology 42(4) (2020), 1021–1032.

Grimes

G.J.

, Digital data entry glove interface device, 1983.

Fels

S.S.

and Hinton

G.E.

, Glove-talk II – a neural-network interface which maps gestures to parallel formant speech synthesizer controls, IEEE Transactions on Neural Networks 8(5) (1997), 977–984.

, Chen

, Cao

, et al., Exploration of Chinese Sign Language Recognition Using Wearable Sensors Based on Deep Belief Net, IEEE Journal of Biomedical and Health Informatics (2020), 1310–1320.

Chu

, Liu

and Shimamoto

, A Sensor-Based Hand Gesture Recognition System for Japanese Sign Language, 2021 IEEE 3rd Global Conference on Life Sciences and Technologies (LifeTech) (2021), 311–312.

Huang

, Deep Learning Based Sign Language Recognition, University of Science and Technology of China, 2018.

Chai

X.K.

, Liu

and Chen

, Continuous Gesture Recognition with Hand-Oriented Spatiotemporal Feature, 2017 IEEE International Conference on Computer Vision Workshops (ICCVW), 2017, 3056–3064.

Feng

, Signer-Independent Continuous Chinese Sign Language Recognition with Kinect, Shandong University, 2018.

Bencherif

M.A.

, Algabri

, Mekhtiche

M.A.

, et al., Arabic Sign Language Recognition System Using 2D Hands and Body Skeleton Data, IEEE Access 9 (2021), 59612–59627.

10.

Raheja

, Mishra

and Chaudhary

, Indian sign language recognition using svm, Pattern Recognition and Image Analysis 26(2) (2016), 434–441.

11.

Ten

H.G.A.

and Reinders

M.J.T.

, Multi-dimensional dynamic time warping for gesture recognition, Annual Conference on the Advanced School for Computing and Imaging, 2007.

12.

Kishore

, Prasad

C.R.

, et al., 4-camera model for sign language recognition using elliptical Fourier descriptors and Ann, International Conference on Signal Processing and Communication Engineering Systems (2015), 34–38.

13.

Alam

M.S.

, Tanvir

, Saha

D.K.

, et al., Two Dimensional Convolutional Neural Network Approach for Real-Time Bangla Sign Language Characters Recognition and Translation, SN COMPUT, 2021.

14.

Park

H.J.

, Lee

and Ko

J.G.

, Enabling Real-time Sign Language Translation on Mobile Platforms with On-board Depth Cameras, ACM Interact, 2021.

15.

Konecny

and Hagara

, One-shot-learning gesture recognition using HOG-HOF features, J Mach Learn 15 (2014), 2513–2532.

16.

Huang

D.F.

and Yang

J.D.

, Dynamic gesture recognition based on improved ND-DTW algorithm, Electronic Science and Technology 30(3) (2017), 37–40.

17.

Zhou

Z.H.

, Dai

Y.K.

and Li

W.S.

, Gesture Recognition Based on Global Template DTW for Chinese Sign Language, Journal of Intelligent & Fuzzy Systems 35(2) (2018), 1969–1978.

18.

Yan

, Liu

, Huang

, et al., Research on gesture recognition based on HMM, Journal of Huazhong Normal University (Natural Sciences) 46(5) (2012), 555–559.

19.

Saha

, Lahiri

, Konar

, et al., HMM-based gesture recognition system using kinect sensor for improvised human-computer interaction, 2017 International Joint Conference on Neural Networks (IJCNN), Anchorage, 2017, 2776–2783.

20.

Yang

R.D.

and Sarkar

, Gesture Recognition using Hidden Markov Models from Fragmented Observations, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06), 2006, 766–773.

21.

Koller

, Ney

and Bowden

, Deep Hand: How to Train a CNN on 1 Million Hand Images When Your Data is Continuous and Weakly Labelled, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, 3793–3802.

22.

Guo

and Yang

, Gesture recognition based on HMM-FNN model using a Kinect, J on Multi-modal User Interfaces 11(1) (2017), 1–7.

23.

Chen

, Liu

and Zhao

, Temporal Hierarchical Dictionary with HMM for Fast Gesture Recognition, 2018 24th International Conference on Pattern Recognition (ICPR), 2018, 3378–3383.

24.

Kang

, Tripathi

and Nguyen

T.Q.

, Real-time sign language fingerspelling recognition using convolutional neural networks from depth map, 2015 3rd IAPR Asian Conference on Pattern Recognition (ACPR), 2015, 136–140.

25.

Karpathy

, Toderici

, Shetty

, et al., Large-Scale Video Classification with Convolutional Neural Networks, 2014 IEEE Conference on Computer Vision and Pattern Recognition, 2014, 1725–1732.

26.

, Zhou

, Chen

, et al., Arm removal for static hand gesture recognition, Journal of Intelligent and Fuzzy Systems 35(8) (2018), 1–12.

27.

Tran

, Bourdev

, Fergus

, et al., Learning Spatiotemporal Features with 3D Convolutional Networks, 2015 IEEE International Conference on Computer Vision (ICCV), 2015, 4489–4497.

28.

Zhu

, Zhang

, Mei

, et al., Large-scale Isolated Gesture Recognition using pyramidal 3D convolutional networks, 2016 23rd International Conference on Pattern Recognition (ICPR), 2016, 19–24.

29.

Y.N.

, Large-scale gesture recognition with a fusion of RGB-D data based on the C3D model, 2016 23rd International Conference on Pattern Recognition (ICPR), 2016, 25–30.

30.

Escalante

H.T.

, ChaLearn Joint Contest on Multimedia Challenges Beyond Visual Analysis: An overview, 2016 23rd International Conference on Pattern Recognition (ICPR), 2016, 67–73.

31.

Molchanov

, Gupta

, Kim

, et al., Hand gesture recognition with 3D convolutional neural networks, 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2015, 1–7.

32.

Miao

, Li

, Ouyang

, et al., Multi-modal Gesture Recognition Based on the ResC3D Network, 2017 IEEE International Conference on Computer Vision Workshops (ICCVW), 2017, 3047–3055.

33.

Molchanov

, Yang

, Gupta

, et al., Tyree

and Kautz

, Online Detection and Classification of Dynamic Hand Gestures with Recurrent 3D Convolutional Neural Networks, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, 4207–4215.

34.

Zhu

, Zhang

, Shen

, et al., Multi-modal Gesture Recognition Using 3-D Convolution and Convolutional LSTM, IEEE Access 5 (2017), 4517–4524.

35.

Al-Hammadi

, Muhammad

, Aslulaiman

, et al., Deep Learning-Based Approach for Sign Language Gesture Recognition with Efficient Hand Gesture Representation, IEEE Access 8 (2020), 192527–192542.

36.

, Luo

, Wu

, et al., Hand gesture recognition based on attentive feature fusion, Concurrency and Computation Practice and Experience 32(1) (2019).

37.

Pigou

, Dieleman

, Kindermans

P.J.

, et al., Sign language recognition using convolutional neural network, European Conference on Computer Vision, 2014, 572–578.

38.

Koller

, Bowde

and Ney

, Automatic alignment of hanNoSys subunits for continuous sign language recognition, Language Resources and Evaluation Conference, 2016, 121–128.

39.

Kopuklu

, Kose

and Rigoll

, Motion fused frames: Data level fusion stratrgy for hand gesture recognition, IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2018, 2103–2111.

40.

Huang

, Zhou

W.G.

, Li

, et al., Sign Language Recognition using 3D convolutional neural networks, 2015 IEEE International Conference on Multimedia and Expo (ICME), Turin, 2015, 1–6.

41.

Liang

Z.J.

, Liao

S.B.

and Hu

B.Z.

, 3D Convolutional Neural Networks for Dynamic Sign Language, The Computer Journal 61(11) (2018), 1724–1736.

42.

and Shao

, Deep Dynamic Neural Networks for Multi-modal Gesture Segmentation and Recognition, IEEE Transactions on Pattern Analysis and Machine Intelligence 38(8) (2016), 1583–1597.

43.

Huang

, Zhou

, Li

, et al., Attention-Based 3D-CNNs for Large-Vocabulary Sign Language Recognition, IEEE Transactions on Circuits and Systems for Video Technology 29(9) (2019), 2822–2832.

44.

Sharma

and Kumar

, ASL-3DCNN: American sign language recognition technique using 3-D convolutional neural networks, Multimedia Tools and Applications 80(17) (2021), 26319–26331.

45.

Miao

Q.G.

, Li

Y.N.

, Ouyang

W.L.

, et al., Multi-modal gesture recognition based on the resc3d network, IEEE International Conference on Computer Vision Workshops, 2017, 3047–3055.

46.

Liao

, Xiong

, Min

, et al., Dynamic Sign Language Recognition Based on Video Sequence With BLSTM-3D Residual Networks, IEEE Access 7 (2019), 38044–38054.

47.

Razieh

, Kourosh

and Sergio

, Hand sign language recognition using multi-view hand skeleton, Expert Systems with Applications, 2020, 0957–4174.

48.

Jiang

, Sun

, Wang

, et al., Skeleton Aware Multi-modal Sign Language Recognition, 2021 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021, 3413–3423.

49.

Meng

and Li

, An Attention-Enhanced Multi-Scale and Dual Sign Language Recognition Network Based on a Graph Convolution Network, Sensors, 2021.

50.

Chai

X.J.

, Liu

Z.P.

, Yin

, et al., Two streams recurrent neural networks for large-scale continuous gesture recognition, International Conference on Pattern Recognition, 2016, 31–36.

51.

Y.C.

, Tian

Y.L.

, Huenerfauth

, et al., Recognition American sign language gestures from within continuous video, IEEE Conference on Computer Vision and Pattern Recognition, 2018, 2064–2073.

52.

Huang

, Mao

, Tao

, et al., A Novel Chinese Sign Language Recognition Method Based on Keyframe-Centered Clips, IEEE Signal Processing Letters 25(3) (2018), 442–446.

53.

Lin

, Wan

, Liang

, et al., Large-Scale Isolated Gesture Recognition Using a Refined Fused Model Based on Masked Res-C3D Network and Skeleton LSTM, 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), 2018, 52–58.

54.

Ameur

, Khalifa

A.B.

and Bouhlel

M.S.

, A novel Hybrid Bidirectional Unidirectional LSTM Network for Dynamic Hand Gesture Recognition with Leap Motion, Entertainment Computing 35 (2020).

55.

Santos

C.C.D.

, Samatelo

and Vassallo

R.F.

, Dynamic Gesture Recognition by Using CNNs and Star RGB: a Temporal Information Condensation, Neurocomputing 400 (2020), 238–254.

56.

Huang

and Ye

, Boundary-Adaptive Encoder With Attention Method for Chinese Sign Language Recognition, IEEE Access, 2021, 70948–70960.

57.

Rodri

, Moreno

, Jos, et al., Sign Language Recognition by Means of Common Spatial Patterns, ICMLSC 21:2021 The 5th International Conference on Machine Learning and Soft Computing, 2021.

58.

Camgoz

C.N.

, Hadfield

, Koller

, et al., Neural sign language translation, IEEE Conference on Computer Vision and Pattern Recognition, 2018, 7784–7793.

59.

Koller

, Zargaran

, Ney

, et al., Deep Sign: Hybrid CNN-HMM for Continuous Sign Language Recognition, Proceedings of the British Machine Vision Conference (BMVC) 136 (2016), 1–12.

60.

Koller

, Zargaran

and Ney

, Re-Sign: Re-Aligned End-to-End Sequence Modelling with Deep Recurrent CNN-HMMs, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, 2017, 3416–3424.

61.

, Zhou

and Li

, Dilated Convolutional Network with Iterative Optimization for Continuous Sign Language Recognition, International Joint Conference on Artificial Intelligence, 2018, 885–891.

62.

Guo

, Wang

, Tian

, et al., Dense Temporal Convolution Network for Sign Language Translation, International Joint Conference on Artificial Intelligence, 2019, 744–750.

63.

Camgoz

, Hadfield

, Koller

, et al., SubUNets: End-to-End Hand Shape and Continuous Sign Language Recognition, 2017 IEEE International Conference on Computer Vision (ICCV), 2017, 3075–3084.

64.

Cui

, Liu

and Zhang

, A Deep Neural Framework for Continuous Sign Language Recognition by Iterative Training, IEEE Transactions on Multimedia 21(7) (2019), 1880–1891.

65.

Cui

, Liu

and Zhang

, Recurrent Convolutional Neural Networks for Continuous Sign Language Recognition by Staged Optimization, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, 1610–1618.

66.

Wang

, Guo

, Zhou

W.G.

, et al., Connectionist Temporal Fusion for Sign Language Translation, ACM International Conference on Multimedia, 2018, 1483–1491.

67.

Song

, Guo

, Xin

, et al., Parallel Temporal Encoder for Sign Language Translation, 2019 IEEE International Conference on Image Processing (ICIP), 2019, 1915–1919.

68.

Yang

, Shi

, Shen

, et al., SF-Net: Structured Feature Network for Continuous Sign Language Recognition, IEEE Conference on Computer Vision and Pattern Recognition, 2019.

69.

Min

, Hao

, Chai

, et al., Visual Alignment Constraint for Continuous Sign Language Recognition, 2021 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.

70.

Gao

L.Q.

, Li

H.B.

, Liu

Z.J.

, et al., RNN-Transducer based Chinese Sign Language Recognition, Neurocomputing, 2021, 45–54.

71.

Bahdanau

, Cho

and Bengio

, Neural Machine Translation by Jointly Learning to Align and Translate, International Conference on Learning Representations, 2015.

72.

Guo

, Wang

, Zhou

W.G.

, et al., Continuous sign language video automatic translation method based on asymmetric multi-layer LSTM, 2018.

73.

Guo

, Zhou

, Li

, et al., Hierarchical LSTM for Sign Language Translation, AAAI Conference on Artificial Intelligence, 2018, 6845–6852.

74.

Guo

, Tang

S.G.

and Wang

, Connectionist Temporal Modeling of Video and Language: a Joint Model for Translation and Sign Labeling, Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, 2019, 751–757.

75.

Guo

, Zhou

, Li

, et al., Hierarchical Recurrent Deep Fusion Using Adaptive Clip Summarization for Sign Language Translation, IEEE Transactions on Image Processing 29 (2020), 1575–1590.

76.

J.F.

, Zhou

W.G.

and Li

H.Q.

, Iterative Alignment Network for Continuous Sign Language Recognition, IEEE Conference on Computer Vision and Pattern Recognition, 2019, 4165–4174.

77.

Huang

, Zhou

, Zhang

, et al., Video-based Sign Language Recognition without Temporal Segmentation, AAAI Conference on Artificial Intelligence, 2018.

78.

Cheng

K.L.

, Yang

, Chen

, et al., Fully Convolutional Networks for Continuous Sign Language Recognition[M], European Conference on Computer Vision, 2020, 697–714.

79.

Camgoz

N.C.

, Koller

, Hadfield

, et al., Sign Language Transformers: Joint End-to-End Sign Language Recognition and Translation, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, 10020–10030.

80.

Camgoz

N.C.

, Koller

, Hadfield

, et al., Multi-channel Transformers for Multi-articulatory Sign Language Translation, European Conference on Computer Vision, 2020, 301–319.

81.

Zhou

M.J.

, Ng

, Cai

Z.X.

, et al., Self-Attention-Based Fully-Inception Networks for Continuous Sign Language Recognition, European Conference on Artificial Intelligence, 2020, 2832–2839.

82.

Papastratis

, Dimitropoulos

and Daras

, Continuous Sign Language Recognition through a Context-Aware Generative Adversarial Network, Sensors, 2021.

83.

Efthimiou

and Fotinea

S.E.

, GSLC: Creation and annotation of a Greek sign language corpus for HCI, International Conference on Universal Access in Human Computer Interaction, 2007, 657–666.

84.

Vonagris

, Zieren

, Canzler

, et al., Recent developments in visual sign language recognition, Universal Access in the Information Society, 2008, 323–362.

85.

Ong

E.J.

, Cooper

, Pugeault

, et al., Sign language recognition using sequential pattern trees, 2012 IEEE Conference on Computer Vision and Pattern Recognition, 2012, 2200–2207.

86.

Neidle

, Thangali

and Sclaroff

, Challenges in development of the American Sign Language lexicon video dataset (ASLLVD) corpus, The 5th Workshop on the Representation and Processing of Sign Languages: Interactions between Corpus and Lexicon, 2012.

87.

Oszust

and Marian

, Polish sign language words recognition with Kinect, International Conference on Human System Interaction, 2013, 219–226.

88.

Chai

X.J.

, Wang

H.J.

and Chen

X.L.

, The DEVISIGN large vocabulary of Chinese sign language database and baseline evaluations, Technical Report VIPL-TR-14-SLR-001, 2014.

89.

Forster

, Schmidt

, Hoyoux

, et al., RWTH-PHOENIX-Weather: A large vocabulary sign language recognition and translation corpus, The 8th International Conference on Language Resources and Evaluation, 2012, 3785–3789.

90.

Huang

, Zhou

W.G.

, Zhang

Q.L.

, et al., Video-based sign language recognition without temporal segmentation, The 32nd AAAI Conference on Artificial Intelligence, 2018, 2257–2264.

91.

Adaloglou

, Chatzis

, Papastratis

, et al., A comprehensive study on deep learning-based methods for sign language recognition, IEEE Transactions on Multimedia 99 (2021), 1–1.

92.

Escalera

, Baroo

, Gonzaalez

, et al., Chalearn looking at people challenge 2014: Dataset and results, European Conference on Computer Vision, 2014, 459–473.

93.

Wan

, Zhao

Y.B.

, Zhou

, et al., ChaLearn Looking at People RGB-D Isolated and Continuous Datasets for Gesture Recognition, 2016 IEEE Conference on Computer Vision and Pattern Recognition, 2016.

94.

Zhang

, Cao

, Cheng

, et al., EgoGesture: A New Dataset and Benchmark for Egocentric Hand Gesture Recognition, IEEE Transactions on Multimedia, 2018, 1038–1050.

95.

Sincan

O.M.

and Keles

H.Y.

, AUTSL: A Large Scale Multi-Modal Turkish Sign Language Dataset and Baseline Methods, IEEE Access, 2020, 181340–181355.

96.

Sincan

O.M.

, Junior

, Escalera

, et al., ChaLearn LAP Large Scale Signer Independent Isolated Sign Language Recognition Challenge: Design, Results and Future Research, 2021 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.