Manual and non-manual sign language recognition framework using hybrid deep learning techniques

There are primarily three core components of a one-stage identifier including its backbone, neck and head. The overall features of the input image with different sizes are extracted by using convolutional neural network of a backbone. The network layers of neck combine the previously extracted features by backbone network with predetermined rules to enhance the semantic facts of the features. YOLO here act as one-stage identifier and take object detection and regression as same where as the model only requires to execute one function on the entire input image by utilizing end-to-end neural network. After splitting the image into grid region, the network also predicts rectangular boxes for these particular regional grids. Grids determine the score and location for most of these rectangular boxes but only for those whose centroid lies within. A 5D output consisting coordinates along with confidence level is eventually produced for an individual rectangular box. Unfortunately, this inference rapid technique lacks in accuracy. On the other hand this version of YOLO has several advantages including faster inference, simplified architecture and user friendly interface.

With crucial modules like Focus, CSPbottleneck, SPP, and PANet, right now, YOLOv5 has a number of benefits regarding speed and precision. The backbone is chosen to be CSPDarknet53, which has the structure of several residual networks stacked on top of one another. For convolutional layers the mash acts as the activation function, for an individual Residual block body component inside the residual model passes via one down sampling as well as numerous residual models. The feature map output from the backbone is used as the model’s input by YOLOv5, which uses PANet as the neck of the structure. After this feature map is fused, features with more robust semantic data is created and supplied to the head network for identification. The PANet architecture is improved in terms of extracting features depending on FPN, which provides semantic as well as geographic knowledge. A bottom-up pyramid is introduced to the FPN architecture to transfer powerful coordinate features from the bottom level to the higher layer in order to supplement the FPN feature fusion [27]. The architecture of YOLOv5 shown in Fig. 2.

OPEN IN VIEWER

Fig. 2

YOLOV5 architecture [27].

When it comes to object detection YOLO has gained a lot of attention recently. YOLOv5 utilizes down sampling in multiples of 8 times to identify and acquire more feature information of various size and shapes. As a result, a lot of positional data has been lost which makes small objects hard to find. Information from three different scales is merged to generate multiscale information that takes into account small targets’ semantic as well as position information through down sampling at various levels three outputs of YOLOv5 are generated. Each contain distinctive semantic as well as positional information.

The proposed model keeping the fact that most of the targeted objects in the dataset may be small. Before characterizing them in dimensions of channel up and down sampling is performed on y1 and y3. To get better extracted features, an attention module is embedded. GC block is used to improve the network even more. Feature fusion is advantageous in this case since it records cross channel dependencies. Then after performing the operations for up and down sampling y1’, y2’, and y3’ are generated as new output as shown in Fig. 3.

OPEN IN VIEWER

Fig. 3

Improved module in YOLOv5.

C3D can concurrently simulate both type of information based on movement as well as visuals, and it can be trained on substantially large datasets. Beyond 2D convolutional networks, C3D expands and improves convolution models and learns from temporal directions in constant RGB frames excluding the preprocessing steps it can be trained on temporal relations and discriminative features both simultaneously. The input data is acquired by distinguished trainable features of C3D network which captures spatial and temporal characteristics automatically for classification [28].

Therefore, the proposed work uses C3D approach to extract features of the input video. The 3D kernel and pooling size is given as d x k x k, here kernel temporal depth is mentioned as d, and the spatial size as k. The calculation made for activation layer Leaky ReLU, 3D max pooling and 3D convolution layer explain below.

A 3D image is convolved to create a 3D convolutional cube by piling several adjacent frames at once. The feature maps are also generated by using the same method. Future layers are linked with the previous layers’ numerous consecutive frames. Specifically, the jth feature maps’ ith value at (x, y, z) position is determined by Equation (1) v ij xyz = f { b ij + ∑ k = 1 m ∑ p = 0 pi - 1 ∑ q = 0 qi - 1 ∑ r = 0 ri - 1 w ijk pqr v ( i - 1 ) ( x + p ) * ( y + q ) * ( z + r ) }(1)

here kernel size of the 3D convolution is denoted by P_i, Q_i and R_i the number of prior layer feature map (i - 1) is denoted by m, the k_th feature map value is v ( i - 1 ) ( x + p ) * ( y + q ) * ( z + r ) which is determined in the (i - 1) layer by (x + p) , (y + q) , (z + r) the conventional kernel applied to this map is w ijk pqr that shows the leaky ReLU function, and partial set is determined by b _ij .

Leaky ReLU is capable of converting the input with the total weight. By a distinctive linear function, if positive input is given it will directly return the same output else the output is null. In order to produce nonlinearity leaky ReLU with nonlinearity is applied with ConvNet. 3D max pooling layer increase the reliability of the 3D CNN model, it gives certain degree of invariance to the feature of cube in time dimension. The max pooling formula is mentioned in Equation (2):

v ij xyz = 0 ⩽ i , j , k ⩽ S 1 , S 2 , S 3 Maximum ( U ( x . s ) + i , j + ( y . t ) , k + ( z . r ) )(2)

Here u and v are the input and output vector in the 3D pooling operation. The sampling size is given as S₁ x S₂ x S₃ where the sampling set in three dimension (x,y,z) is (s,t,r). The proposed C3D module consist of five blocks as shown in Fig. 4. Both block 1 and block 2 consist of leaky ReLU function, a 3D convolutional layer and a pooling layer, the rest of the blocks contain two of each activation function, 3D pooling and convolutional layer. In all these five convolutional layers 64 to 512 number of filters are used. The proposed model uses kernel size of 3x3x3 during the convolutional process, in this way it helps the differences in spatial and temporal features to get acquired easily. These five blocks are parallel to one another in classifying different behaviors subjectively as well as categorize visual content at various level.

OPEN IN VIEWER

Fig. 4

Proposed C3D feature extractor for face ROIs.

For pooling purpose 3D max pooling was applied. To keep the temporal information and to avoid blending the temporal signals the pooling layer uses a 3x3x3 kernel size. The same order was kept in the last layers which reduced the 3D CNN feature by 10 folds. When both higher and lower spatial resolutions are compared, they keep equal feature maps. Feature vectors have been generated by transforming the input frames, these vectors are capable of capturing motion information using various conventional layers and subsampling from input videos. Spatio-temporal characteristics can be considered as an output given by an individual convolutional layer. Many semantic and separated features are generated by higher layers by underlying the features by lower layers such as colors and edges. Among the five convolutional layers the last consist of larger receptive fields including invariant and discriminative features. Using the above mechanism, the extracted features of face ROIs as well as hand ROI are concatenated.

The high-level temporal attributes are modelled by using LSTM because low level temporal and spatial features can successfully be learned by using C3D. Softmax classifier is dropped from the classic C3D model and features are incremented to the last 1x1x1 layer of the LSTM model in actual practice. Figure 5 shows the architecture of LSTM.

OPEN IN VIEWER

Fig. 5

LSTM network.

Here input signal, state of the cell and hidden state are represented by “x”, “c” and “where σ and tanh are the two-activation function. Forget gate, input gate and output gate keep the state of the LSTM cell updated by keeping the desires information from the history and removing the rest. The Equation (3) shows the calculation process.

f TM = σ * [ W g * ( h TM - 1 , x TM ) ] + b g(3)

In the Equation (3), bias and weight matrix are represented by W _g and b _g . The current timestep of the input state x _TM is combined with the previous timestep of the hidden state h_TM-1. The forget gate can be made to remove or keep the previous state the output of σ function ranging from 0 to 1. The input gate can be represented by Equation (5) independently, the process of updating the cell is shown in equation (6) j TM = σ * [ W j * ( h TM - 1 , x TM ) ] + b j(4) y TM = tanh [ W y * ( h TM - 1 , x TM ) ] + b y(5) c TM = f TM * c TM - 1 + j TM * y TM(6)

Equation (7) and Equation (8) exhibit the formation of the output gate. o TM = σ * [ W o * ( h TM - 1 , x TM ) ] + b o(7) h TM = o TM * tanh ( c TM )(8)

4.1 Datasets

The study of manual and non-manual sign language recognition is not yet fully explored. Non-manual gestures specifically facial expression plays an important part in terms of emotion detection due to which it has gained a lot of attention. The proposed study used three different datasets to evaluate and train the model. All datasets; PkSLMNM [29], RWTH-PHONIX WEATHER 2014T [22, 30] and SILFA [31] are publicly available. Frames from all datasets were extracted and then annotated for training and testing purpose. Both C3D and LSTM were trained on extracted and annotated frames of these datasets individually, as the affective manual expression are similar to all but manual signs are different for all languages worldwide discussed in section 1 of this research study.

4.2 Implementation details

For implementation PyTorch deep learning framework is used. The model is processed on Ubuntu 20.04 with NVIDIA GeForce GTX 1080Ti and Intel(R) Core (TM) i7-3770 CPU with 3.40 GHz and 64GB RAM. The proposed work utilizes stochastic gradient descent for training advanced YOLOv5 in a point-to-point manner. With each regularization, weight of the models is updated every time, 64 batch size is set for training the model. Having 400 epochs both rate decay and momentum are set to 0.0005 and 0.910.

4.3 Evaluation and results

In Fig. 6 contains loss curves of training and validation. The graph represents trend of loss value that reduces drastically from the beginning of the first Iterations and becomes stable after training of 2000 alterations at 0.3426.

OPEN IN VIEWER

Fig. 6

Error loss of improved YOLOV5.

Quantitative analysis to evaluate the proposed model is made using precision and recall. The formula to calculate precision and recall is mentioned in Equations (10).

PR = TP TP + FP(9) R = TP TP + FN(10)

Here the correctly detected and wrongly detected objects are shown by True positive (TP) and False positive (FP) whereas the correct objects which could not get detected were shown by False Negative (FN), The harmonic average between the calculated accuracy and recall is calculated by F1 score mention in Equation 11, to evaluate the performance of the model it is one of the important criteria. F 1 = 2 * Precision * Recall Precision + Recall(11)

To verify the performance of the proposed model, it has been test on the different databsets. An evaluation Table 3. is given below which shown precision, recall and F1 of the proposed model on multiple datasets.

LSTM is proposed for the classification of the video. To train and validate the model, 250 epochs are run on PkSLMNM dataset as well as on others. The model was trained on the extracted features from proposed C3D. The training and validation shown in Fig. 7.

OPEN IN VIEWER

Fig. 7

LSTM training and validation accuracy using PkSLMNM dataset.

The proposed model achieves training accuracy of 83% and 79% validation accuracy as shown in Fig. 7. Compared with traditional model, it showed higher test accuracy, its training time also reduced two folds it occupied less GPU memory, didn’t showed any over-fitting issues when epochs were increased, which proved that is out performed the traditional model shown in Table 4. The sample result of the model shown in Fig. 8.

OPEN IN VIEWER

Fig. 8

Sample results on proposed model.

Table 4 shows the evaluation of models based on their performances, the computation required to process Giga FLOPs (floating point operations per second) and parameters. The result show that the proposed model outperformed with better performance. The confusion matrix is plotted against a graphical 2D map shown in Fig. 9. 20 videos are used for validation among which the model identified true positive, false negative, false positive and true negative values successfully.

OPEN IN VIEWER

Fig. 9

Confusion matrix.