Turkish sign language recognition using fuzzy logic asisted ELM and CNN methods

1 Introduction

Hearing and speech-impaired people can communicate using sign language comprising certain hand, finger, wrist, and arm movements. Numerous studies on sign language recognition have been conducted in various languages. To truly assist deaf/mute people and create a sustainable system, the developed equipment should be used in real-time to convert translated words into sound. According to the literature, various types of cameras are primarily used for hand and body gesture recognition. However, in real-time use, these systems are limited by equipment transportation, installation, and environmental difficulties (light and weather conditions, for example). Data gloves are also used in sign language recognition studies. They are difficult to produce because they must be comfortable to wear, the majority of connections must be wireless, and sensors must be carefully placed. When the studies of both types were examined, it was observed that the majority of the studies only performed letter and number recognition, whereas word and sentence recognition studies were less frequent.

In this paper, a new fuzzy logic-assisted ELM-based classification method is proposed. The number of hidden layer neurons is dynamically adjusted with this method to minimize training errors. In the basic ELM, the number of neurons in the hidden layer does not change, and the output weights are calculated without iteration. The proposed method uses a technique similar to fuzzy logic control to iteratively update the hidden layer neuron numbers until the training error converges to zero. In addition, a data glove design and application that can produce fast and accurate results in the communication of deaf/mute people was developed. A total of 34 sensors for both hands were used in the design to increase the accuracy of the results. A total of 120 short and long words and their three-word sentences were classified using Fuzzy Logic Assisted Extreme Learning Machines (FLA-ELM), Convolutional Neural Networks (CNN), and classical Artificial Neural Network (ANN) techniques. A new end-point detection method has been developed to support sentence classification. Since the sensors were sewn from the inside, they did not prevent hand movements. In addition, the minicomputer system (NVIDIA Jetson Nano) that will process the collected data is pocket-sized, and all connections are wireless.

All studies can be classified as static or dynamic sign-based techniques. Static signs are simply movements made up of palm and finger movements combined. It consists of stationary hand and finger shapes with no time component. These approaches do not deal with hand and arm trajectories and usually express letters and numbers [1]. Image processing-based approaches or sensor data collected from data gloves are used to recognize these signs. Because of advancements in imaging technology, dynamic signal-based methods are preferred. In the dynamic sign-based approach, the data of hands’ starting and ending positions, hand trajectories, and finger movements are collected by cameras or sensors. These methods have been classified into two categories in the literature: image-based and sensor-based. In image-based techniques, single, stereo, and Kinect-style cameras [2] are employed. Colored gloves [3] and wristbands [4] were also utilized in some studies to improve the clarity of hand gestures. In sensor-based approaches, inertial sensors (Inertial Measurement Units-IMU), EMG, flex, electromagnetic, and radar-like sensors are used. Image-based techniques are classified into two types [5]: appearance-based [6] and model-based [7]. Feature extraction is one of the most important steps for improving appearance-based sign recognition performance. The image sequences’ features are extracted and compared to the referenced features. 2D or 3D models of hand gestures are created and compared with reference models in model-based methods. In [8], the authors developed a multi-scale contour-based 3D model for hand movements, including segmentation and sign recognition operations. Using correlation and mutual information methods for similarity measures in recognition procedures, they achieved success rates of up to 96.4 percent. In the study conducted in [9], transition-movement models (TMM) were used to determine the transitions between adjacent signs. After removing the transition parts, the signals were clustered using the improved k-means method for the remaining part, and a recognition success of 94.3% was obtained.

The most commonly used technique in sensor-based approaches is data gloves. In sign language recognition studies with data gloves, the twist and tilt angles of the fingers and the position and the trajectory of the hands during the movement should be determined. Unlike vision-based approaches, this technique allows for creating 3D motion modeling with depth information. In this technique, the signers must wear a glove with various sensors. Flex sensors are used to measure the twisting of the fingers and palm, while gyro meters and accelerometers are employed to detect hand and arms’ rotation and translation. In [10], the authors conducted a data glove study in which letters and numbers were attempted to be recognized in Pakistani sign language using flex, gyro, and accelerometers, with a success rate of 93.4 percent. In addition to vision-based and sensor-based research, studies have also been carried out to perform sign recognition by analyzing RF signals that hit the hands and return. 87.5 % accuracy was obtained in [11] using a single RF signal source. The Leap Motion Controller (LMC) plays a significant role in sensor-based techniques. The LMC is an optical and infrared sensor-equipped electronic device released by the Leap Motion Company in 2013. Human hand and finger motions can be measured at 200 fps using this equipment [12]. This device can generate temporal data, which is crucial for recognizing a hand gesture. In addition to sign language recognition, it is used in research fields including medical diagnostics, identification recognition [13], robotics, etc. [14]. When the literature is examined, it is discovered that LMC is widely employed in sign language recognition studies. It was employed with a meta-learning approach in [14] and ELM-based approaches in [15] for Turkish sign language recognition and with SVM in [16] for American sign language recognition.

There are two challenges in sign language recognition applications: environmental and methodical. The necessity of transporting the equipment (e.g., cameras, computer systems, and electronic equipment) to the installation location, extensive setup times, and measurement challenges are all environmental challenges in vision-based systems. Wearability and comfort problems are coming first in sensor-based systems. The most important methodological challenge in recognizing continuous hand gestures is that transitions between adjacent signs are difficult to detect due to their non-standard nature. In addition, hand gesture signals are not of the same size temporally, so they cannot be easily aligned with reference signals. They differ significantly from person to person temporally and spatially. This problem is known as end-point zone (EPZ) detection. When the literature is reviewed, it is discovered that much research is aimed at recognizing a single sign at a time; therefore, end-point localization is unnecessary [7,17,18, 7,17,18]. In some studies where EPZs were analyzed, a threshold value was calculated for repetitive signals, and signal segments below the threshold were accepted as end-points [19]. Models of repetitive transition movements were developed in [8], and end-point recognition was achieved by comparing them to real-time data. In [9], a transition movement model for transition movements was built and segmented using the k-means clustering algorithm.

The recognition phase follows the segmentation phase. Learning-based or vector-distance-based approaches can be applied to recognize static hand gestures. Classification of the collected data is performed using methods such as SVM [20], NN [6], ELM [14,15,21, 14,15,21], CNN [22], Recurrent Neural Networks (RNN) [23], and Euclidean Distance [24]. Dynamic hand gestures are more difficult to recognize because even the length of signal samples of the same word can vary from person to person. This challenge necessitates the use of time-domain alignment algorithms that are specifically designed for signal comparison purposes. Dynamic Time Warping (DTW) [25] and Hidden Markov Model (HMM) [26] are the most often utilized approaches for the time-domain alignment of dynamic gesture signals. Varying signal lengths in the temporal domain are also a problem for learning-based methods. Since equal-sized inputs are required when training neural networks, when the signal sizes differ, it is necessary to equalize the input sizes using techniques such as interpolation. Sentence recognition is also one of the challenging processes. End-points must be correctly identified in the transition from one word to the next for successful sentence recognition. Therefore, fewer studies have been conducted compared to studies on letter, number, and word recognition. The sections of this paper are as follows: In section 2, the structure of the data glove, data acquisition, and classification were discussed, and the results were analyzed in section 3.

2 Proposed methodology

The presented technique incorporates data gathering, feature extraction, and classification processes. These steps are described in detail in the subsections below.

Data Acquisition: It is the most critical step of the study. Because the sensor data’s sensitivity and accuracy directly affect the system performance. Data gloves have been designed to be portable, wearable, and installable to create a long-term solution. The NVIDIA Jetson Nano, which will perform the recognition processes, is packed so that it may be easily carried in the pocket. In order to improve the overall performance of the system, it is required to gather data that provides the most precise representation of hand gestures possible. We designed the data glove and mounted the sensors. Sensor assemblies were constructed by sewing instead of gluing. The ease of replacing the sensors was considered during the sewing process. Figure 1 depicts the sensors used for the right data glove and their locations.

OPEN IN VIEWER

Fig. 1

(a) Bottom view, (b) top view of right data glove.

As a result, the sensors to be employed in the glove and their placement are critical. Flex sensors were used to measure the amount of finger joint twisting. For each data glove, six were used. An additional flex sensor was used to detect if the thumb and index finger were adjacent. Hall sensors and magnets are used to detect signs touched on the palm with fingertips. Five hall sensors and one magnet were used for each data glove. The IMU (gyro and accelerometer) was employed to measure hand rotation angles and their trajectories during the movement. Each data glove has its own IMU sensor.

Data Transfer: All data collected by the sensors must be transferred to the central processing unit (CPU) as fast and precisely as possible. The first step is to send analog data to an Analog-Digital Converter (ADC) for conversion. Because of its serial connection capabilities and simplicity of use, the Arduino Pro Mini was used as an ADC. The hardware components and the data glove’s final state are shown in Fig. 2.

OPEN IN VIEWER

Fig. 2

(a) Initial data glove, (b) final data glove with the hardware box.

However, it does not have enough pins to capture 17 analog signals from one hand. A 16-channel MUX was used to address this problem, and all signals were converted to digital. The 2.4 GHz Bluetooth module is used to send digital inputs to the minicomputer NVIDIA Jetson Nano. The right glove data is sent first, followed by the left glove data during data transmission. A printed circuit board (PCB) is produced for the equipment and placed in a box that can be worn on the wrist. Additionally, the predicted words are read out using the minicomputer’s speaker.

Feature Extraction: The classifiers used in this study were based on CNN and ANN. As a result, separate feature extraction algorithms were applied for each approach. As previously stated, one of the most significant difficulties in researching sign language recognition is the variation in the duration of hand gestures across individuals. As a result of sampling the hand gesture signals, different-sized data arrays will be generated for each word. However, the input vectors for each word should be equal in size in the learning-based neural networks used for classification in this research. The procedures outlined below were used to equalize the inputs’ sizes used in training ANNs and CNNs.

1- The data obtained per word was initially normalized for ANN-based classifiers. In order to equalize the input sizes, all input vectors were set to 1×1360 for each word by performing cubic interpolation [27]. Hand gesture signals are mostly a curve. Therefore, the data size can be increased or decreased by finding the intermediate values in the data array by cubic interpolation (3rd-degree polynomial). If we consider the equation of the signal as y = a₀ + a₁t + a₂t² + a₃t³ at a given time interval, it is necessary to calculate the coefficients a₀, a₁, a₂ and a₃ to find the value of the signal at any time t _x . If the values previously obtained at t₁, t₂, t₃ and t₄ are y₁, y₂, y₃ and y₄, the polynomial coefficients can be calculated as shown in Equations 1 and 2. y 1 = a 0 + a 1 t 1 + a 2 t 1 2 + a 3 t 1 3 y 2 = a 0 + a 1 t 2 + a 2 t 2 2 + a 3 t 2 3 y 3 = a 0 + a 1 t 3 + a 2 t 3 2 + a 3 t 3 3 y 4 = a 0 + a 1 t 4 + a 2 t 4 2 + a 3 t 4 3(1) [ 1 t 1 t 1 2 t 1 3 1 t 2 t 2 2 t 2 3 1 t 3 t 3 2 t 3 3 1 t 4 t 4 2 t 4 3 ] [ a 0 a 1 a 2 a 3 ] = [ y 1 y 2 y 3 y 4 ](2)

The polynomial coefficients can be computed, and the intermediate value at the time t _x can be calculated by solving the equation. This guarantees that the hand gesture data sampled is always 34×40 (1×1360, reshaped).

When using CNN to classify words, the sensor data acquired for each word was converted into grayscale pixels, which were then used in the classification process. Although there are 34 columns of data in this set, the number of rows varies depending on how long the hand gesture takes. In the experiments using 150 words, it was determined that the average signal duration was 1.1 seconds, and the sensors could be sampled an average of 40 times during this period. For cases when the number of rows is different from 40, all of the data is scaled using bicubic interpolation [28] to a size of 40×34

2.1 End-point detection

It is essential to detect the transition between two adjacent signs in order to perform accurate classification in continuous sign language recognition. Therefore, a new approach for decomposing adjacent signs was developed in our research. At the end of a hand movement, the sensors’ signals remain stationary. A sudden change happens when the new word’s hand motion is initiated. According to our algorithm, the moment of this sudden change is the end-point of the first word. We computed the variance in successive sensor signals at transition points and referred to it as the variation index. We calculated a threshold value based on the average of the various indexes for all classified words as a result of the experiments. When the vectorial distance between consecutive sensor data from the right and left data gloves exceeds this threshold value, it is assumed that the first word’s hand movement has ended, i.e., the end-point has been detected. The following word’s sensor data starts to gather after this procedure. The calculation of the variation index is shown in Equations 3–6. F t L = [ Flex 1 , … Flex 6 , IMU 1 , IMU 2 … IMU 6 ](3) F t = [ F t L , F t R ](4) F t - 1 = [ F t - 1 L , F t - 1 R ](5) d t = ∑ | F t - F t - 1 |(6)

F t L and F t R denote instant flex and IMU sensor data from the left and right gloves, respectively, F t - 1 L and F t - 1 R denote previous sensor data from the left and right gloves, and d _t is the variation index. If d _t  > threshold, means that the end-point has been recognized and the hand movement is complete.

In this situation, all of the data obtained for a word is transferred as word data to the classification phase, and data gathering for the new word begins. Figure 3 depicts the image-transformed 3-word hand gestures data gathered from both hands’ sensors. The figure shows end-points, EPZs, single-word and three-word sentence data.

OPEN IN VIEWER

Fig. 3

Image-transformed data of hand gestures of a 3-word sentence.

2.2 FLA – ELM-based classifications

ELM is a machine learning method based on a feedforward neural network [29]. It consists of three layers: an input layer, a hidden layer, and an output layer. The ELM architecture used in the study is given in Fig. 4. ELM has the advantages of fast learning speed and better generalization performance over the traditional single hidden-layer feedforward neural network [30]. As a result, the ELM method has been widely applied in various fields. There is no need to adjust the training process because ELM generates the connection weights and neuron thresholds of the input layer and hidden layer randomly. The best solution can only be obtained by adjusting the number of neurons in the hidden layer [31]. It is a high-speed method because iteratively gradient-based backpropagation is not necessary. The Moore-Penrose generalized inverse is used to calculate its weights.

OPEN IN VIEWER

Fig. 4

Designed FLA-ELM architecture.

In the above architecture, where L is the number of hidden layer neurons (3000 were used in the initial state), g () is the activation function (radial basis function was used in this study), w is the weight of the input layer, Word denotes the actual output of the network, Δ denotes the number of dynamically adjusted neurons, and b is the bias: Word i = ∑ 1 L β i g ( w i . I i + b i )(7)

In short, we can write H . β = Word matrix. In this case, matrix H is the output matrix of the hidden layer, and it can be arranged as in Equation 8. H = [ g ( w 1 . I 1 + b 1 ) ⋯ g ( w L . I 1 + b L ) ⋮ ⋱ ⋮ g ( w 1 . I 1360 + b 1 ) ⋯ g ( w L . I 1360 + b L ) ](8)

In the ELM algorithm, H is determined uniquely by the input weight and the hidden layer thresholds, chosen randomly before training and kept constant throughout. Thus, to solve the output weights given by Equation 9, the training of the ELM network can be converted into a linear system optimization problem. The output weights β ˆ can be calculated as shown in Equation 10 [31]. min β ∥ H β - Word ∥(9)

Finally, β matrix can be calculated as shown in Equation 10. β ˆ = H + Word T(10)

The Moore-Penrose generalized inverse matrix of the output matrix H is denoted by H⁺ and T denotes the transpose of a matrix. To obtain the best solution using the ELM method, simply change the number of neurons in the hidden layer. However, random assignment of input weights occasionally results in deviations in β weights, reducing generalization ability. To avoid this situation, FLA-ELM was proposed in this study. In the FLA-ELM method, first, the number of initial neurons (L), the number of neurons to be added or removed (Δ), and the acceptable error range (ER) are assigned. In the basic ELM method, calculating the β weights only once are sufficient. In FLA-ELM, the number of neurons in the hidden layer is dynamically changed. The number of neurons is adjusted based on the error (ɛ) and the derivative of the error (dɛ) value. The error is the absolute sum of the differences between the desired ( Word i , j D ) and actual ( Word i , j A ) network outputs, as shown in Equation 11–12.

ɛ T = [ Word 1 , 1 D ⋯ Word 1 , N D ⋮ ⋱ ⋮ Word 120 , 1 D ⋯ Word 120 , N D ] - [ Word 1 , 1 A ⋯ Word 1 , N A ⋮ ⋱ ⋮ Word 120 , 1 A ⋯ Word 120 , N A ](11) ɛ = | ∑ i = 1 120 ∑ j = 1 N ɛ T |(12) ER = ɛ low ⩽ ɛ < ɛ high(13) where ɛ _low is the low threshold, and ɛ _high is the high threshold of the range. A fuzzy logic inference system (FIS) with inputs ɛ and dɛ and output Δ was created to calculate Δ value. Figure 5 depicts the configuration of the fuzzy sets used.

OPEN IN VIEWER

Fig. 5

Fuzzy sets configuration.

In this system, For ɛ and Δ, 5 Gaussian membership functions were used, and 3 for dɛ. For implication and aggregation, prod() and sum() functions were selected, respectively. Two of the 25 rules prepared to generate the output are shown below as examples.

IF ɛ is NH and dɛ is PH, THEN Δ is Z

IF ɛ is PL and dɛ is NH, THEN Δ is NL

In Fig. 4 it is represented by the cluster labels NH: Negative High, NL: Negative Low, Z: Zero, PL: Positive Low, and PH: Positive High. To reduce ɛ, this method iteratively repeats the β weight calculation by adding or removing Δ neurons from the hidden layer. When the ɛ falls within the ER range, the optimization is over. The FIS system was integrated into the main system using a lookup table. The simplified algorithm of the method is given in Fig. 6.

OPEN IN VIEWER

Fig. 6

The algorithm used to dynamically change the sizes of the hidden layer.

3 Experimental results

In this study, FLA-ELM, ANN, and CNN networks were used to classify 120 of the most often-used words in Turkish sign language. Fifty signers were divided into two groups of 25 to gather sign samples, and each group member was requested to gesture 60 different words. As a result of sampling 25 samples from 120 different words, a total of 3000 sample data was obtained. 80% of the samples were used for training ANN and CNN, 10% for validation, and 10% for testing. The selected 120 words are divided into two subgroups short-term signs and long-term signs. For example, since the hand movement time of the word “you” is about 0.2 seconds, it is included in the group of short-term signs. However, since the hand movement time of the word “friend” is about 0.9 seconds, it is included in the group of long-term words. The threshold value between long and short-term signals was determined as 0.5 s. Thus, short and long-term signals could be evaluated separately in performance tests. As mentioned before, the word data of short and long-term signals are fixed to 34×40 dimensions. Therefore, in terms of data size, the signs in both groups are equal to each other. Hand movement times were obtained by averaging the signers’ timings. 3-word sentences are groups of words selected from a sample pool of 120 words, ordered consecutively without any meaning. They were categorized as short and long sentences. Short sentences consist of three short words and long sentences consist of three long words. They were used to measure the ability of the proposed method for consecutive recognition and end-point detection. Some sample words and sentences are given in Table 1.

3-word sentences were classified after they were converted to three single words as a result of end-point detection. Therefore, no particular classification model for three-word sentences has been developed. The primary problem here is detecting the end points accurately and capturing the consecutive word. Some experimental studies have been conducted to observe whether the hand gesture signals produced by different signers for the same words can be generalized. The input’s power spectral densities (PSDs) were computed in these experiments to observe how the hand gesture signals for identical words varied with different signers. PSD indicates the frequency ranges in which time-series signals are more prominent than at other frequencies. Figure 7 shows the PSD graphics of the words “Go”, “Nice”, “Home”, “Father”, “Mother” and “Friend” performed by three different signers.

OPEN IN VIEWER

Fig. 7

PSD graphics of the words “Go”, “Nice”, “Home”, “Father”, “Mother” and “Friend”.

The graphs show that the signals have similar characteristics. As a result, the constructed FLA-ELM, CNN, and ANN networks were able to generalize the input data reliably.

This study aims to develop a portable data glove that translates Turkish sign language to voice for hearing and/or speech-impaired people. Sensor signals from the data glove were transformed into digital data (word data) and classified concurrently with hand movements. A total of 120 words and sentences consisting of these words were successfully classified using ANN and CNN methods. In experimental studies, 120 words were divided into short and long-term words. In order to assess the system’s continuity, sentences including three consecutive short/long words were classified, and performance measures were taken. A Euclidean distance-based approach has been developed for end-point detection, which is the most challenging phase in continuous gesture recognition. When the literature is examined, it is discovered that most studies with data gloves use letter and number recognition, with only a few studies using word recognition. Table 4 shows the characteristics of the studies examined.

When we examine the accuracies in our study, it has been observed that although CNN is more accurate, it consumes roughly 5–9 percent more time than ANN. According to findings, the conversion of each word data into images increases the processing time in CNN. This increase, however, is within acceptable limits. The CNN approach achieved a 97 percent accuracy rate for classifications of single words. The processing times were quite close since the short and long-term words were interpolated to the same dimensions. It is noticed that the accuracy of continuous gesture recognition reduces due to end-point detection. Because the transitions between the two hand movements vary significantly amongst individuals, it has been observed that neural networks have less success generalizing these transitions. Although ANNs and CNNs quickly learned hand gestures with certain trajectories, the learning rate was reduced because the trajectories of the transitions between the two signs were unclear and varied significantly across individuals. While there was no significant difference in processing times between short and long words, there was a comparable difference between short and long sentences. The most important limitations experienced in this study are the training of signers who do not know Turkish Sign Language in data collection and the use of extra filters to clean the noise signals from the sensors compressed into a narrow space. In the continuation of this study, neural network-based methods will be developed for end-point detection, and improvements will be made for the more accurate translation of longer sentences.