PARLOMA – A Novel Human-Robot Interaction System for Deaf-Blind Remote Communication

2.1 Hand Tracking

Object tracking techniques can be classified into two main classes: either invasive approaches, based on tools which are physically linked to the object (sensitized gloves [12] or markers [13]), or non-invasive approaches. The former are usually fast and computationally light, but often very expensive and cumbersome. The latter require more computational resources, but are generally based on low-cost technologies, and moreover do not require a physical link to the object to be tracked. As such, the object is free to move and not entangled.

Non-invasive approaches can be classified according to the kind of input data they need (2D, 3D) and the output that they provide [14]. Obviously, as real-world life is embedded in a 3D universe, the best performances are obtained when 3D feature characterization is performed [15], since 3D information is usually both more informative and more robust [16]. Moreover, low-cost acquisition systems, such as RGB-D cameras, represent a powerful tool for projects that aim at developing cheap and affordable solutions.

Non-invasive approaches are classified into partial tracking and full tracking. Tracking is defined as partial when it deals with only some insights of the kinematics of the hand (e.g., fingertip positions, discarding the rest of the hand), while in full approaches the whole hand is tracked. Of course, full tracking approaches are more useful for HRI applications, but they require much greater computational resources [17]. Full tracking approaches can be further divided into model-based and appearance-based approaches.

Model-based approaches [18] formulate the pose estimation as an optimization problem that minimizes the discrepancy between 3D hand hypotheses and the actual observations [19]. This problem is usually solved using optimization algorithms. In [8], the authors adopt a 3D hand model consisting of a set of assembled geometric primitives, animated by a vector of 27 parameters. A stochastic optimization method, called “Particle Swarm Optimization” (PSO) [20], is used to estimate the 27 model parameters' values that minimize the distance between the model itself and the input.

Appearance-based approaches employ discriminative techniques based on feature extraction from input data. At run-time algorithm extracts features from the input and tries to map them to a known hand pose. Appearance-based approaches are often implemented using database-retrieval [21] or machine learning [22] techniques. Learning how to map features and hand poses is the most computational intensive phase. Since this task is performed offline, while only fast feature extraction is required at run-time, appearance-based approaches easily achieve real-time performances. The accuracy of these algorithms is strongly related to the kinds of features and to the quality of the training set (or database), particularly to its variety and capacity to cover the whole set of poses.

2.2 Gesture Recognition

Currently, HGR is one of the main important research areas of computer science. Researchers are investigating different approaches by using Machine Learning, Neural Networks, Fuzzy Systems, Evolutionary Computation or Probabilistic Reasoning. Surveys on HGR algorithms are given in [23, 2, 24]. The last two compare performances of various systems which have already been applied to human-machine interaction.

The accuracy of the system presented in [25] is about 90%, but it strongly depends upon the lighting conditions of the working environment. The average accuracy of the system presented in [26] is 80.77%, but this system only discriminates among simple gestures (e.g., showing a direction). [27] presents a system which is 90–95% accurate in recognizing open fingers while the success rate for closed fingers drops to 10–20% only. [28] claims around 90.45% accuracy, through hidden fingers could not be detected using this approach. The authors in [29] achieved 96% accuracy over 300 tests, but they developed an algorithm which does not meet the real-time constraint. The system presented in [30] achieves an accuracy rate of more than 95%, but it only works on six classes of gestures and it can only recognize bended fingers, no matters the degree of bending.

A remarkable work is presented in [31]. This paper focuses on building a robust part-based hand gesture recognition system using a Kinect sensor. The authors propose a method to handle the noisy hand shapes obtained from the depth camera. They propose a novel distance metrics, namely the Finger-Earth Mover's Distance, to measure the dissimilarity between hand shapes. Experimental results demonstrate that such a HGR system is quite accurate (a 93.2% mean accuracy is reported), but it works only on a 10-gesture dataset and it is quite slow (it achieves a 13 fps operating frequency). In [32], the authors only use RGB information, thus achieving different results in normal, brighter and darker conditions. The average accuracy of the system is in any case higher than 90% on discriminating hand poses for the 10 one-digit numbers (from 0 to 9). The operating frequency of the system is not reported.

2.3 Anthropomorphic Haptic Interfaces

Haptic devices elicit human perception through the sense of touch. Haptics therefore extends the communication channels for human-machine interaction in addition to the typical senses of vision and hearing.

Haptics includes wearable devices, such as gloves, and robotic devices, such as robotic arms and hands. With respect to robotic hands, despite the significant progress in recent decades in electronic integrated circuits and in applied computer science, challenges remain in increasing dexterity, robustness and efficiency, as well as in matching cost constraints [33]. Examples of dexterous robotic hands for humanoid robots are the Awiwi [34] and the Shadow hand [35].

Robotic hands designed to enable deaf-blind communication have already been proposed in the literature. The first attempt at creating a finger-spelling hand was patented in 1978 by the Southwest Research Institute (SWRI) [36]. Later, the Dexter hand was developed [37]. Dexter improved over the hand built by SWRI, but was extremely bulky and required compressed air to drive the pneumatic actuators. The whole hand had seven pneumatic actuators. Each finger was actuated by a single pneumatic actuator with a linear spring to provide some resistance and return. Both the thumb and the index finger had a second pneumatic actuator to perform complex letters.

The most successful design seems to be RALPH [38]. This hand was built in 1994 by the Rehabilitation Research and Development department of Veterans Affairs. RALPH fixed many of the problems of the Dexter hands, but it was still not robust or attractive. RALPH was only half a hand, as it only had fingers but no forearm and no wrist, which made it hard to read since it is in an unnatural position for the reader. In addition, it could perform only a limited subset of signs.

3.1 Input Module

The Input Module is in charge of extrapolating 3D information from the depth map in order to understand the gesture performed by the user in real-time. It comprises three ROS nodes, namely the Depth Camera Driver node, the Hand Tracker node and the Hand-shape Classifier node.

The Depth Camera Driver node exposes the depth image stream from the depth camera as an ROS topic. In the proposed implementation, we use the OpenNI ROS driver for the Asus Xtion sensor [39].

3.1.1 Hand Tracker

The Hand Tracker node is a modified implementation of the algorithm proposed in [6], where the authors propose a full-DoF appearance-based hand tracking approach that uses a Random Forest (RF) classifier [40].

This node extracts the hand skeleton and publishes the 3D position of each joint of the hand with respect to the camera reference frame. It accomplishes three main tasks performed sequentially on each incoming frame, as shown in Figure 2. In detail, the first task is accomplished by the Hand Segmenter block, which aims to distinguish the hand from the background.

OPEN IN VIEWER

Figure 2.

The Hand Tracker node. This node computes the hand joints' positions from depth images. It consists of three sequential tasks. The first one segments the hand (foreground) from the background. The second one classifies each pixel of the foreground to the hand region that it should belong to. The last one extrapolates the regions' centroids corresponding to the joints' positions.

Next, the Hand Labeller block executes an appearance-based approach to recognize different parts of the hand in order to isolate its joints. The Joints Position Estimator block approximates the joints' 3D positions, starting from the input depth measurements and the outcome of the first block. As in [40], in our approach an RF classifier [41] is used to label pixels of the depth image according to the region of the hand they belong to. Successively, each region is processed to find the position of its centre. At the end of the clustering process, the algorithm outputs the 3D position of each joint of the hand.

In our approach, we perform a per-pixel classification, where each pixel x of the hand is described using the following feature:

F ( x ) = { F u , v ( I , x ) , ‖ u ‖ < R , ‖ v ‖ < R } ,

(1)

where I(·) represents the depth value of the image at a given point, u,v are two offsets limited to a finite R length, and the function F_u,v (I,x) is defined as:

F u , v ( I , x ) = I ( x + u I ( x ) ) − I ( x + v I ( x ) ) .

This feature succeeds very quickly in discriminating hand parts [6]. Hand joints can be estimated by labelled segmented depth-maps using the Mean Shift (MS) algorithm [42]. In addition, the MS local mode-finding algorithm (as in [22]) reduces the risk of outliers, which might have a significant effect on the computation of the joints. By implementing MS, we obtain a more reliable and coherent estimation of the joints set S.

Our labelling algorithm can recognize 22 different parts of the hand, namely the palm, the wrist and four joints for each of the fingers.

The joints' positions are approximated applying the MS clustering algorithm on the hand sub-parts. This approach shows promising results: experiments with real-world depth map images demonstrate that it can properly label most parts of the hand in real-time without requiring excessive computational resources.

Note that (1) is not invariant to rotations, while on the other hand it is invariant to distance and 3D translations (thanks to the normalization factor I(x)). As such, it is necessary to build a wide training set containing many instances of the same gesture captured from different points of view, according to [43]. For this reason, we have also investigated ways to effectively and automatically build comprehensive large training sets. Since manually building a dataset is a time-consuming and error-prone process, we also developed a tool that is able to create a synthetic training set. Such a system is based on the 3D model of a human hand shown in Figure 3. Essentially, the tool is able to generate intermediate hand-shapes from a small set of hand-shapes defined by the user. Next, all the hand-shapes are used to build the synthetic training using a model of the depth camera.

OPEN IN VIEWER

Figure 3.

3D model used to generate the synthetic training set. Our solution is able to generate intermediate hand-shapes from a small set of user-defined hand-shapes, in order to obtain a very large training set.

Table 1 summarizes the main learning parameters describing the RF used in our implementation. The proposed values have been experimentally evaluated. Each tree is trained with 2,000 pixels randomly sampled from each training image. Offset vectors u and v from (1) are sampled uniformly within a radius of 30 pixels.

OPEN IN VIEWER

Table 1.

The optimal values that we propose to train the RF classifier

Parameter	Value
R	30 pixel
Threshold	10
Sample pixels per image	2000
Trees in the forest	3
Depth of each tree	18

3.1.2 Hand-shape Classifier

The Hand-shape Classifier node is devoted to classify the current hand-shape according to the gesture that the user is making. This second classification is not performed directly on the array of joints' positions, but rather on a pattern containing the joint-to-joint Euclidean distances for all pairs of joints of the hand:

P ( S ) = { d k , l = ‖ j k − j l ‖ , ∀ j k , j l ∈ S , k < l } ,

(2)

where j _k/j describes generic joints of the hand skeleton S, while k and l are generic indexes within S. The proposed pattern ensures invariance to rotation and translation in space.

Given P ( S ) , another RF is used to evaluate the probability for a pose to actually reproduce one of the accepted gestures (i.e., hand-shapes shown in 3). The gesture is valid if the posterior probability associated with that hand-shape is above a p_th threshold and if it is recognized over N consecutive frames. In our experiments, p_th = 0.3 and N = 5.

Finally, the hand-shape is encapsulated in a network message that is sent to the reproduction device. Secure and lossless remote communication are guaranteed by the ROS (secure socket layer SSL).

3.2 Reproduction Module

The Reproduction Module is in charge of reproducing the recognized hand-shape by using the haptic interface, so that a t-SL-proficient receiver can understand the transmitted messages. This module is composed by two main ROS nodes, namely the Sign Converter node and the Hand Driver node. In the developed solution, this module runs on a low-cost computer (the Raspberry Pi). This architecture allows the system to have different output devices connected, enabling one-to-many communication among users.

3.3 Haptic Interface

The Haptic Interface used is a right-handed version of the CyberHand [44]. It consists of four under-actuated anthropomorphic fingers and a thumb, and it is actuated by six DC motors. Five of them, located remotely, control finger flexion/extension. One motor, housed inside the palm, drives the thumb abduction/adduction. The hand is comparable in size to the adult human hand, and the remote actuators are assembled in an experimental platform which mimics the shape of the human forearm. The remote actuators act on their respective fingers using tendons and a Bowden cable transmission. Active flexion is achieved as follows: when a tendon is pulled, the phalanxes flex synchronously, replicating the idle motion (i.e., free space motion) of a human finger. When a tendon is released, torsion springs located within the joints extend the fingers. The hand includes encoders on each motor and an electronic controller that implements position control by receiving commands sent over a serial bus.

4.1 Input Module Evaluation

The tests evaluate and quantify the system performance in recognizing hand-shapes (the first step of the pipeline). Sixteen subjects were recruited (10 men, six women, mean age 25 years, range ± four years). None of the subjects had any expertise with LIS. Indeed, the subjects were chosen in order to check the accuracy of the system even in the case of potential beginner users of the final product. The experiment consists in the production of isolated hand-shapes corresponding to the manual alphabet letters. Subjects were sat at a table in front of a laptop (a Macbook PRO, late 2011, mounting an Intel Core i7@2.7GHz CPU, 4GB of RAM, an Intel HD Graphics 3000 512MB GPU) and a depth camera (Asus Xtion PRO), and had to repeat each of the hand-shapes appearing on the laptop's monitor (as shown in Figure 5). Each subject was 50–55 cm away from the camera lens. For each pose, the system recorded 100 depth maps (at a rate of 30 fps). This part of the experiment lasted about three minutes per person (six or seven seconds per hand-shape per person). In total, 40 thousand depth maps were collected. Each pose was then linked with the corresponding LIS alphabet letter.

OPEN IN VIEWER

Figure 5.

Experimental setup. On the left, a signer performing hand-shapes in front of the RGB-D Camera. Information is elaborated and sent over the net to a robot hand. On the right, the receiver is able to understand the sign performed by the robot hand using tactile sensing.

With respect to the 40 thousand depth maps acquired, we performed 10 leave-one-out cross-validations to investigate the accuracy of the Input Module. For each validation, depth maps from 15 subjects were used for training the RF classifier from the Hand-shape Classifier node (the training procedure was similar to that in [43]), while data from one random subject are used for testing it. The results are summarized in Figure 6, which shows that the average accuracy of the Input Module system with respect to the ground truth set is 74%, with a confidence interval of ±2%.

OPEN IN VIEWER

Figure 6.

Confusion matrix of the Input Module. On the Y axis, the ground truth, while on the X axis are the gestures recognized by the module. Similar hand-shapes are presented in adjacent cells.

It should be noticed that noisy data from the sensor are the main source of errors in the case of the gesture “P”, which on 75% of occasions was recognized as “A”. “P” hand-shapes differ from the latter only because the index is pointing at the camera (instead of being closed near the palm, see Figure 4); however, very often noise totally covers the index finger in such a configuration, such that is impossible for our algorithms to discriminate between these two gestures. Different is the case of the gesture “K”, which on 39% of occasions was recognized as “P” and 17% of the times as “L”. These errors are due to the intrinsic properties of (2), which cannot discriminate enough different poses in which the relative joint distances do not vary. In fact, note that “K” seems like a “P” that is rotated clockwise by 90 degrees, and an “L” in which the thumb and middle-finger bending are exchanged.

The average accuracy obtained by our system is comparable with [46], especially considering that we track the human hand frame by frame. Our algorithms do not require extra computation to estimate the arm or even the full body pose. However, other state-of-the-art approaches perform slightly better (e.g., [47, 48]). This is typically due to our per-pixel classification (within the Hand Tracker node), which is not robust enough yet, and to the depth camera that we use, which produces very noisy data. In any case, our experiments confirm that the average accuracy achieved by our approach is sufficient to effectively track the hand, even with respect to challenging backgrounds, and to discriminate among valid hand-shapes for SL-based interactions (even if they are similar).

4.2 Transmission Evaluation

Another test section involved the entire pipeline. Seven subjects with some expertise in LIS but no previous knowledge about the project participated in the test (four men, three women, mean age 30 years, range ± four years) together with an LIS signer (female, 24 years old). Since the status of the project is still in its preliminary stages, the test was not assessed by deaf-blind signers.

Each experiment was performed as hereby described and depicted in Figure 5. The results along the whole pipeline are illustrated in Figure 7. A list of 125 hand-shapes (LoS) to be reproduced (up to 10 repetitions for each hand-shape) was randomly generated at run-time. The LoS appeared on a monitor close to the LIS expert signer. She reproduced with the right hand each hand-shape in front of the depth camera. Also in this case, signs were made from a distance of 50–55 cm from the camera lens. To simulate a real-use case scenario, the LIS expert was placed in unfavourable lighting conditions, and people were allowed to pass behind her. Recognized hand-shapes were saved for successive processing in a dedicated list (SRI).

OPEN IN VIEWER

Figure 7.

Sign recognition and reproduction efficiency along the pipeline of the experimental apparatus

In a different room in the same building (so that remote communication via the Internet could be simulated), a Raspberry Pi received the ROS messages, decoding them and generating the commands needed to control the hand. The task for the subjects (who were in this other room), was to visually recognize the hand-shape actually performed by the robotic hand. Answers were recorded and saved in a separate list (SRS). The subjects assessed the task individually. The hand reproduced each hand-shape for five seconds. After this period of time, it returned to the rest position (with all the fingers opened) before the next sign was reproduced.

During the experiments, all possible feedback was collected and stored: the list of hand-shapes to be reproduced and the list of signs recognized by the Input Module and by the volunteer, depth maps from the camera, data from the network, and feedback about the joint positions of the robotic hand. Network latency during all the experiments was, on average, below half a second, and was never higher than one second.

The experiment lasted for approximately 12 minutes per person. At the end, the experimenters asked the volunteers for their comments, especially regarding any major difficulties that they experienced and their opinions on the usability of the hand for their purposes.

The collected results (summarized in Figure 7) demonstrate the feasibility of our system. In total, 875 hand-shapes were performed by the hand (125 hand-shapes per seven volunteers). The average accuracy of the acquisition module was 88.14%; however, few errors reported for the Input Module were actually caused by the expert in LIS performing the wrong hand-shape (e.g., performing gesture “A” while the experiment was asking for “I”). No errors where registered for the transmission and conversion systems. On 82.78% of occasions, the signs were correctly recognized by the volunteers and the total accuracy of the system in transmission was 73.32%.

In Figure 7, Recognition Efficiency refers to the percentage of hand-shapes correctly recognized by the Input Module. This comparison was to evaluate the effectiveness of the recognition module. Here, errors were due to classification errors, noisy data from the depth camera and finger occlusions occasionally deceiving the recognition algorithm.

Reproduction Efficiency refers to the percentage of hand-shapes correctly recognized by the subjects. Not surprisingly, we found that most of the time the volunteers simply confused a hand-shape for a similar one. As a consequence of the constrained number of degrees of freedom of the robotic hand in comparison to the human hand, many subjects reported minor difficulties in discriminating similar signs when different degrees bending would have been essential for the signs' discrimination. The most frequent recognition errors were the following:

Finally, Transmission Efficiency measures the efficiency of the whole experimental apparatus. The accuracy of the entire pipeline from the hand-shape produced by the signer to the visual recognition by the subjects is significantly high (χ²=215.764, df =1, p<0.001).

4.3 Reproduction Module Evaluation

A deaf-blind signer member of the Lega del Filo d'Oro, Francesco Ardizzino (male, 64 years old) volunteered to test the robot hands during the design process. His experience was then reported to the association panel and the PARLOMA project received official approval from the association.

In order to test the Reproduction Module, we checked whether a LIS-t deaf-blind signer was able to: i) recognize isolated hand-shapes produced by an anthropomorphic robotic hand via tactile exploration (i.e., the natural setting for t-SL communication); ii) to read-off a finger-spelled word from a sequence of hand-shapes.

An Italian LIS-t interpreter was present during the entire phase of testing (two hours and 30 minutes, including some breaks between stages and major tasks). The interpreter explained each task to the subject, who gave consent to participating in the experiment and authorized us to record the whole section.

Before assessing the accuracy in recognizing hand-shapes from the robotic hand, we made sure that the signer was able to assess the same tasks when performed by a human as a baseline. As such, we asked the interpreter to produce a few hand-shapes in isolation and then a few finger-spelled words. All of them were recognized by our subject.

The whole section is structured as a sequence of stages. The first is concerned with isolated digit and hand-shape recognition. The hand performs a set of poses (hand-shapes) commanded by an experimenter using an Acer Aspire 5810TZ laptop running a custom graphics interface program. Since, in this case, we are not interested in testing the entire pipeline, we ensured that the robotic hand correctly performed the hand-shape. The subject was asked to explore the robotic hand and then report which sign he comprehended. Tactile exploration is very quick and, especially in the first stages, it affected all parts of the hand. This is somehow different from what normally happens during human t-SL communication, where the exploration is more holistic. We attribute this to the fact that the subject was experiencing a robotic hand for the first time.

The first task was to recognize digits from one to five (including some variants, such as “1” made by extending the index finger or the thumb). The task was easily assessed and the digits were recognized appropriately. In this phase, we noticed that the subject was not actually counting the finger but only assessing the global hand shape of the hand, confirming that part-by-part exploration is related to the novelty of the interaction. The only problematic digit to recognize was “5”. In total, the volunteer was asked to recognize eight digits, and he failed only once (success rate 88%).

The second task involved the recognition of hand-shapes corresponding to the letters of the manual alphabet. All the letters were presented randomly. Only the letters “G”, “J”, “Z” and “R” were excluded, the first three because they require dynamic gestures, and the last one because it involves the crossing of the index and middle fingers (an option not available to the hand). When needed, the hand was manually oriented, such as in the case of the letters “M”, “N” or “Q”. The subject recognized most of the hand-shapes very quickly. The most problematic hand-shapes were “Q”, which was often mistaken for “P”; “E” and “O” were often mistaken for “A”. Following each letter, the LIS interpreter would tell the user whether his answer corresponded to the real hand-shape. In total, the subject was asked to recognize 19 digits, and he failed twice (success rate 89%).

The second stage concerned finger-spelled words. For this task, we asked the subject to recognize sequences of hand-shapes corresponding to the letters of the manual alphabet, and retained them until he could read-off an Italian word. We were interested not only in “simple” words, made of letters that are easily recognized (e.g., C-A-N-I = “dogs”, S-L-I-T-T-A = “sledge”), but especially in more complex words containing problematic letters not easily recognized in the previous step (e.g., A-C-Q-U-A = “water”, A-I-U-O-L-E = “flowerbeds”). As soon as the subject was able to read-off the Italian word, he was asked to produce the corresponding LIS sign. No errors were registered in this final experiment, showing that the integration with context improves a correct hand-shape recognition.