Teleoperation of an Anthropomorphic Robot Hand with a Metamorphic Palm and Tunable-Stiffness Soft Fingers

Abstract

Teleoperation in soft robotics can endow soft robots with the ability to perform complex tasks through human–robot interaction. In this study, we propose a teleoperated anthropomorphic soft robot hand with variable degrees of freedom (DOFs) and a metamorphic palm. The soft robot hand consists of four pneumatic-actuated fingers, which can be heated to tune stiffness. A metamorphic mechanism was actuated to morph the hand palm by servo motors. The human fingers' DOF, gesture, and muscle stiffness were collected and mapped to the soft robotic hand through the sensory feedback from surface electromyography devices on the jib. The results show that the proposed soft robot hand can generate a variety of anthropomorphic configurations and can be remotely controlled to perform complex tasks such as primitively operating the cell phone and placing the building blocks. We also show that the soft hand can grasp a target through the slit by varying the DOFs and stiffness in a trail.

Introduction

In robotic manipulation, the space of possible scenarios is expansive, and the objects of interest are quite diverse. To address this issue, one promising solution is to implement the human-robot collaborative methodology to control the robotic manipulator. Specifically, human cognitive abilities can be integrated through teleoperation to effectively manage difficult scenarios that may prove challenging for fully autonomous operation. Compared to intelligent programming, the use of human-robot teleoperation offers superior speed of decision-making and capability for dealing with complex operational issues.

The use of capturing human hand movements to control robot hands has great potential for intuitive control in numerous applications.^1,2 However, this approach presents challenges when it comes to soft robot hands, which possess different materials and kinematics compared to their human counterparts.³ Despite these challenges, previous research has demonstrated the versatility, robustness, and cost-effectiveness of soft robotic hands.^4,5 To overcome these kinematic differences, researchers have proposed various mappings that require the pose of the teleoperator's hand as input. These mappings require the pose of the teleoperator's hand as input and include fingertip,⁶ joint,⁷ and pose mapping,⁸ as well as teleportation subspace mapping.^9–11 Typically, the required information is obtained through instrumented data gloves or vision. Although data gloves and vision are robust, they are limited to mapping hand gestures and poses and do not convey information related to strength and stiffness, which can be important for certain applications.

Electromyography (EMG) is an experimental technique that involves the development, recording, and analysis of myoelectric signals. EMG arises from physiological variations in the state of muscle fiber membranes. Depending on the placement of the sensing device, the measured EMG can be classified into intramuscular EMG and surface EMG (sEMG).¹² sEMG is acquired through the sensing device placed on the skin surface, and it provides information on the combined sEMG of muscle groups. Due to its ease of acquisition and rich information characteristics, sEMG has been widely used in myoelectric prosthesis hand control research, such as gesture recognition^13,14 and force estimation.^15,16 Hu et al.¹⁷ proposed a novel myoelectric control scheme by utilizing multitask learning technique and postprocessing algorithm. The multitask learning technique was used to simultaneously predict grasp gestures and instantaneous grasp forces, while the postprocessing algorithm aimed to overcome the negative effects of force variations on gesture recognition accuracy.

Baldacchino et al.¹⁸ introduced a Bayesian mixture expert model within the framework of sEMG-based intention recognition. This approach naturally incorporates the uncertainty of sEMG signals into the model structure, enabling simultaneous recognition of gestures and fingertip forces. Zhang et al.¹⁹ proposed a deep learning model that achieved high-precision estimation of human motion and interaction forces simultaneously. Li et al.²⁰ introduced a novel framework that combines hand motion classification and continuous grasp force estimation based on a new spectral feature of EMG signals. The effectiveness and superiority of this method were validated on both able-bodied and amputee subjects.

The widespread use of sEMG has enabled force information to be applied in manipulator teleoperation. However, it should be noted that variable stiffness technology can improve grasping performance and increase the degrees of freedom (DOFs) of the soft hand. To our knowledge, there are no studies on intention decoding based on sEMG implementation of hand stiffness changes. Furthermore, there are few reports on the control of soft robot hand stiffness by hand mapping.

In this study, we propose an anthropomorphic soft robot hand with variable DOFs soft fingers and metamorphic palms as shown in Figure 1A. For the sake of clarity, the main contributions of this article are:

FIG. 1.

An anthropomorphic soft robot hand that maps the gesture, stiffness, and DOF of the human hand. (A) The soft robot hand consists of four segmented heated pneumatic fingers and a metamorphic palm. (B) The inserted panels show that the actuator can support a weight of 100 g at the rigid state (25°C) without bending, but it can be bent by the gravity of the weight at the soft state (50°C). The inferior panels show the details of the multilayer bottom portion. This part has three distinct layers: the VEL, the Ni–Cr wire, and the silicon rubber. (C) Soft fingers can map the stiffness of the human fingers. (D) Soft fingers can map two major joints (MCP, PIP). DOF, degree of freedom; MCP, metacarpophalangeal, PIP, proximal interphalangeal point; VEL, variable effective length.

A novel design for a soft robot hand consists of four segmented heated pneumatic fingers and a metamorphic palm (Fig. 1A) that allows the hand to transform and optimally utilize its actuation degrees to breakthrough its actuation limitations.

Proposes a feasible motion mapping method to enhance recognition of the stiffness and the load of human hand in the teleoperation of the manipulator. (Fig. 1C, D).

Verify the effect of palm configuration on typical grasping tasks.

This article proposes a feasible motion mapping method to improve the recognition of stiffness and load of human hand during teleoperation of the manipulator. The multisensor sensing system (MSS) consisting of low-cost commercial sEMG and data gloves is utilized to map the information related to the DOF, gesture, and stiffness of the human hand to the soft robot hand. This facilitates the soft robot hand to generate diverse anthropomorphic configurations, and by altering the palm configuration, it can achieve a wider range of object grasping capabilities.

Materials and Methods

In this section, we describe the molding of the robot hand and its mechanical properties, as illustrated in Figure 2.

FIG. 2.

Design and fabrication of the soft robotic hand. (A) The orange area shows the driving bar. (B) The kinematic analysis and modeling of the variable DOFs fingers of the metamorphic palm. (C) Pullout force results of different diameters of spheres in the soft hand having the highest stiffness and lowest stiffness. The inserted panels show the setup for the pullout test. (D) Two palm movements called palm bending and thumb adduction.

Finger design

The structure of the multistiffness soft finger is shown in Figure 1B. The soft finger consists of a silicone rubber part, a variable effective length multistiffness material layer²¹ composed of shape memory polymer (SMP), and a heating layer. For the silicone rubber parts, we used a Mold Star 30 (Smooth on Inc., USA), which was poured into a geometrically asymmetrical structure by a three-dimensional printed mold. The tunable-stiffness material and heating layer are poured into the bottom layer of silicone rubber. The multistiffness layer is made of epoxy resin (M04-A/B, PengSheng Inc., China) and poured into a sheet by pouring molds made of silicone rubber. The heating layer is a Ni–Cr alloy processed by laser cutting, and the heating layer of each finger is composed of three independent circuits at the top, middle, and root of the finger. By selectively heating different parts of the finger, the joints will selectively be softened, which map the human's motions of metacarpophalangeal (MCP) and proximal interphalangeal point (PIP) (Fig. 1D), as shown in Supplementary Video S1.

In the scenario depicted in Figure 1C, the robot fingers can discern variations in stiffness similar to human fingers. When human fingers are relaxed, the fingers can be in a low-stiffness state, deforming akin to human fingers in response to external forces (as illustrated on the left side). Conversely, when the muscles in human fingers are tense, resisting external forces, the mechanical fingers also assume a high-stiffness state, rendering them less susceptible to deformation (as illustrated on the right side). The properties of the elastic modulus of SMP materials with respect to temperature are shown in Supplementary Figure S2.

The multistiffness material presents itself in high stiffness at room temperature. According to previous research, a single soft finger containing multistiffness material can lift weights over 10 N horizontally.²² After heating to about 50°C by the heating circuit, the multistiffness material will lose its high stiffness, and the overall stiffness of the finger will be similar to those of soft fingers without multistiffness material.

The introduction of tunable-stiffness materials improves the gripping force of soft hands.²³ To systematically investigate the main factors affecting the gripping ability of soft hands, we designed a mechanical experimental platform to test the gripping power changes of soft hands when holding objects of different sizes under different stiffness conditions, as shown in Figure 2C. To simplify the experiment, we affixed the entire robot hand to a robot arm and then vertically pulled it upward to simulate a grabbing motion. As depicted in the figure, the captured object is attached to a six-dimensional force sensor (Mini 40 F/Tsensor, ATI, USA) using a fitting. The base of the force sensor is attached to the arm, while the palm is connected to the optical platform. When the mechanical arm moves vertically upward, it drives the object to move upward and gradually disengages from the palm. In the experiment, we first zeroed the weight of the object through the force sensor data acquisition program written by LabVIEW. The variable stiffness layer of the soft hand was heated and softened, and then the soft hand is driven to the preset air pressure to cover the object. After the variable stiffness layer was cooled and solidified, the robot arm was moved upward at a speed of 18 mm/s until the soft hand separated from the object. Throughout the grasping process, the force in the z-axis direction of the force sensor is collected to the personal computer at a frequency of 500 Hz and processed with a 15 Hz low-pass filter. Finally, the maximum force of each measurement is extracted as the gripping force, and the final gripping force result is obtained by averaging the gripping force of 5 trials.

We conduct pullout force experiments on spheres of varying diameters using the soft hand with the highest and lowest stiffness settings. We also measure the force changes using a force sensor when the sphere model is pulled out from the soft hand at low air pressure. The pullout force curve indicates that the use of tunable-stiffness materials in the highest stiffness setting enabled the soft hand to provide a maximum gripping force of ∼25 N. This demonstrates that the use of multistiffness materials overcomes the soft hand's weakness of lacking strength.

Palm metamorphic mechanism

The design of the palm is based on the metamorphic robotic hand mechanism proposed by Dai and Wang.²⁴ Our metamorphic palm comprises a five-bar spherical geometric metamorphic mechanism as shown in Figure 2B. The four fingers are affixed at designated points F₁ to F₄, respectively. The angle δ₁ represents the angular displacement between joint B and OF₁, while δ₂, δ₃, and δ₄ denote the respective angular displacements between joint D and OF₂, OF₃, and OF₄. Figure 2B illustrates how points F₁, F₂, F₃, and F₄ form diverse quadrilaterals for different palm configurations.

In the spherical five-bar linkage, joints A and E are actuated by the two servos, while joints B, C, and D remain passive. A global coordinate system is established at point O with its z-axis aligned with joint E and its y-axis directed along z₄ × z₁, coinciding with y₄ in Figure 2B. Given the values of angles θ₁ and θ₅ under this premise, the coordinates of B, C, and D in the global coordinate system can be determined.²⁴ $p_{B} = [\begin{matrix} x_{B} \\ y_{B} \\ z_{B} \end{matrix}] = R (y, α_{5}) R (z_{1}, θ_{1}) R (y_{1}, α_{1}) k$ (1)

p_{C} = [\begin{matrix} x_{C} \\ y_{C} \\ z_{C} \end{matrix}] = R (z_{5}, θ_{5}) R (y_{4}, α_{4}) R (z_{4}, θ_{4}) R (y_{3}, α_{3}) k

(2)

and $p_{D} = [\begin{matrix} x_{D} \\ y_{D} \\ z_{D} \end{matrix}] = R (z_{5}, θ_{5}) R (y_{4}, α_{4}) k$ (3)

To get a relationship between the palm motion and the motion of the fingers, local coordinate system F_i-x_Fiy_Fiz_Fi is defined at points F₁ to F₄ with z_Fi-axis aligned with OF_i, yF₁ directed along z_F1 × z₃, y_F2 is aligned with z_F2 × z₄, and y_Fi (i = 3, 4) directed along z_Fi × z₅. Local coordinate frames M_i-x_i1y_i1z_i1 (i = 1, 2, …, 4) are established for the MCP joints of each finger, where x_i1-axis aligned with the i-th MCP joint and z_i1-axis directed along F_iM_i. The angle γ_i represents the orientation difference between z_Fi and z_i1, while a_i0 denotes the distance between F_i and M_i as shown in Figure 2B. It should be noted that γ₁ equals zero. The unit vector k can be represented as k = [0, 0, 1]^T.

According to the above analysis, the coordinate transformation of the global coordinate system can be obtained as: $R_{F i} = \{\begin{matrix} R (y, α_{5}) R (z_{1}, θ_{1}) R (y_{1}, α_{1}) R (z_{2}, θ_{2}) R (y_{2}, δ_{1}) i f i = 1 \\ R (z_{2}, θ_{5}) R (y_{4}, α_{4}) R (z_{4}, θ_{4}) R (y_{3}, δ_{2}) i f i = 2 \\ R (z, θ_{5}) R (y_{4}, α_{4} - δ_{i}) i f i = 3, 4 \end{matrix}$ (4)

The homogeneous transformation matrix from the finger base coordinate system to the global coordinate system is presented as follows: $D_{F i} = [\begin{matrix} R_{F i} & R_{F i} k' \\ 0 & 1 \end{matrix}] (i = 1, 2, 3, 4)$ (5)

The position vector of point F_i in the global coordinate system can be obtained by multiplying R_Fik with k = [0, 0, R]^T, where R represents the radius of the sphere on which all the links move. Consequently, referring to Figure 2B, we can derive the homogeneous transformation of MCP joints in the global coordinate system:

where $D_{F M i} = [\begin{matrix} c γ_{i} & 0 & - s γ_{i} & - a_{i 0} s γ_{i} \\ 0 & 1 & 0 & 0 \\ s γ_{i} & 0 & c γ_{i} & - a_{i 0} c γ_{i} \\ 0 & 0 & 0 & 1 \end{matrix}]$ (7)

In contrast to typical drivers that are limited to a single motion mode, the soft finger introduced in this article offers versatility by enabling multiple motion modes through the control of SMP stiffness in different regions. We establish a mathematical model to describe the kinematics of the soft finger across various motion modes as in Figure 2B. To streamline the model, we assume that different regions maintain a consistent tangent alignment. Our model primarily focuses on the bottom profile of the soft finger, disregarding the superstructure. The entire process of coordinate transformation can be succinctly represented using a homogeneous transformation matrix, as follows: $_{0}^{2} A = [\begin{matrix} \begin{matrix} c o s k_{l 2} l_{2} & s i n k_{l 2} l_{2} \\ - s i n k_{l 2} l_{2} & c o s k_{l 2} l_{2} \end{matrix} & \begin{matrix} 0 & \frac{1 - c o s k_{l 2} l_{2}}{k_{l 2}} \\ 0 & \frac{s i n k_{l 2} l_{2}}{k_{l 2}} \end{matrix} \\ \begin{matrix} 0 & 0 \\ 0 & 0 \end{matrix} & \begin{matrix} 1 & 0 \\ 0 & 1 \end{matrix} \end{matrix}]$ (8)

Where l_i (i = 1, 2, 3) is the length of the corresponding region of the drive, and $k_{l i}$ is the corresponding curvature. If a region is rigid, as shown by the blue line in Figure 2B, the associated secondary transformation matrix can be expressed as: $_{0}^{1} A = [\begin{matrix} \begin{matrix} 1 & 0 \\ 0 & 1 \end{matrix} & \begin{matrix} 0 & 0 \\ 0 & k_{l 2} \end{matrix} \\ \begin{matrix} 0 & 0 \\ 0 & 0 \end{matrix} & \begin{matrix} 1 & 0 \\ 0 & 1 \end{matrix} \end{matrix}]$ (9)

So the rigid region can also be represented by $_{0}^{2} A$ . After the three partitions are connected in series, the homogeneous transformation matrix of the entire finger can be expressed as: $_{0}^{3} T =_{0}^{1} T_{1}^{2} T_{2}^{3} T$ (10)

Combining the previously derived kinematic analysis of the metamorphic palm and the motion of a single robot finger, we can readily obtain the kinematics of the entire deformable hand. The position and orientation of the fingertip on the hand can be expressed by multiplying Equations (7) and (10). $F_{i} = D_{m i}_{0}^{3} T$ (11)

Based on the variation in the input driving link, we categorize the movements of the palm into palm bending and thumb adduction (Fig. 2D). By altering the initial distribution of fingers, we are able to modify the grasping target accordingly, as illustrated in Supplementary Video S5. The outcome of the grasping experiment is presented in Figure 5.

sEMG-based hand gesture recognition

To record the sEMG signals related to hand gestures, we confirmed the placement of six sEMG sensors in the three different muscles of the forearm.²⁵ As shown in Figure 3A, the green zone represents the flexor digitorum superficialis, the blue zone represents the flexor carpi radialis, and the red zone represents the extensor carpi radialis. The selected muscle's location was determined through palpation methods, which are mainly responsible for flexing the fingers and palms. Informed consent is obtained from all participants in the study. The study was approved by the local Ethics Committee at the Department of Psychology, Tsinghua University (IRB202138).

FIG. 3.

(A) sEMG sensor placement diagram. (B) The sEMG signal and MCP joint signal correspond to the selected movements (the three curves indicate the sEMG signals of the three muscle groups, in which the green curve indicates FDS, the blue curve indicates FCS, and the pink curve indicates ECA). (C) Confusion matrix for different tasks (DOF recognition, Palm recognition, and Grasp recognition). ECA, extensor carpi radialis; FCS, flexor carpi radialis; FDS, flexor digitorum superficialis; sEMG, surface electromyography.

In this work, six hand gestures were selected to demonstrate the advantages of the proposed system: MCP, PIP, PIP + MCP, palm bending, thumb adduction, and relaxing. As shown in Figure 3B, the sEMG signals of the six types of hand gestures are acquired simultaneously, where different gestures correspond to different recognition tasks. It is worth noting that this work addresses the DOF recognition task, grasp detection task, stiffness recognition task, and palm configuration recognition task. MCP, PIP, and PIP + MCP characterize the different DOFs of the finger, corresponding to the DOF recognition task. It is worth noting that the DOF mode is targeted toward all fingers as a whole, rather than individual fingers, which means that the DOF recognition module maps the DOF mode of the five fingers of a human hand to the four fingers of the robotic hand. PIP + MCP and relaxing characterize muscle contraction and relaxation, corresponding to the grasp detection task and the stiffness recognition task. For the palm configuration recognition task, palm bending and thumb adduction characterize changes in palm structure.

The convolutional neural network (CNN) was initially proposed in 1980²⁶ and has been widely used in various fields, including image processing, video classification, and robot control, among others. It has proven to be effective in motion intent recognition as well.^27–29

In this study, we designed a lightweight classification algorithm based on CNN, as shown in Supplementary Figure S1. Specifically, the classification model we adopted consists of four convolution modules and two dense layers. A sliding window of 128 ms with a stride of 10 ms was used to feed the raw sEMG data to the classification model. Therefore, the input of the model is a two-dimensional (2D) sEMG matrix with dimensions of 256 × 6. Each convolution module contains one 2D convolution layer, one Batch Normalization layer, and one dropout layer. The kernel sizes of the convolution layers are (1,6), (1,5), (1,21), and (1,5), with the strides of 1, 1, 3, and 3, respectively. The hidden units of the two dense layers attached to the convolution modules are set to 32 and N, where N is determined by the choice of recognition task (DOF recognition: N = 3; Grasp detection/Palm configuration recognition/Stiffness recognition: N = 2). The aforementioned network hyperparameters are determined through sweep using grid search.

The computer hardware used is Intel Core i9-11900H, 32G memory, 16G video memory, an eight-core processor, and an NVIDIA RTX3080 graphics card. The establishment of CNN-based model was developed with the Python language using a deep learning framework termed Keras and the PyQt4 Library.

Control system

This study focuses on two types of control systems: teleoperation control system and hybrid control system.

Teleoperation control system

The sensing part of the teleoperation system is a sensing glove carrying an inertial measurement unit (IMU). The IMU can sense the overall posture and movement of the hand. The glove can measure the bending of the fingers by means of a mechanical structure and a potentiometer. Through a Bluetooth module, we transmit the finger bending signals and motion and attitude signals to the control part of the robot hand. During teleoperation of the sensing glove, the variable stiffness layer of the finger remains soft. The control system converts the positional information into control signals for the robotic arm and the bending signals of the fingers into driving air pressure for the pneumatic system. As a result, the robotic hand is able to map the movements of the controller's hand in real time through the actuation of the robotic arm and the pneumatic system.

Hybrid control system

As shown in Figure 4, the proposed hybrid control system integrates signals from a data glove and six sEMG sensors as input signals, which explores the feasibility of applying the proposed soft robotic hand to tasks involving grasping in narrow spaces. Specifically, the sEMG signals are used for tasks such as DOF Recognizing, Grasp Detection, and Stiffness Recognition, while the data glove signals are used to control the movement of the robotic arm.

FIG. 4.

Schematic diagram of the overall control system, which contains four main parts: the organism signal system, intention decoding system, actuation control system, and soft dexterous hand propriety. Specifically, the organic signal system is for collecting human biosignals, and the intent decoding system is for decoding human motion intent and converting it into control commands. As for the actuation control system it is for driving the corresponding part of the soft dexterous hand propriety to reproduce the hand movements corresponding to experimenter's motion intent.

First, the operator activates the corresponding DOF mode using a DOF gesture. Simultaneously, the heating system begins to heat up, following a predefined method for the specific DOF mode. During this time, the DOF recognition module is deactivated, while the grasp detection module is activated. Once the heating process is complete, the operator is able to control the robotic arm and hand system toward the targeted object by moving his/her arm. When the operator determines that the robotic hand has reached the appropriate position, the corresponding fingers begin to flex through a grasping gesture. After the robotic hand successfully completes the grasping action, the grasp detection module is deactivated, and the stiffness recognition module is activated. Depending on the task requirements, the operator can choose to activate the variable stiffness function by contracting the forearm muscles. Once activated, the heating system ceases operation, and the robotic hand enters a high stiffness state, maintaining the current finger flexion angle upon cooling down. Finally, the operator lifts the arm to raise the object, thereby completing the gripping operation task in narrow spaces.

Experiment Results

Actuation and control

The proposed overall control system is shown in Figure 4, consisting of four key components: the organism signal system, intention decoding system, actuation control system, and soft dexterous hand propriety.

Figure 3C and Table 1 illustrate the recognition performance of four tasks, with each task being trained separately by distinct classifiers that produce varying outputs. Specifically, the DOF recognition module and grasp recognition module generate three types of gestures to represent DOF motion and grasp motion, respectively. The palm recognition module generates two types of gestures that represent the motion of the palm (thumb inward and MCP motion), while the stiffness recognition module produces two types of states that indicate increased or decreased stiffness (muscle contraction and relaxation). Overall, the average accuracy rate for all tasks exceeds 90%.

Table 1.

The Recognition Effect of Four Tasks

Task name	Motion number	Average accuracy (%)
DOF recognition	4	94
Palm recognition	3	92
Grasp detection	4	94
Stiffness recognition	2	95

DOF, degree of freedom.

An exemplary flow control process is presented in Figure 4. First, the DOF recognition module identifies the active DOF by detecting a specified gesture to activate the corresponding heating system. Subsequently, the grasp recognition module identifies grasping motion and activates the finger pneumatic control system to accomplish soft hand grip function. Afterward, the palm control system is activated based on the detection results of the palm recognition module to regulate servo rotation and achieve desired changes in palm configuration. Ultimately, if the object being grasped has significant mass, the stiffness recognition module will halt heating operations and initiate cooling processes by detecting active muscle contractions from users to enhance soft robot hand stiffness.

Grasp experiments

To assess the grasping capabilities of our robotic hand, we conduct a series of grasping experiments based on the Feix taxonomy.^30–32 This taxonomy encompasses a total of 30 comprehensive grasp types designed for a 4-fingered hand, as illustrated in Figure 5A and Table 2. For detailed information regarding the dimensions and mass of supplementary experimental targets, please refer to Supplementary Figure S4.

FIG. 5.

Grasp experiments with various objects. (A) The experimental platform and the chosen objects as grasping targets. (B) The success criterion was the hand achieving a position in the Feix taxonomy with stable grasping and can still grasp for 30 s after rotating by 90°. (C) Enacted grasps according to the Feix taxonomy, and test whether the capture can be successful in mode palm flat, palm bending, thumb adduction, and both (palm bending + thumb adduction). D/A, Digital-to-Analog Converter; DC SSR, The name of the remote Stiffness Control System Controlled by Direct Current; GDW, Model of digital servo actuator; STM, The name of the single chip microcomputer model.

Table 2.

Experimental Results of Grasping in Mode Palm Flat, Palm Bending, Thumb Adduction, and Both (Palm Bending + Thumb Adduction), “Succeed”: The Gesture Successfully Grasps the Target Object and Can Sustain a 90° Rotation Without Falling

Mode name	1) Large diameter	2) Small diameter	3) Medium wrap	4) Adducted thumb	5) Light tool	6) Prismatic 3 finger	7) Prismatic 2 finger	8) Palmar pinch	9) Power disk	10) Power sphere
Palm flat	√	◯	√	√				√	◯	√
Palm bending	√	√	√	√	√			√		√
Thumb adduction	√		√			◯	◯		√
Both			√			√	√		√

Mode name	11) Precision disk	12) Precision sphere	13) Fixed hook	14) Lateral	15) Lateral	16) Index finger extension	17) Extension type	18) Distal type	19) Writing tripod	20) Tripod variation
Palm flat				√	√	√	√		√	√
Palm bending			√	√	√				√	√
Thumb adduction	√	√	◯	√		◯	◯	√		√
Both		√	√	√		√	◯			√

Mode name	21) Parallel extension	22) Adduction grip	23) Tip pinch grip	24) Lateral tripod	25) Quadpod	26) Stick	27) Palmar	28) Ring	29) Venteal	30) Inferior pincer
Palm flat		√		√		◯	◯		√	√
Palm bending		√		√		√	√		√	√
Thumb adduction	√	√	√		√		√	√
Both		√	√		◯	◯		√

◯, Hold; √, Succeed. “Hold”: The gesture is capable of gripping the object initially, but during the rotation process, there is a tendency for the object to fall.

For each trial, we adjust the palm configuration and ensured proper object positioning before applying 60 kPa atmospheric pressure to the targeted finger for activation. The primary success metric is the achievement of a stable grasp that adhered to the grasp types outlined in the Feix taxonomy (Fig. 5C). To evaluate grasping stability, we have rotated the hand by 90° (Fig. 5B), and we invite readers to refer to accompanying Supplementary Video S4 for demonstrations of the successful grasps achieved in each of the 30 flat palm modes. To differentiate between successful and failed grasp attempts, we added the result of hold in which the target can be caught in the initial state, but will fall in the process of rotation.

Table 2 presents the outcomes of our grasping experiments, including the performance alterations associated with enabling or disabling each palm component.

The anthropomorphic design of our robotic hand enables successful grasping of large and regular objects in a flat palm state. However, force closure becomes more challenging when dealing with small objects due to structural limitations that require palm bending or thumb abduction for improved coverage. By adjusting the palm configuration, previously unachievable gestures can now be accomplished, significantly enhancing tool handling capabilities. Our findings clearly demonstrate that metamorphic palms represent a highly effective approach for enhancing grasping success rates in the context of anthropomorphic soft robots.

Performance in teleoperation

We have demonstrated the capability of a soft robotic hand with a metamorphic palm to perform four complex tasks intuitively with the experimenter teleoperation, as shown in the accompanying video. Specifically, Figure 6A illustrates the tapping and swiping of a cell phone screen using a specially designed double pneumatic cavities thumb while holding the phone with one hand. Figure 6B demonstrates the manipulation of irregular building blocks along a portion of a curved guide rail. However, due to the lack of a feedback sensor on the robotic hand, it encountered difficulty traversing the entire complex trail, especially navigating several curves with ease. Figure 6C illustrates the successful completion of an accurate grasping ring task, where the thin rod is manipulated to touch the throat of a human head model for the purpose of sampling, as depicted in Figure 6D (as shown in Supplementary Video S2).

FIG. 6.

Teleoperation examples of the mapping method. (A) Tapping and sliding on the mobile phone screen. (B) Operate irregular building blocks through part of the curve track. (C) Accurate grasp ring. (D) The thin rod was operated to touch the throat of the human head model. (E) PIP mode grasp in the restricted environment. (F) Operation steps.

We present a novel approach for stiffness control in teleoperation using a soft robotic hand by integrating the MSS and stiffness control system. The experiment involves grasping with a soft robotic hand in a restricted environment comprising of two barriers that restrict its movement space. The target object for grasping is located between these barriers. The experimenter conducts real-time teleoperation of the soft robotic hand via the MSS, as shown in Figure 6F. The operation involves several steps, including DOF recognition, variable stiffness layer heating, gesture recognition, variable stiffness layer cooling, stiffness recognition and movement identification. Figure 6E illustrates the PIP mode grasp, where the experimenter adjusted the finger gesture to recognize DOF by bending the PIP joint of each finger after inflation. This process does not affect the barrier and results in force closure. The target object is successfully lifted once the variable stiffness layer has cooled and hardened (as shown in the Supplementary Video S3). Failure to lift due to insufficient finger freedom is shown in Supplementary Figure S3.

The results demonstrate that the MSS enables teleoperation of a soft robotic hand to accurately replicate the DOF, stiffness, and gestures of the experimenter's fingers, allowing for precise task completion under their control. This approach could find applications in various fields, including manufacturing, healthcare, and prosthetics, where precise and delicate manipulation is required.

Discussion

Anthropomorphic robot hands are typically constructed with multiple joints and connecting components, providing them remarkable DOFs. However, this intricate design complexity makes precise modeling a challenging task, demanding considerations of joint coupling and nonlinear characteristics. Consequently, mathematical expressions become intricate and hard to manage during the modeling process. Various objects and scenarios necessitate diverse grasping strategies, posing a challenge in devising universally applicable algorithms suitable for the complexity of these hands. Selecting optimal grasping approaches across various situations and managing uncertainties posed by soft finger deformations and nonlinear transformations prove to be algorithmically intricate. Furthermore, in real-world applications, achieving real-time adaptation of grasping strategies and finger stiffness is a challenge without a standardized control methodology.

In this article, we propose a promising framework for a soft anthropomorphic robot hand structure and control scheme. This innovative approach relies on only two servos as input to reconfigure the hand's form. By leveraging a priori knowledge derived from human hand grasping and mapping it onto the robotic hand, we achieve more appropriate and effective grasp selections. This is especially vital when handling intricate and irregular objects. Moreover, this scheme enables control of finger stiffness, facilitating adaptation to increasingly complex grasping scenarios. Experiment validation showcases the potential of this reconfigurable architecture to replicate complex human hand movements, enhancing grasp success rates and overall adaptability. In addition, the ability to adjust finger stiffness and DOFs further enhances performance, particularly in constrained operational environments.

Regarding the current limitation of our proposed design, it is necessary to heat and cool the SMP materials during the stiffness tuning process. As a result, significant time is spent on cooling, which further impacts the system's response speed. Due to the heating process taking ∼1 min, DOF recognition and grasp recognition need to be divided into two stages, requiring specific pre-motions to trigger DOF recognition. In our subsequent research, we aim to enhance the proposed design by addressing the inherent issue of slow material reaction through a combination of DOF recognition and grasp recognition. This will eliminate unnatural pre-motion design and improve overall user-friendliness. In addition, we plan to explore structural optimization such as Jamming structure to expedite response time.³³ Furthermore, future research is needed to improve the haptic feedback of the soft robotic hand.

Footnotes

Acknowledgment

The authors thank Chao Chen, Ulas, Bochao Li for their assistance in this work.

Author Disclosure Statement

No competing financial interests exist.

Funding Information

This work was supported by the National Key R&D Program of China (Grant No. 2020YFB1313003) and National Science Foundation support projects, China (Grant Nos. 91848206, 92048302, T2121003).

Supplementary Material

References

, Ma

, Liang

, et al. Vision-based teleoperation of shadow dexterous hand using end-to-end deep neural network. In: 2019 International Conference on Robotics and Automation (ICRA), Montreal, QC, Canada. IEEE; May 2019; pp. 416–422.

Ferre

, Aracil

, Balaguer

, et al. Advances in Telerobotics, Vol. 31. Springer: Berlin, Heidelberg;. 2007.

Chen

, Shao

, Xie

, et al. Soft origami gripper with variable effective length. Adv Intell Syst, 2021; 3(10):2000251.

Zhou

, Chen

, Li

DCF

, et al. 50 benchmarks for anthropomorphic hand function-based dexterity classification and kinematics-based hand design. In: 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Las Vegas, NV, USA. IEEE; October 2020; pp. 9159–9165.

Wang

, Fares

, Le

, et al. A novel soft robotic hand design with human-inspired soft palm. IEEE Robot Autom Magaz, 2021; 28(2):37–49.

Rohling

, Hollerbach

, Jacobsen

. Optimized fingertip mapping: A general algorithm for robotic hand teleoperation. Presence Teleoperat Virt Environ, 1993; 2(3):203–220.

Cerulo

, Ficuciello

, Lippiello

, Siciliano

. Teleoperation of the SCHUNK S5FH under-actuated anthropomorphic hand using human hand motion tracking. Robot Auton Syst, 2017; 89:75–84.

Ekvall

, Kragic

. Interactive grasp learning based on human demonstration. In: IEEE International Conference on Robotics and Automation, 2004. Proceedings. ICRA’04, New Orleans, LA, USA. IEEE; April 2004; Vol. 4; pp. 3519–3524.

Meeker

, Rasmussen

, Ciocarlie

. Intuitive hand teleoperation by novice operators using a continuous teleoperation subspace. In: 2018 IEEE International Conference on Robotics and Automation (ICRA), Brisbane, QLD, Australia. IEEE; May 2018; pp. 5821–5827.

10.

Fang

, Ma

, Wang

, Sun

. Vision-based posture-consistent teleoperation of robotic arm using multi-stage deep neural network. Robot Auton Syst, 2020; 131:103592.

11.

Schäfer

, Reis

, Stricker

. Controlling teleportation-based locomotion in virtual reality with hand gestures: A comparative evaluation of two-handed and one-handed techniques. Electronics, 2021; 10(6):715.

12.

Zi-You

, Xin-Gang

, Bi

, et al. Review of sEMG-based motion intent recognition methods in non-ideal conditions. Acta Automatica Sinica, 2021; 47(5):955–969.

13.

Geng

, Du

, Jin

, et al. Gesture recognition by instantaneous surface EMG images. Sci Rep, 2016; 6(1):36571.

14.

Wei

, Wong

, Du

, et al. A multi-stream convolutional neural network for sEMG-based gesture recognition in muscle-computer interface. Pattern Recogn Lett, 2019; 119:131–138.

15.

Olsson

, Malešević

, Björkman

, Antfolk

End-to-end estimation of hand-and wrist forces from raw intramuscular EMG signals using LSTM networks. Front Neurosci, 2021; 15:777329.

16.

Jiang

, Liu

, Fan

, et al. Random channel masks for regularization of least squares-based finger EMG-force modeling to improve cross-day performance. IEEE Trans Neural Syst Rehabil Eng, 2022; 30:2157–2167.

17.

, Chen

, Zhang

, et al. A novel myoelectric control scheme supporting synchronous gesture recognition and muscle force estimation. IEEE Trans Neural Syst Rehabil Eng, 2022; 30:1127–1137.

18.

Baldacchino

, Jacobs

, Anderson

, et al. Simultaneous force regression and movement classification of fingers via surface EMG within a unified Bayesian framework. Front Bioeng Biotechnol, 2018; 6:13.

19.

Zhang

, Fang

, Zhang

, et al. Simultaneous estimation of joint angle and interaction force towards sEMG-driven human-robot interaction during constrained tasks. Neurocomputing, 2022; 484:38–45.

20.

, Liu

, Zhou

, et al. Simultaneous hand/wrist motion recognition and continuous grasp force estimation based on nonlinear spectral sEMG features for transradial amputees. Biomed Signal Process Control, 2023; 85:105044.

21.

Hao

, Wang

, Xie

, et al. A eutectic-alloy-infused soft actuator with sensing, tunable degrees of freedom, and stiffness properties. J Micromech Microeng, 2018; 28(2):024004.

22.

Hao

, Liu

, et al. A soft gripper with programmable effective length, tactile and curvature sensory feedback. Smart Mater Struct, 2020; 29(3):035006.

23.

Emmanouil

, Wei

, Dai

. Spherical trigonometry constrained kinematics for a dexterous robotic hand with an articulated palm. Robotica, 2016; 34(12):2788–2805.

24.

Dai

, Wang

. Geometric analysis and synthesis of the metamorphic robotic hand. J Mech Des, 2007; 129(11):1191–1197.

25.

Fang

, Wang

, Sun

, et al. Simultaneous sEMG recognition of gestures and force levels for interaction with prosthetic hand. IEEE Trans Neural Syst Rehabil Eng, 2022; 30:2426–2436.

26.

LeCun

, Boser

, Denker

, et al. Backpropagation applied to handwritten zip code recognition. Neural Comput, 1989; 1(4):541–551.

27.

Yang

, Liu

. An EMG-based deep learning approach for multi-DOF wrist movement decoding. IEEE Trans Industr Electr, 2021; 69(7):7099–7108.

28.

, Chen

, Sheng

, Zhu

. Wrist torque estimation via electromyographic motor unit decomposition and image reconstruction. IEEE J Biomed Health Inform, 2020; 25(7):2557–2566.

29.

Qin

, Stapornchaisit

, He

, et al. Multi–joint angles estimation of forearm motion using a regression model. Front Neurorobot, 2021; 15:685961.

30.

, Zhang

, Xu

, et al. A soft neuroprosthetic hand providing simultaneous myoelectric control and tactile feedback. Nat Biomed Eng, 2023; 7(4):589–598.

31.

Wang

, Abu-Dakka

, Le

, et al. A novel design of soft robotic hand with a human-inspired soft palm for dexterous grasping. 2020; arXiv preprint arXiv:2009.00979.

32.

Feix

, Romero

, Schmiedmayer

, et al. The grasp taxonomy of human grasp types. IEEE Trans Hum Mach Syst, 2015; 46(1):66–77.

33.

Stanley

, Gwilliam

, Okamura

. Haptic jamming: A deformable geometry, variable stiffness tactile display using pneumatics and particle jamming. In: 2013 World Haptics Conference (WHC), Daejeon, Korea (South). IEEE; April 2013; pp. 25–30.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

2.55 MB

3.84 MB

2.95 MB

14.86 MB

32.65 MB

4.67 MB

0.00 MB