Abstract
Teleoperation in soft robotics can endow soft robots with the ability to perform complex tasks through human–robot interaction. In this study, we propose a teleoperated anthropomorphic soft robot hand with variable degrees of freedom (DOFs) and a metamorphic palm. The soft robot hand consists of four pneumatic-actuated fingers, which can be heated to tune stiffness. A metamorphic mechanism was actuated to morph the hand palm by servo motors. The human fingers' DOF, gesture, and muscle stiffness were collected and mapped to the soft robotic hand through the sensory feedback from surface electromyography devices on the jib. The results show that the proposed soft robot hand can generate a variety of anthropomorphic configurations and can be remotely controlled to perform complex tasks such as primitively operating the cell phone and placing the building blocks. We also show that the soft hand can grasp a target through the slit by varying the DOFs and stiffness in a trail.
Introduction
In robotic manipulation, the space of possible scenarios is expansive, and the objects of interest are quite diverse. To address this issue, one promising solution is to implement the human-robot collaborative methodology to control the robotic manipulator. Specifically, human cognitive abilities can be integrated through teleoperation to effectively manage difficult scenarios that may prove challenging for fully autonomous operation. Compared to intelligent programming, the use of human-robot teleoperation offers superior speed of decision-making and capability for dealing with complex operational issues.
The use of capturing human hand movements to control robot hands has great potential for intuitive control in numerous applications.1,2 However, this approach presents challenges when it comes to soft robot hands, which possess different materials and kinematics compared to their human counterparts. 3 Despite these challenges, previous research has demonstrated the versatility, robustness, and cost-effectiveness of soft robotic hands.4,5 To overcome these kinematic differences, researchers have proposed various mappings that require the pose of the teleoperator's hand as input. These mappings require the pose of the teleoperator's hand as input and include fingertip, 6 joint, 7 and pose mapping, 8 as well as teleportation subspace mapping.9–11 Typically, the required information is obtained through instrumented data gloves or vision. Although data gloves and vision are robust, they are limited to mapping hand gestures and poses and do not convey information related to strength and stiffness, which can be important for certain applications.
Electromyography (EMG) is an experimental technique that involves the development, recording, and analysis of myoelectric signals. EMG arises from physiological variations in the state of muscle fiber membranes. Depending on the placement of the sensing device, the measured EMG can be classified into intramuscular EMG and surface EMG (sEMG). 12 sEMG is acquired through the sensing device placed on the skin surface, and it provides information on the combined sEMG of muscle groups. Due to its ease of acquisition and rich information characteristics, sEMG has been widely used in myoelectric prosthesis hand control research, such as gesture recognition13,14 and force estimation.15,16 Hu et al. 17 proposed a novel myoelectric control scheme by utilizing multitask learning technique and postprocessing algorithm. The multitask learning technique was used to simultaneously predict grasp gestures and instantaneous grasp forces, while the postprocessing algorithm aimed to overcome the negative effects of force variations on gesture recognition accuracy.
Baldacchino et al. 18 introduced a Bayesian mixture expert model within the framework of sEMG-based intention recognition. This approach naturally incorporates the uncertainty of sEMG signals into the model structure, enabling simultaneous recognition of gestures and fingertip forces. Zhang et al. 19 proposed a deep learning model that achieved high-precision estimation of human motion and interaction forces simultaneously. Li et al. 20 introduced a novel framework that combines hand motion classification and continuous grasp force estimation based on a new spectral feature of EMG signals. The effectiveness and superiority of this method were validated on both able-bodied and amputee subjects.
The widespread use of sEMG has enabled force information to be applied in manipulator teleoperation. However, it should be noted that variable stiffness technology can improve grasping performance and increase the degrees of freedom (DOFs) of the soft hand. To our knowledge, there are no studies on intention decoding based on sEMG implementation of hand stiffness changes. Furthermore, there are few reports on the control of soft robot hand stiffness by hand mapping.
In this study, we propose an anthropomorphic soft robot hand with variable DOFs soft fingers and metamorphic palms as shown in Figure 1A. For the sake of clarity, the main contributions of this article are:

An anthropomorphic soft robot hand that maps the gesture, stiffness, and DOF of the human hand.
A novel design for a soft robot hand consists of four segmented heated pneumatic fingers and a metamorphic palm (Fig. 1A) that allows the hand to transform and optimally utilize its actuation degrees to breakthrough its actuation limitations.
Proposes a feasible motion mapping method to enhance recognition of the stiffness and the load of human hand in the teleoperation of the manipulator. (Fig. 1C, D).
Verify the effect of palm configuration on typical grasping tasks.
This article proposes a feasible motion mapping method to improve the recognition of stiffness and load of human hand during teleoperation of the manipulator. The multisensor sensing system (MSS) consisting of low-cost commercial sEMG and data gloves is utilized to map the information related to the DOF, gesture, and stiffness of the human hand to the soft robot hand. This facilitates the soft robot hand to generate diverse anthropomorphic configurations, and by altering the palm configuration, it can achieve a wider range of object grasping capabilities.
Materials and Methods
In this section, we describe the molding of the robot hand and its mechanical properties, as illustrated in Figure 2.

Design and fabrication of the soft robotic hand.
Finger design
The structure of the multistiffness soft finger is shown in Figure 1B. The soft finger consists of a silicone rubber part, a variable effective length multistiffness material layer 21 composed of shape memory polymer (SMP), and a heating layer. For the silicone rubber parts, we used a Mold Star 30 (Smooth on Inc., USA), which was poured into a geometrically asymmetrical structure by a three-dimensional printed mold. The tunable-stiffness material and heating layer are poured into the bottom layer of silicone rubber. The multistiffness layer is made of epoxy resin (M04-A/B, PengSheng Inc., China) and poured into a sheet by pouring molds made of silicone rubber. The heating layer is a Ni–Cr alloy processed by laser cutting, and the heating layer of each finger is composed of three independent circuits at the top, middle, and root of the finger. By selectively heating different parts of the finger, the joints will selectively be softened, which map the human's motions of metacarpophalangeal (MCP) and proximal interphalangeal point (PIP) (Fig. 1D), as shown in Supplementary Video S1.
In the scenario depicted in Figure 1C, the robot fingers can discern variations in stiffness similar to human fingers. When human fingers are relaxed, the fingers can be in a low-stiffness state, deforming akin to human fingers in response to external forces (as illustrated on the left side). Conversely, when the muscles in human fingers are tense, resisting external forces, the mechanical fingers also assume a high-stiffness state, rendering them less susceptible to deformation (as illustrated on the right side). The properties of the elastic modulus of SMP materials with respect to temperature are shown in Supplementary Figure S2.
The multistiffness material presents itself in high stiffness at room temperature. According to previous research, a single soft finger containing multistiffness material can lift weights over 10 N horizontally. 22 After heating to about 50°C by the heating circuit, the multistiffness material will lose its high stiffness, and the overall stiffness of the finger will be similar to those of soft fingers without multistiffness material.
The introduction of tunable-stiffness materials improves the gripping force of soft hands. 23 To systematically investigate the main factors affecting the gripping ability of soft hands, we designed a mechanical experimental platform to test the gripping power changes of soft hands when holding objects of different sizes under different stiffness conditions, as shown in Figure 2C. To simplify the experiment, we affixed the entire robot hand to a robot arm and then vertically pulled it upward to simulate a grabbing motion. As depicted in the figure, the captured object is attached to a six-dimensional force sensor (Mini 40 F/Tsensor, ATI, USA) using a fitting. The base of the force sensor is attached to the arm, while the palm is connected to the optical platform. When the mechanical arm moves vertically upward, it drives the object to move upward and gradually disengages from the palm. In the experiment, we first zeroed the weight of the object through the force sensor data acquisition program written by LabVIEW. The variable stiffness layer of the soft hand was heated and softened, and then the soft hand is driven to the preset air pressure to cover the object. After the variable stiffness layer was cooled and solidified, the robot arm was moved upward at a speed of 18 mm/s until the soft hand separated from the object. Throughout the grasping process, the force in the z-axis direction of the force sensor is collected to the personal computer at a frequency of 500 Hz and processed with a 15 Hz low-pass filter. Finally, the maximum force of each measurement is extracted as the gripping force, and the final gripping force result is obtained by averaging the gripping force of 5 trials.
We conduct pullout force experiments on spheres of varying diameters using the soft hand with the highest and lowest stiffness settings. We also measure the force changes using a force sensor when the sphere model is pulled out from the soft hand at low air pressure. The pullout force curve indicates that the use of tunable-stiffness materials in the highest stiffness setting enabled the soft hand to provide a maximum gripping force of ∼25 N. This demonstrates that the use of multistiffness materials overcomes the soft hand's weakness of lacking strength.
Palm metamorphic mechanism
The design of the palm is based on the metamorphic robotic hand mechanism proposed by Dai and Wang. 24 Our metamorphic palm comprises a five-bar spherical geometric metamorphic mechanism as shown in Figure 2B. The four fingers are affixed at designated points F1 to F4, respectively. The angle δ1 represents the angular displacement between joint B and OF1, while δ2, δ3, and δ4 denote the respective angular displacements between joint D and OF2, OF3, and OF4. Figure 2B illustrates how points F1, F2, F3, and F4 form diverse quadrilaterals for different palm configurations.
In the spherical five-bar linkage, joints A and E are actuated by the two servos, while joints B, C, and D remain passive. A global coordinate system is established at point O with its z-axis aligned with joint E and its y-axis directed along z4 × z1, coinciding with y4 in Figure 2B. Given the values of angles θ1 and θ5 under this premise, the coordinates of B, C, and D in the global coordinate system can be determined.
24
and
To get a relationship between the palm motion and the motion of the fingers, local coordinate system Fi-xFiyFizFi is defined at points F1 to F4 with zFi-axis aligned with OFi, yF1 directed along zF1 × z3, yF2 is aligned with zF2 × z4, and yFi (i = 3, 4) directed along zFi × z5. Local coordinate frames Mi-xi1yi1zi1 (i = 1, 2, …, 4) are established for the MCP joints of each finger, where xi1-axis aligned with the i-th MCP joint and zi1-axis directed along FiMi. The angle γi represents the orientation difference between zFi and zi1, while ai0 denotes the distance between Fi and Mi as shown in Figure 2B. It should be noted that γ1 equals zero. The unit vector k can be represented as k = [0, 0, 1]T.
According to the above analysis, the coordinate transformation of the global coordinate system can be obtained as:
The homogeneous transformation matrix from the finger base coordinate system to the global coordinate system is presented as follows:
The position vector of point Fi in the global coordinate system can be obtained by multiplying RFik with k = [0, 0, R]T, where R represents the radius of the sphere on which all the links move. Consequently, referring to Figure 2B, we can derive the homogeneous transformation of MCP joints in the global coordinate system:
where
In contrast to typical drivers that are limited to a single motion mode, the soft finger introduced in this article offers versatility by enabling multiple motion modes through the control of SMP stiffness in different regions. We establish a mathematical model to describe the kinematics of the soft finger across various motion modes as in Figure 2B. To streamline the model, we assume that different regions maintain a consistent tangent alignment. Our model primarily focuses on the bottom profile of the soft finger, disregarding the superstructure. The entire process of coordinate transformation can be succinctly represented using a homogeneous transformation matrix, as follows:
Where li (i = 1, 2, 3) is the length of the corresponding region of the drive, and
So the rigid region can also be represented by
Combining the previously derived kinematic analysis of the metamorphic palm and the motion of a single robot finger, we can readily obtain the kinematics of the entire deformable hand. The position and orientation of the fingertip on the hand can be expressed by multiplying Equations (7) and (10).
Based on the variation in the input driving link, we categorize the movements of the palm into palm bending and thumb adduction (Fig. 2D). By altering the initial distribution of fingers, we are able to modify the grasping target accordingly, as illustrated in Supplementary Video S5. The outcome of the grasping experiment is presented in Figure 5.
sEMG-based hand gesture recognition
To record the sEMG signals related to hand gestures, we confirmed the placement of six sEMG sensors in the three different muscles of the forearm. 25 As shown in Figure 3A, the green zone represents the flexor digitorum superficialis, the blue zone represents the flexor carpi radialis, and the red zone represents the extensor carpi radialis. The selected muscle's location was determined through palpation methods, which are mainly responsible for flexing the fingers and palms. Informed consent is obtained from all participants in the study. The study was approved by the local Ethics Committee at the Department of Psychology, Tsinghua University (IRB202138).

In this work, six hand gestures were selected to demonstrate the advantages of the proposed system: MCP, PIP, PIP + MCP, palm bending, thumb adduction, and relaxing. As shown in Figure 3B, the sEMG signals of the six types of hand gestures are acquired simultaneously, where different gestures correspond to different recognition tasks. It is worth noting that this work addresses the DOF recognition task, grasp detection task, stiffness recognition task, and palm configuration recognition task. MCP, PIP, and PIP + MCP characterize the different DOFs of the finger, corresponding to the DOF recognition task. It is worth noting that the DOF mode is targeted toward all fingers as a whole, rather than individual fingers, which means that the DOF recognition module maps the DOF mode of the five fingers of a human hand to the four fingers of the robotic hand. PIP + MCP and relaxing characterize muscle contraction and relaxation, corresponding to the grasp detection task and the stiffness recognition task. For the palm configuration recognition task, palm bending and thumb adduction characterize changes in palm structure.
The convolutional neural network (CNN) was initially proposed in 198026 and has been widely used in various fields, including image processing, video classification, and robot control, among others. It has proven to be effective in motion intent recognition as well.27–29
In this study, we designed a lightweight classification algorithm based on CNN, as shown in Supplementary Figure S1. Specifically, the classification model we adopted consists of four convolution modules and two dense layers. A sliding window of 128 ms with a stride of 10 ms was used to feed the raw sEMG data to the classification model. Therefore, the input of the model is a two-dimensional (2D) sEMG matrix with dimensions of 256 × 6. Each convolution module contains one 2D convolution layer, one Batch Normalization layer, and one dropout layer. The kernel sizes of the convolution layers are (1,6), (1,5), (1,21), and (1,5), with the strides of 1, 1, 3, and 3, respectively. The hidden units of the two dense layers attached to the convolution modules are set to 32 and N, where N is determined by the choice of recognition task (DOF recognition: N = 3; Grasp detection/Palm configuration recognition/Stiffness recognition: N = 2). The aforementioned network hyperparameters are determined through sweep using grid search.
The computer hardware used is Intel Core i9-11900H, 32G memory, 16G video memory, an eight-core processor, and an NVIDIA RTX3080 graphics card. The establishment of CNN-based model was developed with the Python language using a deep learning framework termed Keras and the PyQt4 Library.
Control system
This study focuses on two types of control systems: teleoperation control system and hybrid control system.
Teleoperation control system
The sensing part of the teleoperation system is a sensing glove carrying an inertial measurement unit (IMU). The IMU can sense the overall posture and movement of the hand. The glove can measure the bending of the fingers by means of a mechanical structure and a potentiometer. Through a Bluetooth module, we transmit the finger bending signals and motion and attitude signals to the control part of the robot hand. During teleoperation of the sensing glove, the variable stiffness layer of the finger remains soft. The control system converts the positional information into control signals for the robotic arm and the bending signals of the fingers into driving air pressure for the pneumatic system. As a result, the robotic hand is able to map the movements of the controller's hand in real time through the actuation of the robotic arm and the pneumatic system.
Hybrid control system
As shown in Figure 4, the proposed hybrid control system integrates signals from a data glove and six sEMG sensors as input signals, which explores the feasibility of applying the proposed soft robotic hand to tasks involving grasping in narrow spaces. Specifically, the sEMG signals are used for tasks such as DOF Recognizing, Grasp Detection, and Stiffness Recognition, while the data glove signals are used to control the movement of the robotic arm.

Schematic diagram of the overall control system, which contains four main parts: the organism signal system, intention decoding system, actuation control system, and soft dexterous hand propriety. Specifically, the organic signal system is for collecting human biosignals, and the intent decoding system is for decoding human motion intent and converting it into control commands. As for the actuation control system it is for driving the corresponding part of the soft dexterous hand propriety to reproduce the hand movements corresponding to experimenter's motion intent.
First, the operator activates the corresponding DOF mode using a DOF gesture. Simultaneously, the heating system begins to heat up, following a predefined method for the specific DOF mode. During this time, the DOF recognition module is deactivated, while the grasp detection module is activated. Once the heating process is complete, the operator is able to control the robotic arm and hand system toward the targeted object by moving his/her arm. When the operator determines that the robotic hand has reached the appropriate position, the corresponding fingers begin to flex through a grasping gesture. After the robotic hand successfully completes the grasping action, the grasp detection module is deactivated, and the stiffness recognition module is activated. Depending on the task requirements, the operator can choose to activate the variable stiffness function by contracting the forearm muscles. Once activated, the heating system ceases operation, and the robotic hand enters a high stiffness state, maintaining the current finger flexion angle upon cooling down. Finally, the operator lifts the arm to raise the object, thereby completing the gripping operation task in narrow spaces.
Experiment Results
Actuation and control
The proposed overall control system is shown in Figure 4, consisting of four key components: the organism signal system, intention decoding system, actuation control system, and soft dexterous hand propriety.
Figure 3C and Table 1 illustrate the recognition performance of four tasks, with each task being trained separately by distinct classifiers that produce varying outputs. Specifically, the DOF recognition module and grasp recognition module generate three types of gestures to represent DOF motion and grasp motion, respectively. The palm recognition module generates two types of gestures that represent the motion of the palm (thumb inward and MCP motion), while the stiffness recognition module produces two types of states that indicate increased or decreased stiffness (muscle contraction and relaxation). Overall, the average accuracy rate for all tasks exceeds 90%.
The Recognition Effect of Four Tasks
DOF, degree of freedom.
An exemplary flow control process is presented in Figure 4. First, the DOF recognition module identifies the active DOF by detecting a specified gesture to activate the corresponding heating system. Subsequently, the grasp recognition module identifies grasping motion and activates the finger pneumatic control system to accomplish soft hand grip function. Afterward, the palm control system is activated based on the detection results of the palm recognition module to regulate servo rotation and achieve desired changes in palm configuration. Ultimately, if the object being grasped has significant mass, the stiffness recognition module will halt heating operations and initiate cooling processes by detecting active muscle contractions from users to enhance soft robot hand stiffness.
Grasp experiments
To assess the grasping capabilities of our robotic hand, we conduct a series of grasping experiments based on the Feix taxonomy.30–32 This taxonomy encompasses a total of 30 comprehensive grasp types designed for a 4-fingered hand, as illustrated in Figure 5A and Table 2. For detailed information regarding the dimensions and mass of supplementary experimental targets, please refer to Supplementary Figure S4.

Grasp experiments with various objects.
Experimental Results of Grasping in Mode Palm Flat, Palm Bending, Thumb Adduction, and Both (Palm Bending + Thumb Adduction), “Succeed”: The Gesture Successfully Grasps the Target Object and Can Sustain a 90° Rotation Without Falling
◯, Hold; √, Succeed. “Hold”: The gesture is capable of gripping the object initially, but during the rotation process, there is a tendency for the object to fall.
For each trial, we adjust the palm configuration and ensured proper object positioning before applying 60 kPa atmospheric pressure to the targeted finger for activation. The primary success metric is the achievement of a stable grasp that adhered to the grasp types outlined in the Feix taxonomy (Fig. 5C). To evaluate grasping stability, we have rotated the hand by 90° (Fig. 5B), and we invite readers to refer to accompanying Supplementary Video S4 for demonstrations of the successful grasps achieved in each of the 30 flat palm modes. To differentiate between successful and failed grasp attempts, we added the result of hold in which the target can be caught in the initial state, but will fall in the process of rotation.
Table 2 presents the outcomes of our grasping experiments, including the performance alterations associated with enabling or disabling each palm component.
The anthropomorphic design of our robotic hand enables successful grasping of large and regular objects in a flat palm state. However, force closure becomes more challenging when dealing with small objects due to structural limitations that require palm bending or thumb abduction for improved coverage. By adjusting the palm configuration, previously unachievable gestures can now be accomplished, significantly enhancing tool handling capabilities. Our findings clearly demonstrate that metamorphic palms represent a highly effective approach for enhancing grasping success rates in the context of anthropomorphic soft robots.
Performance in teleoperation
We have demonstrated the capability of a soft robotic hand with a metamorphic palm to perform four complex tasks intuitively with the experimenter teleoperation, as shown in the accompanying video. Specifically, Figure 6A illustrates the tapping and swiping of a cell phone screen using a specially designed double pneumatic cavities thumb while holding the phone with one hand. Figure 6B demonstrates the manipulation of irregular building blocks along a portion of a curved guide rail. However, due to the lack of a feedback sensor on the robotic hand, it encountered difficulty traversing the entire complex trail, especially navigating several curves with ease. Figure 6C illustrates the successful completion of an accurate grasping ring task, where the thin rod is manipulated to touch the throat of a human head model for the purpose of sampling, as depicted in Figure 6D (as shown in Supplementary Video S2).

Teleoperation examples of the mapping method.
We present a novel approach for stiffness control in teleoperation using a soft robotic hand by integrating the MSS and stiffness control system. The experiment involves grasping with a soft robotic hand in a restricted environment comprising of two barriers that restrict its movement space. The target object for grasping is located between these barriers. The experimenter conducts real-time teleoperation of the soft robotic hand via the MSS, as shown in Figure 6F. The operation involves several steps, including DOF recognition, variable stiffness layer heating, gesture recognition, variable stiffness layer cooling, stiffness recognition and movement identification. Figure 6E illustrates the PIP mode grasp, where the experimenter adjusted the finger gesture to recognize DOF by bending the PIP joint of each finger after inflation. This process does not affect the barrier and results in force closure. The target object is successfully lifted once the variable stiffness layer has cooled and hardened (as shown in the Supplementary Video S3). Failure to lift due to insufficient finger freedom is shown in Supplementary Figure S3.
The results demonstrate that the MSS enables teleoperation of a soft robotic hand to accurately replicate the DOF, stiffness, and gestures of the experimenter's fingers, allowing for precise task completion under their control. This approach could find applications in various fields, including manufacturing, healthcare, and prosthetics, where precise and delicate manipulation is required.
Discussion
Anthropomorphic robot hands are typically constructed with multiple joints and connecting components, providing them remarkable DOFs. However, this intricate design complexity makes precise modeling a challenging task, demanding considerations of joint coupling and nonlinear characteristics. Consequently, mathematical expressions become intricate and hard to manage during the modeling process. Various objects and scenarios necessitate diverse grasping strategies, posing a challenge in devising universally applicable algorithms suitable for the complexity of these hands. Selecting optimal grasping approaches across various situations and managing uncertainties posed by soft finger deformations and nonlinear transformations prove to be algorithmically intricate. Furthermore, in real-world applications, achieving real-time adaptation of grasping strategies and finger stiffness is a challenge without a standardized control methodology.
In this article, we propose a promising framework for a soft anthropomorphic robot hand structure and control scheme. This innovative approach relies on only two servos as input to reconfigure the hand's form. By leveraging a priori knowledge derived from human hand grasping and mapping it onto the robotic hand, we achieve more appropriate and effective grasp selections. This is especially vital when handling intricate and irregular objects. Moreover, this scheme enables control of finger stiffness, facilitating adaptation to increasingly complex grasping scenarios. Experiment validation showcases the potential of this reconfigurable architecture to replicate complex human hand movements, enhancing grasp success rates and overall adaptability. In addition, the ability to adjust finger stiffness and DOFs further enhances performance, particularly in constrained operational environments.
Regarding the current limitation of our proposed design, it is necessary to heat and cool the SMP materials during the stiffness tuning process. As a result, significant time is spent on cooling, which further impacts the system's response speed. Due to the heating process taking ∼1 min, DOF recognition and grasp recognition need to be divided into two stages, requiring specific pre-motions to trigger DOF recognition. In our subsequent research, we aim to enhance the proposed design by addressing the inherent issue of slow material reaction through a combination of DOF recognition and grasp recognition. This will eliminate unnatural pre-motion design and improve overall user-friendliness. In addition, we plan to explore structural optimization such as Jamming structure to expedite response time. 33 Furthermore, future research is needed to improve the haptic feedback of the soft robotic hand.
Footnotes
Acknowledgment
The authors thank Chao Chen, Ulas, Bochao Li for their assistance in this work.
Author Disclosure Statement
No competing financial interests exist.
Funding Information
This work was supported by the National Key R&D Program of China (Grant No. 2020YFB1313003) and National Science Foundation support projects, China (Grant Nos. 91848206, 92048302, T2121003).
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
