Soft Robot Proprioception Using Unified Soft Body Encoding and Recurrent Neural Network

Abstract

Compared with rigid robots, soft robots are inherently compliant and have advantages in the tasks requiring flexibility and safety. But sensing the high dimensional body deformation of soft robots is a challenge. Encasing soft strain sensors into the internal body of soft robots is the most popular solution to address this challenge. But most of them usually suffer from problems like nonlinearity, hysteresis, and fabrication complexity. To endow the soft robots with body movement awareness, this work presents a bioinspired architecture by taking cues from human proprioception system. Differing from the popular usage of smart material-based sensors embedded in soft actuators, we created a synthetic analog to the human muscle system, using paralleled soft pneumatic chambers to serve as receptors for sensing body deformation. We proposed to build the system with redundant receptors and explored deep learning tools for generating the kinematic model. Based on the proposed methodology, we demonstrated the design of three degrees of freedom continuum joint and how its kinematic model was learned from the unified pressure information of the actuators and receptors. In addition, we investigated the response of the soft system to receptor failures and presented both hardware and software level solutions for achieving graceful degradation. This approach offers an alternative to enable soft robots with proprioception capability, which will be useful for closed-loop control and interaction with environment.

Introduction

Proprioception is a critical bodily neuromuscular sense that enables human to perceive posture of limbs in the space and helps to control body movements.^1,2 Equipping soft robots with comparable proprioception systems is also essential for closed-loop feedback control and interaction with environment. But due to the unique compliant nature, it is very challenging to sense the high dimensional deformation of soft robots such as the octopus-inspired gripper³ and the elephant's trunk like robot manipulator.⁴

To endow soft robots with body movement awareness, deciding the sensory component is a key step. Depending on the morphology of the soft robots and the task requirements, diverse potential sensing technologies were developed over the past years.^5–9 Embedding soft strain sensors into the soft robots was one of the most popular solutions for directly measuring the body deformation of the soft robots. Stretchable resistive^10–12 and capacitive^13–15 materials were commonly used for building the strain sensors. However, most of these soft strain sensors showed nonlinearity and suffered from hysteresis. Some researchers proposed to fill the soft elastomers with conductive liquids for proprioceptive sensing.^16–18 But these sensors filled with conductive liquids usually had complex structures and might have the problem of leakage.

Recent researches proposed to incorporate complicated sensors within the soft actuators in a tightly integrated way by leveraging the promising and low-cost three-dimensional (3D) printing technologies. One of the noteworthy examples is a soft somatosensitive actuator constructed with three 3D-printed elastomeric matrices, which were filled with conductive inks for sensing the curvature, inflation, and contact.¹⁹ Yang et al. proposed to simultaneously integrate pressure and position sensors into a soft actuator using 3D printing.²⁰ Hainsworth et al. also produced a sensor-actuator system entirely using multimaterial 3D printing tools.²¹ Scharff et al. proposed a color-based sensing approach to reconstruct the shape of a bellows actuator, which was fabricated with multicolor structure by a multimaterial 3D printer.²²

Some other soft sensors embedded with fiber Bragg gratings^23–25 and optical waveguides^26–28 were also reported. These options could provide sensitive deformation measurement while they might affect the compliance and the dynamics of the system. An inductance-based sensor was recently developed for measuring the orientation of bellows driven continuum joints.²⁹ Although the inductance-based sensor system was not bulky and off-the-shelf, it was susceptible to electromagnetic interference. Festo corporation demonstrated a bellows driven continuum arm with whole kinematics awareness, which was achieved by an optical shape sensor placed along the arm's longitudinal axis.⁴ External vision techniques were also leveraged for sensing the shape of soft robots,^30–32 yet they were restricted by the brightness of light and the delay of the visual sensor.

In addition, some studies proposed to build soft sensors based on deformable chamber structures and off-the-shelf pressure sensors. Tawk et al. evaluated four types of 3D-printed soft pneumatic sensing chambers and, respectively, demonstrated their abilities in sensing touch, bending, torsional, and rectilinear deformation.³³ In another work carried out by Tawk et al., a soft robotic finger was integrated with soft pneumatic sensing chambers for detecting the joint position and the touch at the fingertip.³⁴ Similarly, Yang et al. also fabricated a soft pneumatic sensor for measuring the contact force and curvature of a soft bending actuator.³⁵ The soft pneumatic sensor was composed of a gas pressure sensor and an air chamber made of silicon rubber. There also has been an attempt to construct tactile array sensors by casting miniature barometric sensor chips in rubber.³⁶ The abovementioned examples showed the versatility of pneumatic sensing technology, which we also applied in this work.

Decoding the raw sensor readings into the system states is another critical step when the sensor hardware has been integrated in the soft system. Mapping from the sensor readings to the system states based on analytical models is usually difficult due to the fabrication error and the complex dynamics of the soft systems. Recent studies in the field of soft robotics have explored machine learning methods for solving the challenges of modeling.³⁷ For examples, Kim et al. handled the noise of the sensor output with probabilistic modeling and characterized the hysteresis using a Bayesian network.³⁸ Han et al. implemented a hierarchical recurrent sensing network to calibrate the soft sensors.³⁹ Fang et al. applied both feed-forward neuronal network (FNN) and long short-term memory (LSTM) to learn the forward and inverse kinematics of a soft continuum joint and a planar finger manipulator.⁴⁰ Van Meerbeek et al. trained a multioutput regression model for simultaneously estimating the bend and twist angles of an elastomeric foam sensor system.⁴¹

To explore the best mapping between the sensor data and the 3D shape of pneumatic soft robots, Scharff et al. tested different machine learning models, including a LSTM network, a FNN, a support vector regression model, and multivariate linear regression model.⁴² A deep convolutional neural network was used for learning the patterns in the raw data of the sensor array.⁴³ Recurrent neural network (RNN) architectures were also used in some studies to estimate the system states from time series data.^44,45 Loo et al. adopted a RNN-based adaptive unscented Kalman filter to estimate both the internal and external state of a soft pneumatic finger.⁴⁶ Sakurai et al. proposed to use an echo state network to estimate the length of McKibben pneumatic artificial muscles from the real time pressure sensor data and system dynamics.⁴⁷ Soter et al. trained a three-layer deep LSTM type of RNN for reconstructing the deformation of soft interface from multidimensional time serial sensor data.⁴⁸

Concept of proprioception scheme for soft robots

This work takes cues from the human proprioceptive system for endowing soft robots with a similar capability. As schematically shown in Figure 1A, the neurological basis of the human proprioceptive system originates from organs called muscle spindles. The intrafusal muscle fibers enclosed in the muscle spindles act as receptors that provide the information of the muscle length, while those extrafusal muscle fibers form the units to generate force and movement. The muscle spindles orient parallelly to the extrafusal muscle fibers, so that the receptors are deformed when the extrafusal muscle fibers lengthen or contract. Information of the body posture is encoded by populations of these passively deformed receptors, generating nerve impulses to the spinal cord and cerebral cortex for high-level processing.^49,50

FIG. 1.

(A) Human proprioceptive system. (B) Overview of the proprioception architecture for soft robots and its analogy to the human sense of proprioception. a – a, section view of the structure of the parallel soft pneumatic chambers. RNN, recurrent neural network.

By creating an analogy to the paralleled morphology of the muscle fibers and its neurological basis, we presented a bioinspired architecture (Fig. 1B) to endow a soft pneumatic robot with proprioceptive capability. Particularly, we demonstrated and validated the proposed architecture on three degrees of freedom (3-DoF) soft continuum joint.

Differing from the popular usage of dedicated embedding sensors, we proposed to use soft pneumatic chambers to serve as receptors for sensing the body deformation, which can be signaled by inner pressure changes. The soft actuators and soft receptors are designed with unified bellows structure, which have been explored by many previous studies.^4,29,51–53 With the actuators and receptors being parallelly arranged, the receptors will be passively deformed when the actuators are pressurized. The deformation can be signaled by the corresponding inner pressure changes of the receptor chambers. All these pressure information encode the system motion states. Inspired by the muscle system that has highly redundant receptors, we proposed to build redundant receptor segments in the soft system. Learning-based approaches were explored to decode pressure information for modeling system kinematics.

Compared with the state of the art, the primary contribution of this work is the idea of using soft pneumatic chambers as receptors for encoding the deformation of the soft robot system. Such a concept can be easily generalized to different soft robot systems. Particularly, the receptors share the same structural design with the actuators; therefore, off-the-shelf technologies can be leveraged to fabricate them without special treatment. The feedback from the actuators and receptors is the same type of pressure signals, which can be easily obtained from readily-available and low-cost pneumatic pressure sensors commonly incorporated in most soft robots. The learning algorithms proposed in this work are directly fed with unified pressure data, while most of the existing methods may require preprocessing of the data from diverse sources. Another contribution is that we proposed both the software and hardware level proposals for graceful degradation, which enable the soft robots to maintain their performance even when they suffer receptor failures.

Materials and Methods

The proposed approach was validated on a 3-DoF pneumatically actuated soft continuum joint. As shown in Figure 1B, the soft continuum joint consisted of two plates connected to seven distributed soft bellows, with one located at the center and the other six spaced around the central one in a circular array pattern. All the bellows were of an identical structure design. The central bellows were designated as receptor, and the outer six bellows were alternately configured as actuators and receptors, so that in total the joint system had three actuators and four receptors.

The circular array configuration of the bellows created 3-DoF motion by inflating the actuator bellows. As the actuator bellows were pressurized, the receptor bellows were passively stretched or compressed. All the bellows were connected to pressure sensors for monitoring their inner pressure changes due to inflation or body deformation. A motion capture system was leveraged for providing the ground truth of the pose of the soft continuum joint. RNN architecture was used for learning the mapping from the sequential pressure signals to the pose of the soft continuum joint, $[p (t), c (t)] \to y (t)$ . In this study, $p (t)$ is the pressure readings of the bellows, $c (t)$ is the current state of the RNN network, and $y (t)$ is the kinematic parameters to be estimated.

Fabrication of the elastomeric bellows

The bellows body was fabricated following the molding and casting process as shown in Figure 2A. All the molds were printed using a 3D printer. The molds were assembled and held together firmly to form the chamber of the bellows. Silicone elastomer was then poured into the mold, and the molds were removed after the silicone rubber cured, creating the bellows chamber. Finally, one open end of the bellows chamber was capped by 3D printed flanges using adhesives. The other open end was capped by a similar 3D printed flange designed with a nozzle for connection of pneumatic tubes.

FIG. 2.

Fabrication and characterization of the pneumatic bellows chambers. (A) Schematic outlining of the elastomeric bellows fabrication process. (B) An in-house developed test machine for characterization of the bellows chamber under stretched deformation. (C) The pressure measurements of the bellows chamber under stretched deformation over 1000 cycles. The characteristics of the bellows chamber were repeatable and consistent across the long-term cycles. 3D, three-dimensional.

Characterization of bellows chamber under stretched deformation

In response to stretched deformation, the bellows chamber exhibited changes in the inner pressure. To characterize the bellows chamber under stretched deformation, an in-house developed stretch test machine (a linear motion platform driven by stepper motor) was used as shown in Figure 2B. In the test, one flange of the soft bellows was mounted on the slider of the linear motion platform, and the other flange was fixed and connected to a pressure sensor (XGZP6847A040KPGPN, Range: −40 to 40 kPa, CFSENSOR Ltd.). We applied controlled motion to stretch the bellows between 0 and 15 mm (KPM16, 50 mm max., MIRAN Ltd.) and simultaneously recorded the pressure of the bellows. The stretch motion was controlled at a specific rate (13.3 mm/s). In response to the stretched length between 0 to 15 mm, the pressure of the bellows varied from 0 kPa (unstretched) to above 12 kPa (at 15 mm stretched length). The pressure measurements of long-term test with over 1000 cycles are shown in Figure 2C. The bellows chamber showed minimal hysteresis, and its characteristics were reliable over the entire set of cycles.

Experimental setup

As shown in Figure 3A, we fixed one plate of the soft continuum joint onto an aluminum-alloyed stand so that the actuators could drive the other plate to move relative to the fixed one. The ground truth of the pose of the active plate was measured by a 6-DoF motion capture system composed of a tracker and base station (NOLO CV1 PRO, NOLO Inc.). We mounted the tracker at the center of the active plate and placed the base station on the ground with the tracker located into its positioning range.

FIG. 3.

(A) Experimental setup for sampling data, training, and testing the kinematic model. (B) The kinematic description of the soft continuum joint.

A control system was developed for measuring the pressures of all the bellows chambers and regulating the pressures inside the actuators. A sequence of reference pressure values ranging from 0 to 20 kPa was randomly generated by the control board. The pressures inside the actuators were controlled under a proportional-derivative rule. Since the motion capture system measured the pose at 50 Hz, we also sampled the pressure readings to 50 Hz.

Kinematic description and sampling

In our experiment, we described the movements of the soft continuum joint relative to its initial pose when all the actuators were not pressurized. As illustrated in Figure 3B, assume that the initial pose of the soft continuum joint was obtained with position $P_{0} = {[\begin{matrix} P_{0 x} & P_{0 y} & P_{0 z} \end{matrix}]}^{T}$ and rotation $R_{0}$ , and the pose of the soft continuum joint was recorded with position $P_{t} = {[\begin{matrix} P_{t x} & P_{t y} & P_{t z} \end{matrix}]}^{T}$ and rotation $R_{t}$ at time $t$ , the movement of the soft continuum joint relative to the initial pose could be obtained $P_{0}^{t} = {[\begin{matrix} P_{t x} - P_{0 x} & P_{t y} - P_{0 y} & P_{t z} - P_{0 z} \end{matrix}]}^{T}$

and $R_{0}^{t} = {R_{0}}^{T} R_{t} = [\begin{matrix} r_{11} & r_{12} & r_{13} \\ r_{21} & r_{22} & r_{23} \\ r_{31} & r_{32} & r_{33} \end{matrix}]$

Particularly, the rotation motion could also be described by Euler angles (yaw, pitch, roll). But the soft continuum joint only had two rotational degrees of freedom, that is, pitch and roll. Thus, the pitch angle $β (t)$ and roll angle $α (t)$ were enough to describe the rotation of the joint at time $t$ , as could be calculated from the rotation matrix $R_{0}^{t}$ : $β (t) = atan 2 (- r_{31}, \sqrt{r_{32}^{2} {+ r}_{33}^{2}}), α (t) = atan 2 (r_{32}, r_{33}) .$

In addition to the rotation parameters, the pose of the soft continuum joint could be determined by one more parameter that was the translational distance $L (t) = ∥P_{0}^{t}∥ = \sqrt{{(P_{t x} - P_{0 x})}^{2} + {(P_{t y} - P_{0 y})}^{2} + {(P_{t z} - P_{0 z})}^{2}}$ for describing the Cartesian position changes. Therefore, for this 3-DoF joint, its kinematics could be described by a 3D parameter $y (t) = {[\begin{matrix} L (t) & β (t) & α (t) \end{matrix}]}^{T}$ , which could be transformed from the 6-DoF pose readings provided by the motion capture system.

To collect data for training the models, the soft continuum joint was actuated to different pose under random pressure inputs while the pose information $y (t)$ and the corresponding pressure data $p (t)$ were collected. The training data were continuously collected when the soft continuum joint was actuated without any external contact. For testing the model performance with contact, the active plate of the continuum joint was occasionally brought in contact at different locations and for different duration. The sampling rate of the pressure and pose data was 50 Hz, and the whole sampling period lasted about 10 min. In total, 30,000 pairs of pressure (predictor) and pose (response) observations were collected. A sequence of 24,000 pairs of pressure and pose observations was used for training. Two sequences of 3000 pairs of pressure and pose observations that collected under noncontact and contact scenario, respectively, were used for testing the performance of the models.

RNNs for kinematics prediction

As the variants of neural networks, RNNs are excellent in processing sequential data. Unlike other neural networks, RNNs infer the output based on both the input and the internal state that memorizes the previous computations. Although RNNs can theoretically be useful for sequential data, they generally cannot be directly deployed due to the problem of long-term dependency in model training. There are two widely used architectures of RNNs which can alleviate the problem of long-term dependency by introducing gate mechanism: LSTM⁵⁴ and gated recurrent unit (GRU⁵⁵).

Long short-term memory

The structure of LSTM is shown in Figure 4A. LSTM consists of three gates, that is, forget gate, input gate, and output gate. At each step $t$ , the current input $p_{t}$ and incoming short memory (i.e., the hidden state $h_{t - 1}$ of the previous step $t - 1$ ) regulate the three gates to control the removal and storage of the information. The forget gate $f_{t}$ decides how much information from the incoming long-term memory (i.e., the cell state $C_{t - 1}$ of the previous step $t - 1$ ) can be kept. The input gate $i_{t}$ controls how much information of the input at the current step can be retained in the long-term memory (i.e., the cell state $C_{t}$ ). The output gate $o_{t}$ takes the newly computed long-term memory to generate new short-term memory (i.e., the hidden state $h_{t}$ ). The LSTM equations are the following:

FIG. 4.

(A) LSTM structure. (B) GRU structure. (C) LSTM or GRU Network architecture. GRU, gated recurrent unit; LSTM, long short-term memory.

f_{t} = s i g m o i d (W_{f} \cdot [h_{t - 1}, p_{t}] + b_{f})

(1)

i_{t} = s i g m o i d (W_{i} \cdot [h_{t - 1}, p_{t}] + b_{i})

(2)

C_{t} = f_{t} ⨀ C_{t - 1} + i_{t} ⨀ t a n h (W_{c} \cdot [h_{t - 1}, p_{t}] + b_{c})

(3)

o_{t} = s i g m o i d (W_{o} \cdot [h_{t - 1}, p_{t}] + b_{o})

(4)

h_{t} = o_{t} ⨀ t a n h (C_{t})

(5)

where $⨀$ denotes the Hadamard product operator; $W_{*}$ and $b_{*}$ are the weight and bias parameters which need to be learned during training.

Gated recurrent unit

Figure 4B shows the structure of GRU. As a variation and simplification of LSTM, GRU incorporates only two gate mechanisms called reset gate and update gate. The reset gate $r_{t}$ is used to decide how much of the relevant information from the previous hidden step (i.e., $h_{t - 1}$ ) should be stored into the current memory content (i.e., ). The update gate $z_{t}$ determines what to collect from the current memory content and what information from the previous step for calculating the new hidden state $h_{t}$ and passing it down to the next step. The GRU equations are the following: $r_{t} = s i g m o i d (W_{r} \cdot [h_{t - 1}, p_{t}] + b_{r})$ (6) $z_{t} = s i g m o i d (W_{z} \cdot [h_{t - 1}, p_{t}] + b_{z})$ (7)

where $⨀$ denotes the Hadamard product operator; $W_{*}$ and $b_{*}$ are the weight and bias parameters which need to be learned during training.

Using MATLAB's Deep Learning Toolbox, both the networks of LSTM and GRU architecture were designed for pose prediction with the pressure signal as input. As illustrated in Figure 4C, the network started with a sequence input layer followed by an LSTM or GRU layer, which was configured with 50 hidden units by considering the performance and computation time (Supplementary Table S1). To prevent overfitting and make predictions more robust to noise, a dropout layer with dropout rate of 0.1 was specified preceding the LSTM or GRU layer. The networks ended with a fully connected layer and a regression layer. The training data were normalized for a better fit and to prevent the training from diverging. For comparison purpose, the networks were trained with the same sets of hyperparameters. The networks were trained on a single graphics processing unit (Nvidia GeForce MX150) with Adam optimizer for a maximum of 500 epochs (as shown in Supplementary Fig. S1). The root mean squared error (RMSE) of the prediction from the test dataset was evaluated for each network.

Results

Kinematic modeling

With the trained models (both LSTM and GRU), the receptor pressure information could be used to predict the pose of the continuum joint at each step. As shown in Figure 5, the trained models were tested under both noncontact (from the 536th second to the 596th second) and random contact (from the 756th second and 816th second) scenario.

FIG. 5.

The kinematic model was trained using the pressure information of four receptors and tested under both noncontact and random contact scenario.

The performance of the LSTM and GRU model is shown in Table 1. On average, it took longer to train a LSTM model (484 s) than GRU model (459 s) because LSTM network used more training parameters. Overall, however, LSTM model was more accurate than GRU model. The results showed that LSTM model could provide effective prediction of pose under noncontact scenario, with the RMSE of 1.10 ± 0.02 mm for predicting $L$ , the RMSE of 0.62 ± 0.05° for predicting $β$ , and the RMSE of 0.71 ± 0.06° for predicting $α$ . The GRU model performed similarly in predicting $L$ , while the accuracies of predicting $β$ and $α$ were worse. But the prediction performance of both the LSTM and GRU model deteriorated upon contact, with the RMSE over 2.3 mm for predicting $L$ , the RMSE over 1.9° for predicting $β$ , and the RMSE over 1.6° for predicting $α$ . In this work, accuracy was the first concern; therefore, we chose LSTM network for predicting the pose.

Table 1.

Performance of the Long Short-Term Memory and Gated Recurrent Unit Network

Models	Training time (seconds)	Test performance (RMSE)
		Noncontact			Random contact
		L (mm)	$β$ (deg)	$α$ (deg)	L (mm)	$β$ (deg)	$α$ (deg)
LSTM	484.0 ± 1.7	1.10 ± 0.02	0.62 ± 0.05	0.71 ± 0.06	2.33 ± 0.02	1.91 ± 0.19	1.66 ± 0.21
GRU	459.0 ± 6.1	1.11 ± 0.01	1.04 ± 0.30	0.93 ± 0.26	2.31 ± 0.03	1.97 ± 0.37	1.77 ± 0.09

The data are shown as mean ± standard deviation. The bold values denote the best performance of each metric.

GRU, gated recurrent unit; LSTM, long short-term memory; RMSE, root mean squared error.

Figure 6 shows the trajectories of the measured and predicted pose of the continuum joint for a period of 20 s using LSTM model. The corresponding error plots for both noncontact and random contact scenario are shown in Figure 7. The prediction of the translational distance ( $L$ ) and rotation ( $β$ and $α$ ) was affected obviously by external contact.

FIG. 6.

Pose prediction for a period of 20 s. (A) Noncontact scenario. (B) Random contact scenario.

FIG. 7.

Error plots of pose prediction. (A) Noncontact scenario. (B) Random contact scenario.

Receptor failure

Soft robots may suffer from body segment failures due to fatigue aging, mechanical damage, or environmental corrosion. For example, the receptor chamber made of silicon rubber may be punctured by sharp objects and the pressure reading of the receptor will drop to zero. In such cases, the receptor's information for prediction will be lost.

In this study, we performed some tests to investigate the robustness of the LSTM model in the face of receptor failures. In the test, we set the receptor pressure readings to zero for simulating the corresponding receptor failure. The performance of the LSTM model in response to receptor failures is shown in Figure 8A and B. Under both noncontact and random contact scenario, the prediction accuracy gradually decreased with more and more receptor failures.

FIG. 8.

Performance of the model in the face of receptor failures. (A) Noncontact scenario. (B) Random contact scenario. (C) Prediction error distribution of tracker position under noncontact scenario. RMSE, root mean squared error.

Effect of receptor layout

Each receptor appeared to contribute differently to the prediction accuracy, which was associated with the receptor layout. For example, the failure of the central receptor (receptor 1) had small and equal influence on the prediction accuracy of $L$ , $β$ , and $α$ under both noncontact and random contact scenario. Failure of receptor 2 had great impact on the prediction accuracy of $β$ but it brought less impact on the accuracy of $L$ and $α$ . The prediction accuracy of $α$ was considerably affected by the failure of receptor 3. However, the accuracy of $L$ and $β$ was affected a little when the information of receptor 3 was lost. Failure of receptor 4 evenly reduced the prediction accuracy of $β$ and $α$ , but had a relatively small effect on the prediction of $L$ .

This effect could also be observed from prediction error distribution of the tracker position in the workspace as shown in Figure 8C. The prediction error of tracker position increased when there were receptor failures. Most importantly, the error distribution showed that receptor failures brought different impact on the tracker position prediction in different regions of the workspace. For example, as failure of receptor 2 caused large error in prediction of pitch angle $β$ , the prediction error of tracker position in the region along the pitch motion would become larger. Similarly, failure of receptor 3 caused larger error in the region along the rolling motion compared to other regions in the workspace.

Effect of actuator information

We also trained a model with the pressure information of all the receptors and the three actuators. Compared with the model only using the receptor information, this model performed better in predicting pose under noncontact scenario but worse under random contact scenario. The performance of the model in response to receptor failures is shown in Figure 9A and B. Similar degradation in the performance could be observed under both the noncontact and random contact scenario. But the model using actuation information suffered less loss in prediction accuracy when receptor failures occurred. This indicated that the actuation information played an important role in proprioception and helped compensate the loss caused by receptor failures.

FIG. 9.

Effect of actuator information on pose prediction. (A) Noncontact scenario. (B) Random contact scenario.

Graceful degradation

Graceful degradation is the ability of a computer, machine, or electronic system to maintain at some reduced level of performance after a portion of its components fail. The purpose of graceful degradation, ideally, is to prevent complete system failure and reduce downtime. Graceful degradation is an important consideration in the design and implementation since soft robots might suffer damage from environment due to the “soft” property. Software and hardware methods could be combined to achieve graceful degradation.

Receptor failures would cause loss of partial input information. For a model trained by feeding the dataset with complete input information, it might fail to handle the scenarios with only partial input, thus causing the bad performance of prediction. We proposed a software level method to alleviate the impact of the partial input information loss. In addition to the model trained with pressure input of all four receptors (denoted as $M_{1234}$ ), we also pretrained models with pressure input of a part of receptors. Four pretrained models $M_{234}$ , $M_{134}$ , $M_{124}$ , $M_{123}$ were developed when pressure information of only three receptors was used, and six pretrained models $M_{34}$ , $M_{24}$ , $M_{23}$ , $M_{14}$ , $M_{13}$ , $M_{12}$ were developed when pressure information of only two receptors was used. Therefore, in total 11 pretrained models were available and saved for use in the task. Table 2 presents the performance of the pretrained models.

Table 2.

Performance of Pretrained Models for Graceful Degradation

Pretraining		Test performance (RMSE)
Input	Model	Noncontact			Random contact
Input	Model	L (mm)	$β$ (deg)	$α$ (deg)	L (mm)	$β$ (deg)	$α$ (deg)
$(p_{1}, p_{2}, p_{3}, p_{4})$	$M_{1234}$	1.10 ± 0.02	0.62 ± 0.05	0.71 ± 0.06	2.33 ± 0.02	1.91 ± 0.19	1.66 ± 0.21
$(0, p_{2}, p_{3}, p_{4})$	$M_{234}$	1.11 ± 0.01	0.80 ± 0.38	0.89 ± 0.35	2.38 ± 0.04	2.04 ± 0.81	1.92 ± 0.45
$(p_{1}, 0, p_{3}, p_{4})$	$M_{134}$	1.10 ± 0.01	0.89 ± 0.22	0.73 ± 0.18	2.36 ± 0.01	2.11 ± 0.66	1.63 ± 0.41
$(p_{1}, p_{2}, 0, p_{4})$	$M_{124}$	1.10 ± 0.01	0.99 ± 0.24	1.03 ± 0.12	2.34 ± 0.02	2.59 ± 0.22	2.09 ± 0.14
$(p_{1}, p_{2}, p_{3}, 0)$	$M_{123}$	1.12 ± 0.02	0.87 ± 0.22	0.82 ± 0.21	2.31 ± 0.01	2.65 ± 0.12	2.14 ± 0.11
$(0, 0, p_{3}, p_{4})$	$M_{34}$	1.37 ± 0.04	2.16 ± 0.10	1.04 ± 0.02	2.54 ± 0.02	2.85 ± 0.18	1.95 ± 0.02
$(0, p_{2}, 0, p_{4})$	$M_{24}$	1.14 ± 0.01	1.01 ± 0.11	1.66 ± 0.08	2.34 ± 0.03	2.27 ± 0.13	2.62 ± 0.08
$(0, p_{2}, p_{3}, 0)$	$M_{23}$	1.82 ± 0.04	1.68 ± 0.12	2.08 ± 0.12	2.73 ± 0.08	2.68 ± 0.15	2.47 ± 0.17
$(p_{1}, 0, 0, p_{4})$	$M_{14}$	1.20 ± 0.02	2.18 ± 0.11	2.42 ± 0.14	2.59 ± 0.02	2.62 ± 0.10	3.05 ± 0.01
$(p_{1}, 0, p_{3}, 0)$	$M_{13}$	1.30 ± 0.01	3.35 ± 0.28	1.20 ± 0.06	2.31 ± 0.04	4.26 ± 0.15	1.49 ± 0.35
$(p_{1}, p_{2}, 0, 0)$	$M_{12}$	1.61 ± 0.07	1.49 ± 0.07	3.34 ± 0.26	2.66 ± 0.12	2.63 ± 0.09	3.92 ± 0.15

The data are shown as mean ± standard deviation. The bold values denote the mean.

Figure 10A shows a flow chart representing the algorithm for achieving graceful degradation based on the pretrained models. Starting with model $M_{1234}$ , the system was able to recover from a receptor failure by switching to the model which was trained without using the pressure information of the failed receptor. For example, when receptor 2 failure was detected by the system, the system could switch to use model $M_{134}$ . Figure 10B and C shows the scenario of receptor 2 failure and how the system responded to the receptor failure. After receptor 2 failure, the system performed poorly in predicting the pose if the system kept using model $M_{1234}$ while the system could maintain the performance by switching to model $M_{134}$ . This was because model $M_{134}$ was trained without using the pressure information of receptor 2; it could continue to run using the pressure information of receptors 1, 3, and 4 to predict the pose.

FIG. 10.

(A) Flow chart of graceful degradation based on pretrained models. (B) System response to receptor 2 failure without graceful degradation strategy. (C) System response to receptor 2 failure with graceful degradation strategy.

The hardware level solution for graceful degradation was to introduce redundancy in receptors. As shown in Table 2, with one receptor failure, the system could still function well based on the remaining three receptors using the pretrained models. But if the system was damaged with less than three receptors left, pretrained models had limited ability to recover. Thus, designing the soft system with redundant receptors would be practical and conducive to system robustness. In practical applications, software and hardware methods could be combined for graceful degradation.

Conclusion and Caveats

This article presented a novel scheme for endowing soft robots with proprioception using soft body encoding and deep learning tools. The concept was inspired by the structural and neurological basis of the human muscular system. We demonstrated the proposed concept on a 3-DoF bellows-driven continuum joint with redundant soft body receptors. Both LSTM and GRU architectures were designed to train the kinematic model for the 3-DoF continuum joint, mapping the sequential pressure readings to body pose. The accuracies of the models were validated under both noncontact and contact scenarios. We demonstrated that the model trained by combining both the receptor and actuator information was more robust in the face of receptor failures. In addition, the redundancy of the receptors contributed to maintaining system function in the event of receptor failures. Finally, both hardware and software level methods were proposed to allow the continuum joint to achieve graceful degradation after receptor failures.

The soft receptor concept presented in this study was demonstrated with bellows chambers. Such bellows structure was considered because of its high sensitivity in response to stretched or compressed deformation. However, the bellows structure showed anisotropic compliance. For example, the bellows chamber was not very sensitive to bending deformation. Thus, other potential designs of the chamber structure should be investigated in the future. But the concept of detecting deformation by measuring chamber pressure changes is generalizable.

Kinematic modeling was purely based on a learning approach in this work, without accurate characterization of the system and precision fabrication of the receptors. Although deep learning tools provide parallel paths to analytical modeling and have produced promising results, there remain several challenges to overcome and unknowns to explore. In this work, the RNNs could successfully learn the mapping between the pressure inputs and the pose of the continuum joint. However, the RNNs failed to identify some exact features of the system, such as the specific effects of the number and the layout of the receptors on the pose estimation. Therefore, if a new system was designed with different receptor configurations (different number or different layout of the receptors), data sampling and retraining of the RNNs would be required.

Footnotes

Author Disclosure Statement

No competing financial interests exist.

Funding Information

This work was supported, in part, by the Science, Technology and Innovation Commission of Shenzhen Municipality under grant no. ZDSYS20200811143601004, NSFC Grant 51975268, Guangdong Provincial Key Laboratory of Human-Augmentation and Rehabilitation Robotics in Universities, Southern Marine Science and Engineering Guangdong Laboratory (Shenzhen).

Supplementary Material

References

Taylor

. Proprioception. In: Encyclopedia of Neuroscience. (Binder MD, Hirokawa N, Windhorst U, et al. eds.) Springer: Berlin/Heidelberg; 2009; pp. 1143–1149.

Proske

, Gandevia

. The proprioceptive senses: Their roles in signaling body shape, body position and movement, and muscle force. Physiol Rev, 2012; 92(4):1651–1697.

Xie

, Domel

, An

, et al. Octopus arm-inspired tapered soft actuators with suckers for improved grasping. Soft Robot, 2020; 7(5):639–648.

Festo. BionicMotionRobot. 2007. Available from: https://www.festo.com/us/en/e/about-festo/research-and-development/bionic-learning-network/highlights-from-2015-to-2017/bionicmotionrobot-id_33307/ [Last accessed: March 3, 2023].

Rus

, Tolley

. Design, fabrication and control of soft robots. Nature, 2015; 521(7553):467–475.

Wang

, Totaro

, Beccai

. Toward perceptive soft robots: Progress and challenges. Adv Sci, 2018; 5(9):1800541.

Beccai

, Lucarotti

, Totaro

, et al. Soft robotics mechanosensing. In: Soft Robotics: Trends, Applications and Challenges: Proceedings of the Soft Robotics Week, April 25–30, 2016, Livorno, Italy (Biosystems and Biorobotics). (Laschi C, Rossiter J, Iida F, et al. eds.) Springer: New York, NY; 2017; pp. 47–56.

Bahramzadeh

, Shahinpoor

. A review of ionic polymeric soft actuators and sensors. Soft Robot, 2014; 1(1):38–52.

Wang

, Gao

, Lee

. Recent progress in artificial muscles for interactive soft robotics. Adv Mater, 2021; 33(19):2003088; doi: 10.1002/adma.202003088

10.

Wakimoto

, Misumi

, Suzumori

. New concept and fundamental experiments of a smart pneumatic artificial muscle with a conductive fiber. Sens Actuators A Phys, 2016; 250:48–54.

11.

Cianchetti

, Renda

, Licofonte

, et al. Sensorization of continuum soft robots for reconstructing their spatial configuration. In: 2012 IEEE RAS & EMBS International Conference on Biomedical Robotics and Biomechatronics (BioRob), Rome, Italy. IEEE; 2012; pp. 634–639.

12.

Kure

, Kanda

, Suzumori

, et al. Flexible displacement sensor using injected conductive paste. Sens Actuators A Phys, 2008; 143(2):272–278.

13.

Larson

, Peele

, Li

, et al. Highly stretchable electroluminescent skin for optical signaling and tactile sensing. Science, 2016; 351(6277):1071–1074.

14.

Atalay

, Sanchez

, Atalay

, et al. Batch fabrication of customizable silicone-textile composite capacitive strain sensors for human motion tracking. Adv Mater Technol, 2017; 2(9):1700136.

15.

Atalay

, Atalay

, Gafford

, et al. A highly stretchable capacitive-based strain sensor based on metal deposition and laser rastering. Adv Mater Technol, 2017; 2(9):1700081.

16.

Helps

, Rossiter

. Proprioceptive flexible fluidic actuators using conductive working Fluids. Soft Robot, 2018; 5(2):175–189.

17.

Russo

, Ranzani

, Liu

, et al. Soft and stretchable sensor using biocompatible electrodes and liquid for medical applications. Soft Robot, 2015; 2(4):146–154.

18.

Kramer

, Majidi

, Sahai

, et al. Soft curvature sensors for joint angle proprioception. In: 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems (ICRA), San Francisco, CA, USA. IEEE; 2011; pp. 1919–1926.

19.

Truby

, Wehner

, Grosskopf

, et al. Soft somatosensitive actuators via embedded 3D printing. Adv Mater, 2018; 30(15):1870106.

20.

Yang

, Chen

. Innovative design of embedded pressure and position sensors for soft actuators. IEEE Robot Autom Lett, 2017; 3(2):656–663.

21.

Hainsworth

, Smith

, Alexander

, et al. A fabrication free, 3D printed, multi-material, self-sensing soft actuator. IEEE Robot Autom Lett, 2020; 5(3):4118–4125.

22.

Scharff

RBN

, Doornbusch

, Doubrovski

, et al. Color-based proprioception of soft actuators interacting with objects. IEEE ASME Trans Mechatron, 2019; 24(5):1964–1973.

23.

Wang

, Zhang

, Chen

, et al. Shape detection algorithm for soft manipulator based on fiber Bragg gratings. IEEE ASME Trans Mechatron, 2016; 21(6):2977–2982.

24.

Ryu

, Dupont

. FBG-based shape sensing tubes for continuum robots. In: 2014 IEEE International Conference on Robotics and Automation (ICRA), Hong Kong, China. IEEE; 2014; pp. 3531–3537.

25.

Chitalia

, Deaton

, Jeong

, et al. Towards FBG-based shape sensing for micro-scale and meso-scale continuum robots with large deflection. IEEE Robot Autom Lett, 2020; 5(2):1712–1719.

26.

Chen

, Xiong

, Liu

, et al. Fabrication and dynamic modeling of bidirectional bending soft actuator integrated with optical waveguide curvature sensor. Soft Robot, 2019; 6(4):495–506.

27.

Zhao

, O'Brien

, Li

, et al. Optoelectronically innervated soft prosthetic hand via stretchable optical waveguides. Sci Robot, 2016; 1(1):eaai7529.

28.

Bai

, Li

, Barreiros

, et al. Stretchable distributed fiber-optic sensors. Science, 2020; 370(6518):848–852.

29.

Felt

, Remy

, Telleria

, et al. An inductance-based sensing system for bellows-driven continuum joints in soft robots. Auton Robot, 2019; 43(2):435–448.

30.

Croom

, Rucker

, Romano

, et al. Visual sensing of continuum robot shape using self-organizing maps. In: 2010 IEEE International Conference on Robotics and Automation (ICRA), Anchorage, AK, USA. IEEE; 2010; pp. 4591–4596.

31.

Camarillo

, Loewke

, Carlson

, et al. Vision based 3-D shape sensing of flexible manipulators. In: 2008 IEEE International Conference on Robotics and Automation (ICRA), Pasadena, CA, USA. IEEE; 2008; pp. 2940–2947.

32.

Wang

, Wang

, Du

, et al. Real-time soft body 3D proprioception via deep vision-based sensing. IEEE Robot Autom Lett, 2020; 5(2):3382–3389.

33.

Tawk

, Panhuis

, Spinks

, et al. Soft pneumatic sensing chambers for generic and interactive human–machine interfaces. Adv Intell Syst, 2019; 1(1):1900002.

34.

Tawk

, Zhou

, Sariyildiz

, et al. Design, modeling, and control of a 3D printed monolithic soft robotic finger with embedded pneumatic sensing chambers. IEEE ASME Trans Mechatron, 2020; 26(2):876–887.

35.

Yang

, Chen

, Sun

, et al. A novel pneumatic soft sensor for measuring contact force and curvature of a soft gripper. Sens Actuator A Phys, 2017; 266:318–327.

36.

Tenzer

, Jentoft

, Howe

. The feel of MEMS barometers: Inexpensive and easily customized tactile array sensors. IEEE Robot Autom Mag, 2014; 21(3):89–95.

37.

Chin

, Hellebrekers

, Majidi

. Machine learning for soft robotic sensing and control. Adv Intell Syst, 2020; 2(6):1900171.

38.

Kim

, Park

. Probabilistic modeling and Bayesian filtering for improved state estimation for soft robots. IEEE Trans Robot, 2021; 37(5):1728–1741.

39.

Han

, Kim

, et al. Use of deep learning for characterization of microfluidic soft sensors. IEEE Robot Autom Lett, 2018; 3(2):873–880.

40.

Fang

, Tian

, Yang

, et al. Efficient Jacobian-based inverse kinematics with sim-to-real transfer of soft robots by learning. IEEE ASME Trans Mechatron, 2022; 27(6):5296–5306; doi: 10.1109/TMECH.2022.3178303

41.

Van Meerbeek

, De Sa

, Shepherd

. Soft optoelectronic sensory foams with proprioception. Sci Robot, 2018; 3(24):eaau2489.

42.

Scharff

RBN

, Fang

, Tian

, et al. Sensing and reconstruction of 3-D deformation on pneumatic soft robots. IEEE ASME Trans Mechatron, 2021; 26(4):1877–1885.

43.

Sundaram

, Kellnhofer

, Li

, et al. Learning the signatures of the human grasp using a scalable tactile glove. Nature, 2019; 569(7758):698–702.

44.

Thuruthel

, Shih

, Laschi

, et al. Soft robot perception using embedded soft sensors and recurrent neural networks. Sci Robotics, 2019; 4(26):eaav1488.

45.

Soter

, Conn

, Hauser

, et al. Bodily aware soft robots: Integration of proprioceptive and exteroceptive sensors. In: 2018 IEEE International Conference on Robotics and Automation (ICRA), Brisbane, QLD, Australia. IEEE; 2018; pp. 2448–2453.

46.

Loo

, Ding

, Baskaran

, et al. Robust multimodal indirect sensing for soft robots via neural network-aided filter-based estimation. Soft Robot, 2022; 9(3):591–612.

47.

Sakurai

, Nishida

, Sakurai

, et al. Emulating a sensor using soft material dynamics: A reservoir computing approach to pneumatic artificial muscle. In: 2020 3rd IEEE International Conference on Soft Robotics (RoboSoft), New Haven, CT, USA. IEEE; 2020; pp. 710–717.

48.

Soter

, Hauser

, Conn

, et al. Shape reconstruction of CCD camera-based soft tactile sensors. In: 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Las Vegas, NV, USA. IEEE; 2020; pp. 8957–8962.

49.

Grünewald

, Yoneda

, Shipman

, et al. Idiopathic focal dystonia: A disorder of muscle spindle afferent processing. Brain, 1997; 120(12):2179–2185.

50.

Lephart

, Fu

. Proprioception and Neuromuscular Control in Joint Stability. Champaign, IL: Human Kinetics;. 2000.

51.

Wang

, Wang

. Mechanoreception for soft robots via intuitive body cues. Soft Robot, 2020; 7:198–217.

52.

Marchese

, Komorowski

, Onal

, et al. Design and control of a soft and continuously deformable 2D robotic manipulation system. In: 2014 IEEE International Conference on Robotics and Automation (ICRA), Hong Kong, China. IEEE; 2014; pp. 2189–2196.

53.

Chen

, Guo

, Duanmu

, et al. Design and modeling of an extensible soft robotic arm. IEEE Robot Autom Lett, 2019; 4(4):4208–4215.

54.

Hochreiter

, Schmidhuber

. Long short-term memory. Neural Comput, 1997; 9(8):1735–1780.

55.

Chung

, Gulcehre

, Cho

, et al. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv: 1412.3555, 2014.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.28 MB

0.02 MB

0.00 MB