Robust Multimodal Indirect Sensing for Soft Robots Via Neural Network-Aided Filter-Based Estimation

Abstract

Sensory data are critical for soft robot perception. However, integrating sensors to soft robots remains challenging due to their inherent softness. An alternative approach is indirect sensing through an estimation scheme, which uses robot dynamics and available measurements to estimate variables that would have been measured by sensors. Nevertheless, developing an adequately effective estimation scheme for soft robots is not straightforward. First, it requires a mathematical model; modeling of soft robots is analytically demanding due to their complex dynamics. Second, it should perform multimodal sensing for both internal and external variables, with minimal sensors, and finally, it must be robust against sensor faults. In this article, we propose a recurrent neural network-based adaptive unscented Kalman filter (RNN-AUKF) architecture to estimate the proprioceptive state and exteroceptive unknown input of a pneumatic-based soft finger. To address the challenge in modeling soft robots, we adopt a data-driven approach using RNNs. Then, we interconnect the AUKF with an unknown input estimator to perform multimodal sensing using a single embedded flex sensor. We also prove mathematically that the estimation error is bounded with respect to sensor degradation (noise and drift). Experimental results show that the RNN-AUKF achieves a better overall performance in terms of accuracy and robustness against the benchmark method. The proposed scheme is also extended to a multifinger soft gripper and is robust against out-of-distribution sensor dynamics. The outcomes of this research have immense potentials in realizing a robust multimodal indirect sensing in soft robots.

Introduction

Background

Inspired by the morphology of biological organisms, soft robots are designed to be flexible and compliant for adaptive operation in unstructured environments.¹ To advance into a fully autonomous system, perception is crucial as it provides the basis of informed decision-making. In soft robots, an example of a perceptive variable is the bending angle, which is useful in describing the bending of soft actuators for feedback control,² or external force, which is important because soft robots generally interact with their environment, including objects and obstacles in real applications.³ However, integrating sensors into soft robots to measure perceptive variables is not as straightforward as for their rigid counterparts, as soft robots have infinite degrees of freedom, and hence, many sensors may be needed to achieve proprioception (measure of self-movement and body position) and exteroception (measure of stimuli outside of the body).

At the same time, installing many sensors could alter the robots' mechanical characteristics and functionality, defeating the purpose of being soft. For example, attaching multiple strain and tactile sensors on a soft robot could stiffen it, thus limiting both its stretchability and reachability.⁴ Moreover, camera systems that track a soft robot's motion are not portable, difficult to set up, and require continuous unoccluded views. Consequently, these methods are impractical especially in unstructured environments, for example, search and rescue⁵ and minimally invasive surgery.⁶ Due to this difficulty, many soft robots are still open-loop systems.⁷

Recently, bioinspired sensor morphology⁸ and soft sensor development⁹ for soft robots have been a growing research area. For instance, careful integration of optical sensors,^10,11 embedded three-dimensional (3D)-printed sensors,^12,13 embedded microelectromechanical sensors,^14,15 and soft dielectric sensors¹⁶ has opened the possibilities for sensor integration in soft robots. Despite these advances, several challenges remain. Unlike traditional sensing technologies, soft sensors have very complicated dynamics as they conform to the surrounding structure, and thus, it is analytically demanding to fully describe the nonlinearity of the sensors for accurate sensing.⁴ Furthermore, in designing new soft sensors, it is challenging to convert the raw sensor data to meaningful perception information such as deformation, contact, and pressure.¹⁷

In addition, soft sensor fabrication and placement generally require significant human input and prior knowledge about the system.¹⁸ Moreover, it is costly to fabricate soft sensors with consistent sensor dynamics as achieving this requires an automated fabrication process in a highly controlled environment. In view of these challenges in integrating sensors to soft robots, a complementary approach is to adopt indirect sensing strategies through an estimation scheme, which incorporates both the sensor measurements and soft robot dynamics to estimate the variables of interest that would have been measured by physical sensors.

For any estimation scheme to work effectively, a model is required. Analytical modeling (based on mathematical equations) of soft robots is more complex compared with rigid robots,¹⁹ and therefore, previous efforts on analytical modeling typically include simplifications and assumptions in their formulations.²⁰ In the current literature, the kinematics in soft robots are mainly derived based on the piecewise constant-curvature approximation,^2,21,22 while many of the dynamics used either Euler/Lagrangian or Lagrangian formulation with quasistatic simplification.^23–26 These models may not accurately capture the complex nonlinear dynamics of a soft robot, in part, due to the simplicity of the models themselves.⁷

Consequently, these models do not necessarily guarantee a good estimation performance. A more complex model may solve this problem, but the process of deriving such models is tedious especially when the soft robot is made up of soft materials with a significantly low Young's modulus value. The aforementioned challenges can be tackled via data-driven (or machine learning) approaches.

With sufficient data, efficient data-driven methods such as deep neural networks can be trained to implicitly account for the nonlinear morphology and material properties of soft robots, without making too many assumptions. Table 1 summarizes the relevant state of the art of various data-driven approaches applied in soft robotics.

Table 1.

Summary of Literature Survey of the State of the Art on Data-Driven Approaches Applied in Soft Robotics

Robotic platform	Design						Modeling (learning)			Estimation (indirect sensing)						Focus of study	Is current state of implementation applicable	Remarks
	Actuation	Channel	Material	Deformation	Robotic function	Planar/nonplanar	Method	Inherent nonlinearity	Consider long-term memory	Method	Estimated values		Measured values		Mathematical proof
	Actuation	Channel	Material	Deformation	Robotic function	Planar/nonplanar	Method	Inherent nonlinearity	Consider long-term memory	Method	State	Unknown Input	Measurement	Sensor	Mathematical proof
Soft gripper²⁷	Pneumatic	Pleated	Silicone rubber, polypropylene	Bending	End-effector: grasping	Planar	Line segment model with parameter identification (nonlinear least squares)	No	No	Model prediction	Position of nodal points along actuator	Horizontal and vertical contact forces	Voltage reading	Embedded flex sensor	No	Data-driven modeling, parameter identification	Yes	Similar experiment setup
Fiber-reinforced bellow²⁸	Pneumatic	Cylindrical	Silicone rubber, fiberglass fabric	Bending	End-effector: grasping	Planar	Polynomial nonlinear state space model	Yes	No	Nonlinear least squares	Tangential angle and arc length of nodal points along actuator	External wrench, contact location	End-effector position and orientation	EM tracking/camera system	No	Data-driven modeling, contact detection	Yes
Soft continuum²⁹	Pneumatic	Pleated	Silicone rubber, paper	Bending	End-effector: grasping	Planar	Linear regression, neural network	Yes (neural network)	No	Model prediction	Aggregate bending angle	N/A	Voltage reading	Embedded flex sensor	No	Data-driven modeling, control	Yes
Soft finger¹⁸	Pneumatic	Pleated	Silicone rubber, cPDMS	Bending	End-effector: grasping	Planar	Long short-term memory	Yes	Yes	Model prediction	Tip and middle position of soft finger	Horizontal contact force	Voltage reading	Soft embedded strain sensor	No	Multimodal indirect sensing, sensor degradation	Yes
Soft silicone arm³⁰	Base rotation	N/A	Silicone rubber	Bending, twisting	Manipulator: reaching, tracking	Nonplanar	Reservoir computing network, NARMA, linear regression	Yes (reservoir computing, NARMA)	Yes (reservoir computing)	Model prediction	Missing bend sensory states	N/A	Available bend sensory states	Embedded flex sensor	No	Reservoir computing, dynamics exploitation	No	Require extending current sensor configuration or camera setup to capture non planar motion
Soft elastomer foam³¹	Base movement	N/A	Lost salt silicone foam, silicone rubber	Bending, twisting	End-effector: grasping	Nonplanar	kNN, SVMs, decision trees, Gaussian process, linear regression, neural network	Yes (Gaussian process, neural network)	No	Model prediction	Deformation mode, aggregate bending or twisting angle of actuator	N/A	Light intensity	Optical fibers	No	Sensor integration, machine learning	No
Soft arm³²	Pneumatic	Cylindrical	McKibben actuators, polyurethane foam	Bending, elongation, shortening	Manipulator: reaching, tracking	Nonplanar	Koopman operator, State space model, NARX	Yes (Koopman operator, NARX)	No	Koopman model predictive controller	Koopman lifted position	N/A	Laser dot position	Camera system	No	Koopman system identification, model predictive control	No
Soft finger (Our Work)	Pneumatic	Pleated	Silicone rubber, paper	Bending	End-effector: grasping	Planar	Long short-term memory, gated recurrent unit	Yes	Yes	Adaptive unscented Kalman filter	Segmental bending angles of adjacent nodal points along soft finger	Horizontal and vertical contact forces	Voltage reading	Embedded flex sensor	Yes	Multimodal indirect sensing, sensor degradation	N/A	N/A

The survey classifies these approaches based on the soft robot designs and functions, types of modeling and estimation involved, and their focus of study.

cPDMS, polydimethylsiloxane (PDMS) impregnated with conductive carbon nanotubes; EM, electromagnetic; kNN, k-nearest neighbors; N/A, not applicable; NARMA, nonlinear autoregressive–moving-average; NARX, nonlinear autoregressive with external input; SVMs, support vector machines.

Initial research into data-driven approaches can be traced back to Elgeneidy et al.²⁹ who proposed using linear regression and neural networks to estimate the bending angles of a pneumatic actuator. Apart from that, Nakajima et al.³⁰ exploited the complex dynamics of a soft silicone arm for reservoir computing network and leverage it to perform time series prediction of the sensory values. In addition, Van Meerbeeket al.³¹ applied various machine-learning techniques to detect the type and magnitude of deformations in terms of twisting and bending of an internally illuminated soft elastomer foam. Recently, Thuruthel et al.¹⁸ trained a long short-term memory (LSTM) network as the kinematic and force model to estimate the tip position and contact force of the soft finger. Other data-driven approaches include that of Bruder et al.³² who identified the dynamical models of a soft arm using the Koopman operator theory for trajectory tracking via model predictive control.

These works in Table 1 demonstrated that data-driven approaches are suitable in addressing the issues of model complexity. Soft robots exhibit rich dynamics such as nonlinearity, elasticity, and high dimensionality due to the use of soft materials. From another perspective, it can be said that these rich dynamics are exploited as computational resources for information processing through these learning-based approaches.³⁰ While there are studies that focus on analyzing the dynamics of soft robots in this context,^30,33 a thorough review in this area such as Tanaka et al.³⁴ is not within the scope of the article. Despite these advantages, data-driven approaches in soft robotics have their own general limitations as well, such as platform-specific dependencies and susceptibility to long-term varying dynamics.

Apart from that, sensor faults in the form of noise and drift, commonly seen in conductive sensors, heavily degrade the performance of an estimation scheme. To mitigate the effect of sensor faults, a graceful degradation strategy was introduced in Thuruthel et al.,¹⁸ which involves embedding multiple sensors in their soft robot for redundancy; however, embedding multiple sensors may compromise a soft robot's inherent softness and consequently limiting its flexibility. To the best of our knowledge, the issue of robustness to sensor faults and anomalies in an estimation scheme is still relatively unexplored in the field of soft robotics.

As an alternative, a Kalman filter-based estimation scheme that processes sensor measurements based on a model could be useful for indirect sensing. If designed properly, the Kalman filter can automatically detect the occurrence of the sensor values becoming noisy faulty and place less dependence on the sensor measurements; this helps the estimation quality remain robust to the sensor noises/drift. However, there are only few works on Kalman filter-based estimation seen in soft robotics. Thieffry et al.³⁵ used a Luenberger observer on a reduced order steady-state model to estimate the position and velocity of a long soft beam. Lunni et al.³⁶ used an adaptive extended Kalman filter on a simplified quasistatic model to estimate the angle of rotation at each flexural joint of a tendon-driven soft robot. Loo et al.^37,38 proposed an extended Kalman filter (EKF) to estimate states in the actuator and configuration space of both planar and nonplanar pneumatic-based soft tentacles.

Nonetheless, these works did not address the issue of multimodal sensing, that is, the ability to perceive several stimuli simultaneously such as the internal and external one, which is not straightforward to achieve. External stimuli such as contact forces are unknown inputs to the system, and to the best of our knowledge, estimating them requires stringent conditions. A possible solution is to integrate the Kalman filter with an additional component such as an optimizer for unknown input estimation. In the field of structural control,^39,40 combined Kalman filter with least-square optimization to estimate the unknown input in nonlinear systems, but for a specific class of systems in which the measurement model is a function of the unknown input.

Contributions

As seen in Table 1, most of the literature used inherently nonlinear data-driven (modeling) approaches to model their respective soft elastomer robots. These types of soft robots exhibit highly complex and nonlinear dynamics,⁷ through which a nonlinear data-driven approach is the only feasible option available in accurately modeling such robots. However, only a few^18,30 considered modeling approaches with long-term dependencies (memory) to incorporate long-term effects such as drift and hysteresis, which are typically found in soft material robots and sensors.¹⁸

Similarly, on the aspect of estimation, few works considered multimodality (i.e., the ability to estimate both the state and unknown input). Most importantly, Table 1 highlights a research gap of addressing the susceptibility of estimation methods (such as direct model prediction or optimization) to sensor faults. To address these issues in soft robotic modeling and estimation, this article presents a novel recurrent neural network (RNN)-aided Kalman filter-based estimation scheme to achieve robust multimodal indirect sensing that is resilient to both sensor noise and drift.

Our contributions are highlighted as follows:

To account for complex and nonlinear dynamics, we model our soft robot using two types of RNNs, that is, LSTM⁴¹ and its variant, gated recurrent unit (GRU).⁴² LSTM and GRU are both state-of-the-art RNNs that are able to learn long-term dependencies (memory) such as the predominant drift and hysteresis effects found in soft robots and sensors.

We develop a Kalman filter-based estimation scheme where the unscented Kalman filter (UKF) is coupled with unknown input estimation to simultaneously estimate the internal state and external unknown input, thus achieving multimodality. The UKF is a statistical extension of EKF, which is known to be more effective for highly nonlinear systems,⁴³ while bypassing Jacobian linearization in EKFs, which is especially difficult for intricate RNNs.

We integrate a residual-based adaptation scheme to the Kalman filter (making it an adaptive unscented Kalman filter [AUKF]) to mitigate the effect of sensor faults (noise and drift) and out-of-distribution sensor dynamics via adaptive dependency on the sensor measurement. Most importantly, we also mathematically analyze and prove that the estimation errors of AUKF are exponentially bounded. The LSTM and GRU recurrent networks are incorporated into the AUKF as part of the novel estimation scheme, which hereafter are referred to as LSTM-AUKF and GRU-AUKF, respectively (or generally as RNN-AUKF).

Overview

To verify the viability of the proposed scheme on a soft robot with complex dynamics, it is applied to a pneumatic-based soft finger (PSF) with pleated air channels that performs planar bending, as shown in Figure 1, which is similar to the ones in Thuruthel et al.,¹⁸ Wang and Hirai,²⁷ and Elgeneidy et al.²⁹ Using measurements from a single embedded flex sensor that can be conveniently integrated to the PSF, our proposed RNN-AUKF scheme simultaneously estimates the bending angles (which is the internal state, thus achieving proprioception) and the contact or grasping forces (which is an unknown input, thus achieving exteroception).

FIG. 1.

Bending of the PSF with and without contact constraint. During pneumatic actuation, one side of PSF is made inextensible while the other pleated side with embedded air channels expands, resulting in unidirectional bending. The tip contact constraint is introduced to change the course of PSF deformation. PSF, pneumatic-based soft finger.

The performance of the proposed RNN-AUKF scheme is validated via case studies across the following platforms, as outlined in Supplementary Figure S1.

Single Pneumatic Soft Finger: Two case studies are considered on the single PSF platform to demonstrate the scalability of the proposed RNN-AUKF with respect to state space dimensionality. The first case study, Segmental Angles, models the nine individual segmental bending angles of a PSF as the system states. The second case study, Resultant Angle, aggregates the sum of the segmental angles of a PSF into a one-dimensional system state. These case studies include two experimental scenarios: Free Bending (free PSF actuation) and Tip Contact (PSF subjected to tip constraint) and three sensor settings: Normal Operation (raw sensor measurement), Adaptive Filtering (sensor subjected to intense noise), and Graceful Degradation (sensor subjected to drift) to verify the capability of RNN-AUKF in achieving multimodalilty and its robustness against sensor noise and drift.

Multifinger Soft Gripper: Using a fully functioning multifinger soft gripper platform, we also carried out a third case study, Multifinger Grasping, which investigates the robustness of the proposed RNN-AUKF against out-of-distribution sensor dynamics. The bending angles of the three PSFs that represent the gripper and the resultant grasping forces are concurrently estimated.

In comparing the performance of the RNN-AUKF against state-of-the-art recurrent networks, we used the straightforward (or direct) LSTM and GRU¹⁸ as the benchmarks. These networks are referred to here as LSTM-Direct or GRU-Direct, respectively (or generally as RNN-Direct). Results from the case studies suggest the following.

Single Pneumatic Soft Finger: The proposed RNN-AUKF achieves a comparable accuracy against the benchmark RNN-direct under Normal Operation. In detail, RNN-AUKF achieves a better overall accuracy in Adaptive Filtering and Graceful Degradation alongside lower noise levels for Adaptive Filtering. These results demonstrate that the proposed scheme is robust to sensor faults (noise and drift), with the guarantee that the estimation errors are exponentially bounded under these circumstances from a theoretical analysis.

Multifinger Soft Gripper: The proposed RNN-AUKF exhibits a higher accuracy against that of the benchmark RNN-Direct under inconsistent sensor dynamics. This in turn exemplifies its robustness against out-of-distribution sensor dynamics.

The rest of this article is organized as follows. The Materials section details the soft robot fabrication method and experimental procedures. The Methods section details the framework of the proposed RNN-AUKF estimation scheme. Results are presented in the Results section and further discussed in the Discussions section. The Conclusions section concludes the article.

Materials

In this section, we first describe the setup process starting from the soft finger fabrication, followed by the experiment procedure, and finally data sampling. These steps are essential as the sampled data are required for data-driven modeling.

Fabrication

The PSF considered in this article is a class of popular soft actuator known as PneuNet,⁴⁴ made of super-soft silicone rubber (Ecoflex 00–50; Smooth-On Inc.) with a shore hardness of 00–50, tensile strength of 315 psi, and 100% modulus of 12 psi. Figure 2 illustrates a cross-sectional view of the fabricated PSF. The PSF consists of 2 parts—a main body with a series of pleated embedded channels and chambers, and a base layer made by staking a flex sensor onto an inelastic material (paper).

FIG. 2.

Cross-sectional illustration of the PSF. The PneuNet design with pleated pneumatic channels is chosen. The base layer is made inextensible with an embedded paper. A flex sensor is placed beneath the paper to provide sensory information on the PSF bending state.

We first cast silicone using a 3D-printed mold to form the main body and the base layer. After that, we embed a flex sensor (4.5 inches, SparkFun) in the base layer together with a piece of paper (inelastic material) to make the layer inextensible. Then, the base layer and the main body are joined together using additional silicone. Finally, we solder the connections of the flex sensor and connect the PSF with a sharp-end pneumatic pipe as air inlet. Figure 1 demonstrates the PSF bending under pneumatic actuation, with and without tip contact.

To expand the experiment setup into a multifinger soft gripper, we repeat the outlined steps to replicate two additional PSFs. However, the fabrication process of manually replicating the PSFs and the use of soft materials resulted in slight variations such as asymmetrical air columns. In addition, the flex sensor embedded in each of the PSFs exhibited inconsistent (out-of-distribution) sensor dynamics, resulting in different sensor responses to the same PSF bending configuration.

Such variability is not uncommon as the task of achieving consistent dynamics for sensors would typically require an automated fabrication process in a highly controlled environment, which is costly and may not be economically practical. Nevertheless, in the Multifinger Grasping case study that follows (i.e., the Multifinger Soft Gripper Platform section), the proposed estimation scheme compensates for this variability in sensor dynamics and therefore sustains a reliable indirect sensing.

Experimental setup and procedure

Our pneumatic system consists of a pneumatic supply connected in series to an electropneumatic regulator (ITV1030; SMC Corporation). The modulated supply pressure is then connected to the PSF inlet. A microcontroller (PSoC^®5LP) is used to control the electropneumatic regulator, which in turn controls the supply pressure to the PSF inlet. We measure the output voltage (indicating the modulated supply pressure level) from a pressure sensor and use it as system input data to reflect the presence of any noise and delay in the modulated supply pressure being fed into the PSF.

To capture the PSF motion in terms of segmental angles or resultant angle, we place 10 reflective camera markers along the PSF's inextensible base layer. The first one is placed on the PSF holder as reference, and the others are aligned evenly onto the PSF body along the base layer line. During actuation, four cameras (OptiTrack Flex 13, NaturalPoint Inc.) are used to track the marker coordinates. In addition, a sensing circuit is used to convert the change in resistance of the embedded flex sensor to voltage readings without analog filtering or amplification. To measure contact and grasping forces, we use a multiaxis load cell (Axia80; ATI Industrial Automation Inc.) with a contact bulb attached to it. Figure 3 illustrates the circuitry of pneumatic control and data sampling.

FIG. 3.

Pneumatic control and data sampling circuits. Circuitry of the pneumatic controller and connection of the sensors for data sampling. The PSoC and the electropneumatic regulator control the supply pressure to the PSF. All the sensing circuits are connected to the Arduino for data sampling. PSoC, programmable system-on-chip.

The experimental setup and procedure on the two soft robot platforms are described in the following.

1. Single Pneumatic Soft Finger Platform: The PSF is fixed onto one of the angled slots of the PSF holder, as shown in Figure 4A and B. Then, it is actuated with a series of pseudorandom pressure levels (i.e., computer-generated numbers with statistical randomness), within a predetermined pressure range. To verify the applicability of the approach to a wide variety of input signals, two types of input voltage signals are fed to the electropneumatic regulator to generate a gradual oscillatory pattern (Oscillatory Actuation) and a faster random pressure actuation pattern (Random Actuation). The intermittent delay between each pressure level is also pseudorandom, with shorter delays in Random Actuation than Oscillatory Actuation.

FIG. 4.

(A) Experimental setup for single PSF platform. Three cameras are placed in front of the PSF to capture its planar motion during data sampling. A robot arm is used to maneuver the position of PSF. (B) Camera views on the single PSF. (Left) A contact bulb is placed near the tip to measure the contact forces exerted by the finger tip. (Right) The finger is allowed to bend freely. (C) Experimental setup for multifinger soft gripper platform. The gripper consists of three PSFs fixed 120 degrees apart from each other. A contact bulb is placed in the center of the three PSFs to measure the resultant grasping forces. (D) Top view of the multifinger soft gripper. Section D-D plane shows the top view of the multifinger soft gripper. (B–D) Blue arrows indicate two-axis (x, z) PSF contact forces exerted by a single PSF, whereas red arrows indicate the three-axis (x, y, z) resultant grasping forces exerted by the compounded multifinger soft gripper. Color images are available online.

In each actuation pattern, we considered two different scenarios. In the first scenario (Free Bending), the PSF is allowed to perform free bending without any obstruction, as shown in Figure 4B (Right). In the second scenario (Tip Contact), the contact bulb is placed in front of the fingertip to mimic a surface contact, as shown in Figure 4B (Left). The PSF fingertip is configured to move in and out (along x-axis), up and down (along z-axis) with respect to the contact bulb. This is to simulate grasping in an unstructured environment where the grasping object could be of any shape and size. Rather than repeatedly changing the shape and size of the contact bulb during data sampling, it is easier to move the PSF around to simulate varying contact instants and positions when grasping different objects.

The aforementioned experiments allow us to thoroughly study the ability of the proposed scheme in achieving proprioception and exteroception, that is, estimating both PSF shape and contact forces. Since the PSF bending occurs around the y-axis, the horizontal and vertical forces are measured on the x–z plane, as shown in Figure 4B (Left). Here, we assume that there is no twist in the PSF. The resulting PSF motion, as well as its tip movement under the aforesaid scenarios, is illustrated in Supplementary Movie S1.

2. Multifinger Soft Gripper Platform: To investigate the robustness of the proposed scheme against out-of-distribution sensor dynamics, we conduct experiments on a fully functioning multifinger soft gripper platform. The gripper consists of three PSFs fixed onto the angled slots of the PSF holder, which set them 120 degrees apart from each other, as illustrated in Figure 4C and D. Note that rotating the gripper's PSFs along the z-axis (as illustrated in Fig. 4D) does not change its gravitational effect. Hence, the gripper's PSFs can be arbitrarily placed about the z-axis to fit the scope of its application.

Due to occlusion of the reflective camera markers, our current camera setup is only able to capture the two-dimensional (2D) position. Hence, we attach camera markers specifically on PSF A, shown in Figure 4C. The contact bulb is placed in the center of the three PSFs to measure the resultant grasping forces. Note that there are three force axes (x, y, z) involved here due to the arrangement of PSFs on the gripper, as shown in Figure 4C and D. Here, a larger contact bulb is used to increase grasping surface. Similarly, the three PSFs are actuated with identically synchronous oscillatory or random input pressures (Oscillatory Actuation and Random Actuation) to grasp the contact bulb. The operations of the multifinger soft gripper are illustrated in Supplementary Movie S1.

Data sampling

For data sampling of the first platform, that is, the single PSF, the sensors (pressure, flex, and load cell) are connected to an Arduino board, whereas the marker coordinates are recorded using OptiTrack's Motive and NatNet SDK. The data consisting of input pressure, two-axis (x, z) contact forces, flex sensor voltage, and marker coordinates are sampled at a rate of 10 Hz using MATLAB. These data are an even mixture of the two scenarios (Free Bending and Tip Contact) and two actuation patterns (Oscillatory Actuation and Random Actuation) discussed earlier so as not to impart any bias to either of the scenarios during model training.

In addition, we normalize (scale to having a range of $0 - 1$ ) the flex sensor measurement according to its resistance range. All data are also standardized to have zero mean and unit standard deviation. Then, we partition the data into training, validation, and testing sets. These data sets are used, respectively, for model training, model selection, and ground truth. For each scenario, there are 12,000, 3000, and 3000 collective data points on each actuation pattern for training, validation, and testing, respectively.

During the testing phase, we replace the camera and the load cell setup with the proposed estimation scheme to represent actual soft robot indirect sensing, using only the pressure and flex sensor readings as input information. In this context, we only refer to the marker coordinates and the force data as the ground truth when assessing the accuracy and noise level of the estimation results.

As for the multifinger soft gripper platform, input pressures to the three PSFs and flex sensor readings of the three embedded flex sensors, as well as the marker coordinates on PSF A, are sampled for ground truth. Similarly, we normalize measurements of all three flex sensors so that they share the same sensing range. Also, we sample the three-axis (x, y, z) resultant grasping forces measured by the load cell as ground truth. Hence, there are 3000 collective data points sampled on each actuation pattern (oscillatory or random), used only as ground truth during testing. All these data are standardized.

At some points during the experiment, the applied random pressure actuation patterns result in an intermittent bulging of the PSF. This condition is not uncommon and has been observed in literature for pneumatic actuators.⁴⁵ Therefore, we continued running the experiments without interruption and the data sampled under this condition are fed into the model training. Despite the intermittent bulge, results in subsequent sections show that the models train well, and the estimations remain accurate.

Methods

In this section, we first outline a planar bending model to characterize the soft finger deformation (bending) in terms of segmental and resultant bending angles. Then, a neural network approach for data-driven modeling of the soft finger is presented, followed by the network structures and training methods. Finally, a novel estimation scheme that incorporates the neural network models with an AUKF is proposed.

Planar bending characterization

To obtain state data from the single PSF in the first platform for model training, we first convert the sampled markers' coordinates into bending angles. Here, we apply a line-segment method²⁷ to model the dynamics of the PSF bending with respect to segmental bending angles, that is, the angle between any two consecutive segments connecting adjacent nodal points (points with reflective camera marker attached). To demonstrate the scalability of the proposed scheme with respect to state space dimensionality, we considered two case studies: Segmental Angles and Resultant Angle. In Segmental Angles, we model the 9 individual segmental angles, $ϑ_{i}$ (as illustrated in Fig. 5A) as the system states. In Resultant Angle, we take the aggregate sum of the segmental angles, $θ$ from (1) as a one-dimensional system state instead. Each of these methods uniquely characterizes the PSF bending.

FIG. 5.

(A) Planar line segment model to characterize PSF bending in term of segmental angles. p_i = [x_i z_i]T is the coordinates of $i^{t h}$ marker on the bending plane of the PSF and $ϑ_{i}$ is the offset angle between consecutive segments (referred as segmental angle). (B) Illustration of PSF planar bending in task space. Idle position (without pressure actuation) of the PSF is shown in green. The blue spectrum indicates the movement of the PSF during free bending, whereas the red spectrum occurs when there is contact at the tip. (A, B) Black solid circles represent the nodal points (points with reflective camera marker attached) of the PSF. Color images are available online.

Figure 5A shows the planar line segment model and Figure 5B illustrates the PSF bending in task space. Taking the PSF bending curve (along the inextensible base layer line) as a series of n line segments, the segmental and resultant bending angles are calculated from nodal point coordinates as follows.

Let the displacement vector between two consecutive nodal points be $[\begin{matrix} X_{i} \\ Z_{i} \end{matrix}] = [\begin{matrix} x_{i + 1} \\ z_{i + 1} \end{matrix}] - [\begin{matrix} x_{i} \\ z_{i} \end{matrix}] i = 1, \dots, n - 1$

where [x_i z_i]^T denotes the 2D coordinates of an individual nodal point on the bending plane of the PSF. Then, the angle offset between any two consecutive segments (referred to as segmental angles) can be calculated as follows: $ϑ_{i} = a r c t a n (\frac{N_{i}}{M_{i}})$

where $N_{i} = \{\begin{matrix} Z_{i} & i = 1 \\ Z_{i - 1} X_{i} - X_{i - 1} Z_{i} & i = 2, \dots, n - 1 \end{matrix}$

M_{i} = \{\begin{matrix} X_{i} & i = 1 \\ X_{i - 1} X_{i} + Z_{i - 1} Z_{i} & i = 2, \dots, n - 1 \end{matrix}

For accurate bending curve reconstruction, we use $n = 10$ nodal points (i.e., reflective camera markers) and calculate $n - 1 = 9$ PSF's segmental bending angles from these nodal points.

Finally, the aggregate sum of the individual segmental angles (referred to as the resultant angle) is calculated as follows:

A segmental angle approach is able to capture the bending configuration of the PSF with minimal loss of information. However, this method gives rise to an unobservable system, through which the mapping from a nine-dimensional configuration space (segmental angles) to a one-dimensional sensor space (flex sensor reading) is nonbijective (i.e., different segmental angles result in the same flex sensor reading). Although this mapping may be bijective in either one of the Free Bending and Tip Contact scenarios, the same cannot be implied when both scenarios are considered. Consequently, the unobservability of a system could adversely affect the performance of the AUKF or any state observer in general. Measuring each segmental angle could resolve this unobservability. However, this method requires additional sensor integration into a PSF or an external camera setup to continuously capture the marker points, which may compromise a soft robot's inherent softness or limits its applicability in an unstructured environment.

To address the problem of unobservability without having to rely on additional sensors or an external camera setup, we introduce a residual-based model switching strategy to the proposed estimation scheme in the Data-Driven Modeling via Neural Networks section. In comparison, the resultant angle approach represents an efficient approach to characterize the current flex sensor, which outputs a single aggregate value based on the degree-of-bend, although losing the local angle information from the PSF segments.

Data-driven modeling via neural networks

In Wang and Hirai,²⁷ the equation of motion that represents the dynamics is formulated using a lumped-parameter Lagrangian approach. However, an energy-based formulation such as Lagrangian usually results in a mathematically complex model that involves tedious analytical derivation, with lumped model parameters identified through experimental data. Hence, in this article, we modeled the PSF dynamics through a machine-learning approach, specifically using RNNs.

The PSF nonlinear dynamics can generally be represented as follows: $ẋ = ϕ (x, u_{1}, u_{2}), y = ω (x)$ (2)

where x is the system state (bending angles, i.e., segmental angles $ϑ_{i}$ or resultant angle $θ$ ), u₁ is the known system input (PSF input pressure), u₂ is the unknown system input (tip contact forces), and y is the measurement output (flex sensor reading).

The continuous models (2) can then be discretized to obtain the following discrete-time models: $x_{k} = Φ (x_{k - 1}, x_{k - 2}, u_{1_{k - 1}}, u_{1_{k - 2}}, u_{2_{k - 1}}, u_{2_{k - 2}}, \dots)$ (3) $y_{k} = Ω (x_{k})$ (4)

where the length of dependencies on states and inputs of previous time-steps determines the order of the model, that is, the amount of “memory” preserved in the current state.

To incorporate this “memory” in (3), we use two types of RNNs, that is, LSTM⁴¹ and GRU,⁴² which are generally formulated as follows: $h_{k}^{x} = g_{x} (x_{k - 1}, u_{1_{k - 1}}, u_{2_{k - 1}}, h_{k - 1}^{x})$ (5)

where $h_{k}^{x}$ is the propagated memory state. In addition, a simple feedforward neural network is trained to convert $h_{k}^{x}$ to the current state, x_k, that is $x_{k} = ℱ_{x} (h_{k}^{x})$ (6)

followed by another feedforward neural network to map x_k to the sensor measurement, y_k, that is $y_{k} = ℱ_{y} (x_{k})$ (7)

Hence, the state model in (3) is modeled as the following RNN block $\begin{matrix} x_{k} = ℱ_{x} (h_{k}^{x}) \\ h_{k}^{x} = g_{x} (x_{k - 1}, u_{1_{k - 1}}, u_{2_{k - 1}}, h_{k - 1}^{x}) \end{matrix}$ (8)

whereas the measurement model in (4) is modeled as the following feedforward neural network $\begin{matrix} y_{k} = ℱ_{y} (x_{k}) \end{matrix}$ (9)

Besides, the unknown input can be estimated from optimization with an objective function as follows: $\begin{matrix} u_{2_{k - 1}} = {argmin}_{u_{2_{k - 1}}} ∥ x_{k} - Φ (x_{k - 1}, u_{1_{k - 1}}, u_{2_{k - 1}}, \dots) ∥^{2} \end{matrix}$ (10)

where x_k can be replaced by y_k but is more susceptible to measurement (sensor) noise and drift. In this scheme, rather than solving (10) from an optimization routine that can be computationally intensive, we use a separate RNN as follows: $h_{k}^{u} = g_{u} (x_{k}, u_{1_{k}}, h_{k - 1}^{u})$ (11)

where $h_{k}^{u}$ is the propagated memory state. Similarly, the memory state $h_{k}^{u}$ is converted to x_k using a feedforward neural network as follows: $u_{2_{k}} = ℱ_{u} (h_{k}^{u})$ (12)

Hence, the unknown input model representing (10) is modeled as the following RNN block: $\begin{matrix} u_{2_{k}} = ℱ_{u} (h_{k}^{u}) \\ h_{k}^{u} = g_{u} (x_{k}, u_{1_{k}}, h_{k - 1}^{u}) \end{matrix}$ (13)

which substitutes the optimization routine (10). Similarly, x_k can be replaced by y_k here.

The benchmark direct RNN model (RNN-Direct) predicts state and unknown input using the available known input and measurement as follows: $\begin{matrix} (x_{k}, u_{2_{k}}) = ℱ_{d} (h_{k}^{d}) \\ h_{k}^{d} = g_{d} (y_{k}, u_{1_{k}}, h_{k - 1}^{d}) \end{matrix}$ (14)

In the case of Resultant Angle, we observe that both the state model (8) and the unknown input model (13) would produce poor predictions. This is likely due to the fact that the same input pressure, $u_{1_{k}}$ , results in different resultant angles under different contact scenarios (Free Bending and Tip Contact), representing a nonbijective actuator space to configuration space mapping. To compensate this issue, the unknown input model would need to rely on the past memory state, $h_{k - 1}^{u}$ , for prediction. However, this reliance causes both the state and unknown input model errors to accumulate over time. Consequently, a high Kalman gain would be required to correct this error, which in turn degrades the AUKF's robustness to sensor faults.

In addition, having segmental angles as system states raises the unobservability issue, which adversely affects the AUKF's performance (as described in the preceding subsection). Considering the aforementioned challenges with both segmental and resultant angles, in the A Novel Neural Network-Aided Filter-Based Estimation Scheme section, we introduce a residual-based model switching strategy between two unknown input models that take state and measurement as model input, respectively. In this strategy, when the adaptation scheme detects that sensor fault is low, the unknown input model with measurement input is chosen, and vice versa. The advantage of this strategy is twofold; it alleviates (1) model error propagation at low-dimensional state space and (2) unobservability issue at high-dimensional state space.

In the following sections, we illustrate how the neural network model blocks, (8), (9), (13), are trained and incorporated into a novel estimation scheme that combines neural networks with AUKF, which is then benchmarked against (14).

Neural network model training

To account for nonlinearity including drift and hysteresis effects typically found in soft robotic systems and conductive sensors,¹⁸ we adopted RNNs to model the nonlinear dynamics of our single PSF in the first platform. A typical RNN comprises 2 parts—a recurrent cell that maps the current input and current memory state to the next memory state, and a feedforward module that maps the memory state to the desired output.⁴⁶

In this article, the recurrent cells we used are the powerful LSTM and its variant GRU for the state model $g_{x} (\cdot)$ , unknown input model $g_{u} (\cdot)$ , and direct model $g_{d} (\cdot)$ , whereas $ℱ_{x} (\cdot)$ and $ℱ_{u} (\cdot)$ are the feedforward modules for $g_{x} (\cdot)$ and $g_{u} (\cdot)$ , respectively. We choose LSTM and GRU because they are widely considered the current state-of-the-art recurrent networks and effective in learning long-term dependency. As for the measurement model $ℱ_{y} (\cdot)$ , a simple fully connected feedforward neural network is used for nonrecurrent mapping between state (bending angles) and measurement (flex sensor reading).

In training, $g_{x} (\cdot)$ and $ℱ_{x} (\cdot)$ are trained together to map $(x_{k - 1}, u_{1_{k - 1}}, u_{2_{k - 1}}) \to x_{k}$ . As mentioned, $ℱ_{x} (\cdot)$ is a fully connected layer that multiplies the output of the LSTM or GRU model with a weight matrix to obtain the final state (bending angles) output. Similarly, the unknown input model that maps $(x_{k}, u_{1_{k}}) \to u_{2_{k}}$ is also a concatenation of $g_{u} (\cdot)$ and $ℱ_{u} (\cdot)$ . The same applies to the direct model, $g_{x} (\cdot)$ and $ℱ_{x} (\cdot)$ , which maps $(y_{k}, u_{1_{k}}) \to (x_{k}, u_{2_{k}})$ .

For both case studies (Resultant Angle and Segmental Angles) in the single PSF platform, the number of hidden states used is consistently 128 in all LSTM and GRU network models. The feedforward measurement model $ℱ_{y} (\cdot)$ is also a feedforward layer with consistent 128 hidden neurons. Similarly, all neural network models are trained with the same training parameters that include an initial learning rate of 0.001, L₂ regularization factor of 0.001, training sequence length of 100, minibatch size of 8, and maximum training epochs of 300.

A standard Adam optimizer with default parameters is used. Adaptive learning rates and early stopping are also used to enhance training performance. The models are trained on CPU (Intel(R) Core(TM) i7-8750H) using MATLAB's Deep Learning Toolbox. For model selection, we trained 5 variants for each model by initializing the model weights using five different random seeds. Based on the performance on validation data, the best models are selected to be used as the final estimation models for testing data.

To further characterize⁴⁷ the proposed scheme, the training loss, validation loss, and training time of the RNN-AUKF and RNN-Direct models are analyzed in this section. Table 2 tabulates this information for each neural network model. Since the proposed RNN-AUKF scheme comprises three submodels: state, unknown input, and measurement models, it requires a longer training time compared with that of the RNN-Direct method, which consists of only a single model. Nevertheless, the longer training time here is justifiable as the proposed scheme is robust against sensor faults as described further in the Results section.

Table 2.

Training Benchmark of Neural Network Models in the Proposed Recurrent Neural Network-Based Adaptive Unscented Kalman Filter and the Benchmark Recurrent Neural Network-Direct

		Resultant angle			Segmental angles
Method	Model	Training loss (RMSE)	Validation loss (RMSE)	Training time (s)	Training loss (RMSE)	Validation loss (RMSE)	Training time (s)
LSTM-AUKF	State	0.0518	0.0650		0.0593	0.0645
	Unknown input	0.3521	0.4071	347	0.3205	0.3874	434
	Measurement	0.2636	0.2600		0.1273	0.1613
GRU-AUKF	State	0.0466	0.0519		0.0621	0.0724
	Unknown input	0.3834	0.4293	341	0.3168	0.3900	427
	Measurement	0.2636	0.2600		0.1273	0.1613
	LSTM-Direct	0.2427	0.3446	195	0.2080	0.2496	199
	GRU-Direct	0.2754	0.3312	169	0.2884	0.3052	195

AUKF, adaptive unscented Kalman filter; GRU, gated recurrent unit; LSTM, long short-term memory; RMSE, root-mean-squared-error.

A novel neural network-aided filter-based estimation scheme

To incorporate the trained models into the Kalman filter framework, we take bending angles as system state, x_k, PSF input pressure as known system input, $u_{1_{k}}$ , tip contact forces as unknown system input, $u_{2_{k}}$ , and flex sensor reading as system measurement, y_k. Hence, state model (8), measurement model (9), and unknown input model (13) can be represented in state-space as follows:

$x_{k} = f (x_{k - 1}, u_{1_{k - 1}}, u_{2_{k - 1}}) + w_{k}$ (15) $y_{k} = h (x_{k}) + v_{k}$ (16) $u_{2_{k}} = g (x_{k}, u_{1_{k}}) + e_{k}$ (17)

where , , and . Hidden states of the recurrent networks are considered deterministic and are made implicit in the equations for brevity.

Here, w_k and e_k represent the state and unknown input model errors, whereas v_k represents the measurement faults. They are assumed to be mutually correlated and have the following covariances:

We propose an RNN-AUKF estimation scheme that consists of an AUKF upon (8) and (9), interconnected with (13), as shown in Figure 6. Compared with using a direct black-box RNN, the AUKF scheme outlined in this section enables adaptive estimation of the internal system states (bending angles) and external unknown input (contact forces) of the PSF that is robust to sensor faults. The UKF is a nonlinear extension of Kalman filters that integrates unscented transformation, where it serves as an effective estimator for highly nonlinear systems.^43,48 The unscented particles generating function $χ_{i, k} = s i g m a s (x_{k}, P_{k})$ are defined as follows:

FIG. 6.

Block diagram of the RNN-AUKF. The block diagram summarized the proposed RNN-AUKF outlined in (22)–(34). At each time-step k, the inputs to the RNN-AUKF are the system input (pressure), $û_{1_{k}}$ and the system measurement (flex sensor reading), $ŷ_{k}$ . RNN-AUKF outputs the optimal estimate of the system states (resultant angle or segmental angles), ${\hat{x}}_{k}$ and the unknown input (contact forces), $û_{2_{k}}$ . Unscented transformation is applied on state model, measurement model, and unknown input model. $χ_{i, k}$ , $γ_{i, k}$ , and $μ_{2_{i, k}}$ are the corresponding unscented particles of ${\hat{x}}_{k}$ , $ŷ_{k}$ , and $û_{2_{k}}$ , respectively. RNN-AUKF, recurrent neural network-based adaptive unscented Kalman filter; GRU, gated recurrent unit. Color images are available online.

$\begin{matrix} χ_{i, k}^{-} = x_{k} & i = 0 \\ χ_{i, k}^{-} = x_{k} + \sqrt{n + κ} {(\sqrt{P_{k}^{x x}})}_{i} & i = 1, \dots, n \\ χ_{i, k}^{-} = x_{k} - \sqrt{n + κ} {(\sqrt{P_{k}^{x x}})}_{i - n} & i = n + 1, \dots, 2 n \end{matrix}$ (19)

where $κ$ is a tuning parameter and is the $i^{t h}$ column of Cholesky decomposition of matrix $A$ .

First, we initialize the Kalman filter parameters as follows:

${\hat{P}}_{0}^{x x} = p_{0} \times I_{n}, \hat{Q} = q \times I_{n}, {\hat{R}}_{0} = 0.01 \times I_{m}$ (21)

where $P_{k}$ is the state error covariance. We use a hat accent $â$ to represent the estimated variable to distinguish from its actual value. Throughout the experiment, we set $p_{0} = 1$ , $q = 0.5$ , and subject ${\hat{R}}_{k}$ to adaptive adjustment. With that, the RNN-AUKF algorithm is outlined as follows:

UKF prediction phase

State model prediction is first performed on each of the state particles $χ_{i, k - 1}$ from (28) and the unknown input particles $μ_{2_{i, k}}$ from (29).

where ${\hat{x}}_{k}^{-}$ is the mean (weighted by W_i) of the predicted (before Kalman correction) state particles $χ_{i, k}^{-}$ . The $κ$ -parameterized sampling weights⁴³ are given as $W_{i} = \frac{κ}{n + κ}$ for $i = 0$ and $W_{i} = \frac{1}{2 (n + κ)}$ for $i = 1, \dots, 2 n$ .

UKF correction phase

Kalman correction is then performed on the predicted mean state ${\hat{x}}_{k}^{-}$ .

where ${\hat{x}}_{k}$ is the weighted mean of the corrected (after Kalman correction) state particles $χ_{i, k}$ .

Unknown input estimation

Unknown input estimation is performed using either the corrected state particles $χ_{i, k}$ or sensor measurement y_k. $μ_{2_{i, k}} = g_{x} (χ_{i, k}, u_{1_{k}}) o r g_{y} (y_{k}, u_{1_{k}})$ (29)

where $û_{2_{k}}$ is the weighted mean of the estimated unknown input particles $μ_{2_{i, k}}$ . Here, we apply an adaptive model switching strategy to overcome the unknown input error propagation in the Resultant Angle case study and the unobservability in the Segmental Angles case study, as mentioned in the preceding subsections. When the estimated measurement covariance exceeds a threshold, that is, ${\hat{R}}_{k} > σ$ , we use $g_{y} (\cdot)$ , which receives measurement y_k as the input. When ${\hat{R}}_{k} \leq σ$ , we use $g_{x} (\cdot)$ , which receives state ${\hat{x}}_{k}$ as input instead, which is robust against sensor faults. Throughout the experiment, we set $σ = 1$ for Resultant Angle and $σ = 0.5$ for Segmental Angles.

Adaptive measurement covariance adjustment

For optimal UKF performance against sensor faults, the measurement covariance ${\hat{R}}_{k}$ is adaptively estimated via a residual-based method.

${\hat{R}}_{k + 1} = (1 - ρ) {\hat{R}}_{k} + ρ (ỹ_{k} ỹ_{k}^{T} + Ŝ_{k}^{y y})$ (34)

where $ỹ_{k} = y_{k} - ŷ_{k}$ and $0 \leq ρ \leq 1$ are the forgetting factor. A smaller $ρ$ increases the weight on previous estimates and thus incurs less fluctuation on ${\hat{R}}_{k}$ , but takes longer time for sensor faults to take effect. A factor of $ρ = 0.01$ is throughout the experiment here for slow mixing.

To sum up, the RNN-AUKF (20)–(34) is summarized in Figure 6 and its process flow is shown in Supplementary Figure S2. Above all, a stability analysis (given in Supplementary Data S1) is conducted to show that the estimation errors of the proposed RNN-AUKF for both states and unknown inputs are exponentially bounded in mean square.

Results

This section analyzes the results of multimodal estimations on two platforms that represent the case studies in this article (i.e., single PSF for the first two case studies and a multifinger soft gripper for the third case study). In each case study, the performance of the proposed RNN-AUKF (i.e., LSTM-AUKF and GRU-AUKF) scheme is compared against that of the RNN-Direct (i.e., LSTM-Direct and GRU-Direct), which serves as a benchmark. The analyses in this section are based on the normalized root-mean-squared-error (NRMSE) and signal-to-noise ratio (SNR) metrics.³¹ NRMSE (calculated from normalizing root-mean-squared-error (RMSE) with the range of the testing data) is used so that a total average can be taken across the bending angle and contact force estimation results, shown in Table 3 and Supplementary Table S1.

Table 3.

Comparisons of the Proposed Recurrent Neural Network-Based Adaptive Unscented Kalman Filter and the Benchmark Recurrent Neural Network-Direct for the Segmental Angles Case Study on the Single Pneumatic-Based Soft Finger Platform

		Sum of segmental angles					Contact forces
Sensor setting	Method	Free bending		Tip contact			Free bending		Tip contact
		Oscillatory actuation	Random actuation	Oscillatory actuation	Random actuation		Oscillatory actuation	Random actuation	Oscillatory actuation	Random actuation
				Accuracy: normalized RMS error (NRMSE %)		Average					Average	Overall
Normal operation	LSTM-AUKF	4.8049	4.2243	4.9969	4.9546	4.7452	0.7061	0.6934	6.3216	5.2966	3.2544	3.9998
	LSTM-Direct	3.2421	4.0542	5.3619	3.4818	4.0350	0.9433	1.0009	6.5996	5.4540	3.4995	3.7672
	GRU-AUKF	4.6341	5.4839	7.0879	5.9481	5.7885	1.8481	2.0449	6.8596	5.5372	4.0725	4.9305
	GRU-Direct	3.7046	5.1784	6.2564	4.7815	4.9802	0.8007	1.2256	7.4655	5.9438	3.8589	4.4196
Adaptive filtering	LSTM-AUKF	5.0201	4.8950	5.4881	5.6051	5.2521	0.6752	1.2963	6.8783	6.9764	3.9566	4.6043
	LSTM-Direct	6.6597	8.1375	6.5365	6.1984	6.8830	2.2254	3.8364	7.0454	6.6171	4.9311	5.9071
	GRU-AUKF	4.9501	5.7735	7.8666	6.4163	6.2516	1.6540	1.8446	7.8302	7.3870	4.6790	5.4653
	GRU-Direct	5.7167	6.1342	7.3051	8.1423	6.8246	2.3455	2.4749	7.8456	7.3201	4.9965	5.9106
Graceful degradation	LSTM-AUKF	6.1506	6.1531	5.5489	5.7125	5.8913	0.7796	2.7249	6.4359	6.7133	4.1634	5.0274
	LSTM-Direct	10.9739	13.0547	6.4916	7.9599	9.6200	5.5061	5.4101	7.5730	7.4289	6.4795	8.0498
	GRU-AUKF	6.3611	6.6065	7.5362	7.5103	7.0035	1.6916	2.9951	6.9662	7.1242	4.6943	5.8489
	GRU-Direct	9.9522	13.3852	6.5664	6.8957	9.1999	6.5373	6.7315	7.3897	6.7243	6.8457	8.0228
			Noise level: signal-to-noise ratio (SNR dB)
Adaptive filtering	LSTM-AUKF	36.0257	32.0972	32.3819	29.9628	32.6169	6.4245	5.1982	20.6144	14.3062	11.6358	22.1264
	LSTM-Direct	23.6819	22.2785	25.7336	21.4496	23.2859	3.3424	7.6539	23.8647	17.0473	12.9771	18.1315
	GRU-AUKF	35.6485	32.7739	32.8537	29.3886	32.6662	3.9326	4.2534	17.5115	12.9976	9.6738	21.1700
	GRU-Direct	24.5646	23.4222	26.4733	21.0661	23.8816	−1.1413	0.1916	22.7275	14.7113	9.1223	16.5019

Estimations are performed under four scenarios: Free Bending with Oscillatory Actuation, Free Bending with Random Actuation, Tip contact with Oscillatory Actuation, and Tip contact with Random Actuation. Overall NRMSEs and SNRs (shown in the last column) are obtained from averaging the results for sum of segmental angles and contact forces across all scenarios.

NRMSE, normalized root-mean-squared-error; RMS, root-mean-squared; SNR, signal-to-noise ratio.

Single pneumatic soft finger platform

As outlined in Supplementary Figure S1, our experiment on the single PSF platform considered two PSF scenarios: Free Bending and Tip contact, with each scenario conducted using two actuation patterns: Oscillatory Actuation and Random Actuation. To examine the robustness of the proposed RNN-AUKF scheme against sensor faults (noise and drift), our results considered three sensor settings: Normal Operation that uses the raw flex sensor reading, Adaptive Filtering where the sensor is subjected to intense white noise, and Graceful Degradation where the sensor is subjected to negative drift.

Multimodal (PSF bending angles and two-axis [x, z] contact forces) estimation results under the aforesaid PSF scenarios, actuation patterns, and sensor settings are obtained for each of the case studies (Segmental Angles and Resultant Angle). Table 3 compares these estimation results of the proposed RNN-AUKF and the benchmark RNN-Direct for Segmental Angles using the NRMSE and SNR metrics. The results for case study Resultant Angles are included in Supplementary Table S1.

Figure 7 shows a time-series estimation analysis specifically on LSTM-AUKF and LSTM-Direct for Segmental Angles. This figure also shows the ground-truth data for comparison analysis. The estimated segmental angles are transformed into 2D task space coordinates (on the bending plane of PSF) using a standard kinematic transformation. The task space coordinates are then used to fully reconstruct the PSF shape. Figure 8 visualizes the estimation of the task space coordinates together with the magnitude of external contact forces for the RNN-AUKF, RNN-Direct, and ground truth. A full-length visualization of this figure is captured in Supplementary Movie S1.

FIG. 7.

Time-series estimation analysis of the proposed LSTM-AUKF, the benchmark LSTM-Direct, and the ground truth for the Segmental Angle case study on the single PSF platform. The flex sensor reading of each scenario (in the $1^{s t}$ row) is first partitioned into three equal sections. Then, sensor noise in the second section is intensified to an SNR of 10. Negative drift of 1%/s (of flex sensor's range) is added for the last section. Estimations are then performed under four scenarios: Free Bending with Oscillatory Actuation, Free Bending with Random Actuation, Tip contact with Oscillatory Actuation, and Tip contact with Random Actuation. End-position coordinates x and z of the PSF (in $3^{r d}$ and $4^{t h}$ row) are obtained via transforming the estimated segmental angles. LSTM, long short-term memory; SNR, signal-to-noise ratio. Color images are available online.

FIG. 8.

Visualization for reconstruction of the single soft finger. PSF shape reconstructions of the proposed LSTM-AUKF, the benchmark LSTM-Direct, and the ground truth are compared under four scenarios: (Top Left) Free Bending with Oscillatory Actuation, (Top Right) Free Bending with Random Actuation, (Bottom Left) Tip contact with Oscillatory Actuation, and (Bottom Right) Tip contact with Random Actuation. Inset figure for each scenario shows the actual PSF configuration (Left) and the reconstructed bending curves (Right) at one instance. Trajectories of the end-position coordinates x and z (across time) under Graceful Degradation (in the presence of sensor drift) are illustrated. The vector plots illustrate the two-axis contact forces under Tip contact using arrows in two-dimensional spherical coordinates. A video (Supplementary Movie S1) showing the real-time reconstructions is given in Supplementary Data S1. Color images are available online.

The Segmental Angles and Resultant Angle case studies uniquely characterize the bending deformation of the PSF and therefore are studied in this article. Despite this unique characterization, results from analyzing these modeling approaches demonstrate that the proposed RNN-AUKF scheme consistently performs better than the benchmark method in Adaptive Filtering and Graceful Degradation settings. Considering the consistent performance between them and for the purpose of brevity, the remainder of this section focuses on analyzing the estimation results of Segmental Angles under the three sensor settings.

1. Normal Operation Setting: Raw sensor measurements are used for this estimation. Based on the tabulated results in Table 3, the proposed RNN-AUKF and the benchmark RNN-Direct yield comparable overall accuracy across all scenarios. Inference results using either RNN-AUKF or RNN-Direct are statistically insignificant. In addition, it is observed that RNN-Direct possesses intrinsic filtering properties. Hence, both RNN-AUKF and RNN-Direct achieve accurate estimation despite the fact that the raw sensor measurements contain moderate noise. Nevertheless, when comparing the LSTM and GRU models, LSTM performs better as it preserves memory better than GRU (owing to its cell state).⁴¹ This indicates that long-term dependency plays an important role in modeling the PSF dynamics.

2. Adaptive Filtering Setting: Sensor measurements for each scenario are first divided into three equal sections. Then, white noise in the sensor readings is progressively intensified in its magnitude, to yield collective SNRs of 20, 10, and 1 dB, respectively.

In analyzing the Adaptive Filtering results from Table 3 for each scenario, the proposed RNN-AUKF achieves a lower overall NRMSE and a higher overall SNR when compared against the benchmark RNN-Direct. These results indicate that the estimations of RNN-AUKF are impervious to the presence of intense white noise. On the contrary, the effect of white noise on RNN-Direct is severe and results in a sharp drop in its SNR.

Besides this, the estimated task space coordinates, segmental angles, and contact forces of RNN-Direct exhibit heavy fluctuations due to sensor noise inheritance, as depicted in Figure 7 and in Supplementary Movie S1. This is because RNN-Direct relies heavily on sensor measurement, which consequently forms a coupling between the sensor noise and its estimates. Comparatively, the adaptive filtering feature in RNN-AUKF is able to automatically detect and minimize sensor noise inheritance.

3. Graceful Degradation Setting: To analyze robustness of the proposed scheme against the impact of sensor drift, negative drift of magnitude 0.2%/s (of flex sensor's range) is introduced to the sensor measurement on each scenario to simulate sensor degradation.

In analyzing the Graceful Degradation results from Table 3 for each scenario, the proposed RNN-AUKF achieves a lower overall NRMSE when compared against the benchmark RNN-Direct. In the Free Bending scenario, RNN-AUKF closely matches the ground-truth values as depicted in Figure 7. This demonstrates its robustness against the impact of sensor drift, offering graceful degradation even without redundant sensors.¹⁸ On the contrary, the estimations of RNN-Direct drift away from the ground-truth values. Despite the improved performance of RNN-AUKF here, we note that this method requires accurate system input information. As such, the estimations from RNN-AUKF exhibit mild degradation in Tip Contact scenario, but is minimal when compared with RNN-Direct.

Multifinger soft gripper platform

As outlined in Supplementary Figure S1, we carried out a third case study, Multifinger Grasping, to investigate the robustness of the proposed RNN-AUKF against out-of-distribution sensor dynamics on the multifinger soft gripper platform. In this case study, segmental angles are used as the system states to enable shape reconstruction. As the camera setup currently captures 2D positions due to occlusion, reflective camera markers are only attached to PSF-A for validation. The three-axis (x, y, z) resultant grasping forces exerted by the soft gripper are calculated via vector sum of the respective estimated two-axis (x, z) contact forces from the three individual PSFs, as shown in Figure 4D. The obtained resultant forces are then validated with the ground-truth data given by the load cell.

Multimodal (bending angles of three PSFs and three-axis resultant grasping forces) estimation results under the two actuation patterns (Oscillatory Actuation and Random Actuation) are obtained for this case study. Table 4 compares these estimation results of the proposed RNN-AUKF and the benchmark RNN-Direct using the NRMSE and RMSE metric. RMSE (without normalization) is used for the resultant grasping forces due to the large difference in magnitude between the horizontal forces in x and y axes, and the vertical force in z-axis.

Table 4.

Comparisons of the Proposed Recurrent Neural Network-Based Adaptive Unscented Kalman Filter and the Benchmark Recurrent Neural Network-Direct for the Multifinger Grasping Case Study on the Multifinger Soft Gripper Platform

	Sum of segmental angles			Resultant grasping forces
	Oscillatory actuation	Random actuation	Average	Oscillatory actuation			Random actuation
				Force x	Force y	Force z	Force x	Force y	Force z
Method	Normalized RMS error (%)			RMS Error (N)						Average
LSTM-AUKF	4.8003	4.3276	4.5640	0.0325	0.0331	0.0745	0.0282	0.0173	0.1195	0.0509
LSTM-Direct	6.2154	5.0811	5.6483	0.0283	0.0299	0.1489	0.0165	0.0240	0.0965	0.0574
GRU-AUKF	5.8059	4.9852	5.3956	0.0356	0.0443	0.1219	0.0275	0.0141	0.0767	0.0534
GRU-Direct	5.8419	5.8726	5.8573	0.0274	0.0233	0.1222	0.0139	0.0205	0.1679	0.0625

Estimation results are obtained under Oscillatory Actuation and Random Actuation. RMSE (in Newton) is used for the resultant grasping forces to capture the large difference in magnitude between horizontal forces in x, y axes and vertical force in z-axis. Average NRMSEs are obtained from averaging the respective segmental angle and contact force results under both actuation patterns.

Figure 9 shows a time-series estimation analysis specifically on LSTM-AUKF and LSTM-Direct for Segmental Angles. This figure also shows the ground-truth data for comparison analysis. Similarly, the transformed end-position coordinates x and z (on the bending plane) of the validated PSF are included in Figure 9. In addition, Figure 10 visualizes the shape reconstruction and grasping force estimation of the three PSFs as a gripper. A full-length visualization of this figure is captured in Supplementary Movie S1.

FIG. 9.

Time-series estimation analysis of the proposed LSTM-AUKF, the benchmark LSTM-Direct, and the ground truth for the Multifinger Grasping case study on the multifinger soft gripper platform. Estimation results are obtained under Oscillatory Actuation and Random Actuation. The flex sensor readings of the three embedded flex sensors corresponding to the identically synchronous pressure inputs are shown in $1^{s t}$ row. End-position coordinates x and z (on the bending plane) of PSF A (in $3^{r d}$ and $4^{t h}$ row) are obtained via transforming the estimated segmental angles. Color images are available online.

FIG. 10.

Visualization for reconstruction of the multifinger soft gripper. Multifinger shape reconstructions of the proposed LSTM-AUKF, the benchmark LSTM-Direct, and the ground truth (only shown for PSF A) are compared. The actual PSF configuration (Left) and the reconstructed bending curves (Right) at one instance are shown. Trajectories of the end-position coordinates x and z (across time) under Oscillatory Actuation are illustrated. The vector plots illustrate the three-axis resultant grasping forces using arrows in three-dimensional spherical coordinates. A video (Supplementary Movie S1) showing the real-time reconstruction is given in Supplementary Data S1. Color images are available online.

In analyzing the results of Table 4, the proposed RNN-AUKF scheme achieves lower overall NRMSE and RMSE scores when compared against RNN-Direct in all scenarios. These results suggest that RNN-AUKF demonstrates a better estimation for both segmental angles and resultant grasping forces despite the presence of inconsistent dynamics across the three flex sensors. In particular, the first row of Figure 9 illustrates this inconsistency where the three flex sensors produce different voltage responses for the identically synchronous pressure inputs.

This in turn suggests that the proposed RNN-AUKF is capable of minimizing the effect of out-of-distribution sensor dynamics by treating it as sensor faults. Hence, this resolves the hurdles in producing sensors with consistent dynamics through costly fabrication processes. However, we noticed that the estimated resultant grasping forces in x and y axes do not completely match the ground truth values, as seen in Figure 9. This is due, in part, to contact slip of one of the PSFs (PSF B) as captured in Supplementary Movie S1, a possible prevention to this problem is discussed in the Limitations and Future Work section.

Processing time

To verify the practicality of the proposed scheme for real-time implementation, the RNN-AUKF model is fed with pressure and flex sensor readings at a sampling rate of 10 Hz to generate estimated bending angles and contact or grasping forces. These estimates are then used to construct a real-time visualization of the PSF as seen in Supplementary Movie S1, with both the frame rate and sampling frequency having the same values. In addition, Table 5 tabulates the processing time of RNN-AUKF and RNN-Direct. In analyzing both the visualizations of the PSF and tabulated results, the processing time per time-step of both RNN-AUKF and RNN-Direct is within a fraction of the data sampling time at 0.1 s. This in turn validates the practicality of the proposed RNN-AUKF model for real-time estimation. On a side note, both RNN-AUKF and RNN-Direct are run in MATLAB.

Table 5.

Processing Time of the Proposed Recurrent Neural Network-Based Adaptive Unscented Kalman Filter and the Benchmark Recurrent Neural Network-Direct

	Resultant angle	Segmental angle
Method	Processing time (ms)	Per time-step
LSTM-AUKF	2.21	3.87
LSTM-Direct	0.75	1.31
GRU-AUKF	2.17	3.72
GRU-Direct	0.72	1.21

Remarks on error stability

The remark of Theorem 1 (given in Supplementary Data S1) stated that parameter $\hat{Q}$ can be set sufficiently large or alternatively $\hat{R}$ can be set sufficiently small to reject w_k and e_k through correction, so that the estimation errors of the state ${\hat{x}}_{k}$ and unknown input $û_{2_{k}}$ are exponentially bounded in mean square. However, a small $\hat{R}$ puts a higher gain on v_k, and thus, an adaptive adjustment scheme is included to estimate the actual $R_{k}$ in each time-step.

Nevertheless, there is always a trade-off between state, unknown input model errors w_k, e_k, and measurement faults v_k. The Kalman filter (and its variants including the UKF) is not optimal when both the model errors w_k, e_k and the measurement faults v_k are on extreme level, exacerbated by the fact that they are not white (Gaussian) most of the time. One instance being the Tip Contact scenario in Graceful Degradation, where there is drift fault on top of poor unknown input prediction. Another example is when both the soft robot and the integrated sensor exhibit large variability, leading to large errors on both ends.

Recall that in the Data-Driven Modeling via Neural Networks section, we introduce a residual-based model switching strategy. This strategy resolves the issue of poor unknown input models (i.e., having large e_k) at low-dimensional state space in the case of Resultant Angle. In addition, given that observability is an important assumption for exponentially bounded estimation errors (as stated in Theorem 1), this strategy also addresses the unobservability problem at high-dimensional state space in the case of Segmental Angles. The overall tabulated and illustrated results demonstrate the efficacy of this strategy in improving the estimations of the proposed AUKF scheme.

Discussions

Summary

This article puts forward a robust Kalman filter-based estimation scheme aided by neural network modeling to realize proprioception and exteroception in soft robots, using a minimal number of proprioceptive sensors (i.e., single embedded flex sensor). The estimation scheme is a combination of four distinct components, that is, state model, measurement model, AUKF, and unknown input models. RNNs (LSTM and GRU) and a simple feedforward network are used to obtain the state model and measurement model, which represent the PSF and flex sensor dynamics. The unknown input models are also RNN models taking either state or measurement as model input.

Lastly, the UKF makes use of the trained network models to compute the optimal estimate. By making the UKF adaptive, we have refined the versatility of our RNN-AUKF estimation scheme in addressing model and sensor anomalies. In addition, the proposed scheme is robust against sensor degradation and inconsistent sensor dynamics, eliminating the need for sensor redundancy. In addition, we have analyzed the stability of the RNN-AUKF and have proven that the estimation errors of both the states and unknown inputs are convergently bounded.

Achievements

By applying a data-driven approach, we circumvent the need to analytically derive a model for a complex soft robot system. Despite that, a common problem for learning-based models is that such models do not provide a physical intuition and meaning of the system. This is especially true when applying a direct mapping between the sensor data and the states for representation learning, as observed in RNN-Direct.¹⁸

However, the proposed RNN-AUKF scheme solves this problem by segmenting the model into interpretative modules. Hence, we are able to retain the system representation that provides physical intuition of the soft robotic system because the system models (state model and measurement model) and the unknown input optimization (unknown input model) are all explicitly identified. In addition, since the measurement is not fed directly into the neural network model, the existence of a bijectivity condition (a nonbijective measurement to state or input function will not be identified) is relaxed to nonlinear observability.

With the proposed RNN-AUKF, we are able to simultaneously achieve both proprioception and exteroception with only the proprioceptive sensors (single embedded flex sensor), realizing multimodal indirect sensing. Crucially, the proposed scheme removes the reliance on exteroceptive sensors such as tactile sensors.

Besides, the residual-based adaptive scheme in the AUKF is able to adaptively adjust the estimation biases on the model and sensor, resulting in robust estimation that provides resilience against adverse noise. If full input information is available, the proposed scheme also offers graceful degradation by applying heavy bias on the state model in the presence of sensor faults. Conversely, this can be done to address the issue of model disturbances or generally poor model quality with heavy bias on the sensor measurement. For this reason, an ideal model or sensor is no longer critical, and thus, the complexity and cost of achieving soft robot perception are significantly reduced.

On top of that, the measurement covariance estimated from the AUKF increases with the discrepancy between the predicted model and sensor output measurement, thus enabling sensor fault detection, where its magnitudes provide an indication on the degree of fault present in the system. Apart from this, by extending the experiments of the proposed RNN-AUKF on a multifinger soft gripper, we have demonstrated the capability of the proposed scheme in coping with out-of-distribution and inconsistent sensor dynamics, despite the fact that the network models were only trained using data from the single PSF. This in turn resolves the hurdles in producing sensors with consistent dynamics, which otherwise would have required a costly fabrication process.

Limitations and future work

Although our proposed framework works well, it has its limitations. The dynamics of soft robots may change over time due to issues such as long-term varying material property or interaction with new environments. Consequently, data-driven models would require retraining using updated platform-specific data sets, which in turn could be costly. Generative approaches^49,50 may offer an alternative approach, through which the use of synthetic data could complement the training of a data-driven model. However, generative networks are not able to precisely dictate the representation of data being generated⁵¹ and require further in-depth study, which can be considered in future work.

On top of that, the current training data for the neural network models were collected based on a fixed orientation of the PSF, as illustrated in Figure 4B. If the orientation about the x and y axes changes, the gravitational effect on the PSF will change, which in turn could result in variations to the PSF dynamics. Hence, the trained models would not be able to generalize to the aforementioned conditions. For future work, we could collect additional training data at different PSF orientations to improve the model's generalization.

In addition, when compared with the benchmark RNN-Direct, our method is more complex in terms of its architecture. As shown in Figure 6, we need to train 3 models for RNN-AUKF, compared with RNN-Direct, which only requires a single model. Hence, we could explore the possibility to concurrently train the required neural network models in a single training loop under the autoencoding variational Bayes (AEVB).⁴⁹ This approach could potentially reduce the overall model training time. Apart from that, active learning with a semisupervised framework such as AEVB, which allows training with the absence of state data, serves as a possible solution to long-term varying soft robot dynamics.

Besides this, the unscented transformation in AUKF requires model computation for each of the unscented particles, which in turn results in a higher computational complexity and increase in processing time. Despite this, the error covariance matrices obtained in the AUKF enable both a robust estimation and predictive uncertainty. Ultimately, these advantages justify its additional computational complexity.

In addition, the estimation error covariance matrices ( ${\hat{P}}^{x x}$ , ${\hat{P}}^{y y}$ , ${\hat{P}}^{u u}$ ) obtained in the A Novel Neural Network Aided Filter-Based Estimation Scheme section provide confidence in the model predictions, offering predictive uncertainty, which quantify the uncertainties associated with the models. Predictive uncertainty is useful in model interpretability especially in safety-critical operations (e.g., human/robot interactions). Moreover, the unscented framework does not assume uncorrelated model (random) variables (heteroscedasticity).

Nevertheless, we wish to emphasize that the processing time of RNN-AUKF is still within a fraction of the data sampling time and therefore ensures the practicality of the proposed scheme for real-time implementation. In future work, we could look into combining state model and unknown input model to reduce the required computational load.

We also noticed that for the Multifinger Grasping case study, the estimated resultant grasping forces in x and y axes deviate slightly from the ground truth due to contact slip. To prevent this in the future, we intend to increase surface friction by integrating a rough material on the contact surface of a PSF. In addition, we hope to extend our current sensor configuration and camera setup to capture the complex 3D motion of nonplanar soft robots. By doing this, the proposed RNN-AUKF framework can be generalized and applied to nonplanar manipulators such as those in Table 1. Based on the outcomes of this research, there are new opportunities to extend the current framework to estimate contact force (and location) along the finger besides the tip.

Apart from this, we have observed in the Results section that the performance of RNN-Direct degrades in the presence of sensor faults. However, these comparisons are conducted with the premise that the recurrent nets in RNN-AUKF and RNN-Direct share the same hyperparameters (e.g., number of hidden nodes). As shown in Normal Operation, LSTM and GRU possess intrinsic filtering capabilities against moderate noise. Considering this, there are opportunities to investigate the relationship between the hyperparameters and robustness of an RNN, which can be considered future work.

Conclusions

In summary, multimodal indirect sensing through estimation is assuredly one of the keys to improve the sensing ability in soft robots. The robust Kalman filter-based estimation scheme undoubtedly plays a significant role in solving this problem. With this achievement, we have contributed toward the ultimate goal of the realization of soft robots with robust perception.

Footnotes

Authors' Contributions

J.Y.L. and Z.Y.D. developed the system, performed the experiments, and prepared the article. V.M.B. supervised the project. S.G.N. and C.P.T. conceived, supervised, and funded the project. All authors read and provided feedback for the article draft and approved the final article. The authors thank Ng Wai Kit for his assistance in developing the soft gripper and Dr. Darwin Gouwanda for his support in acquiring the experimental equipments. Last but not least, they also thank the anonymous reviewers for their insightful comments and suggestions to improve the article.

Data and Materials Availability

All MATLAB codes and data needed to generate the results in this article are uploaded to GitHub repository ().

Author Disclosure Statement

The authors declare that they have no competing interests.

Funding Information

This work was supported by the Ministry of Higher Education Malaysia under the Fundamental Grant Scheme (FRGS) (Grant No. FRGS/1/2016/TK03/MUSM/01/1).

Supplementary Material

References

Kim

, Laschi

, Trimmer

. Soft robotics: A bioinspired evolution in robotics. Trends Biotechnol, 2013; 31:287–294.

Webster

, Jones

. Design and kinematic modeling of constant curvature continuum robots: A review. Int J Robot Res, 2010; 29:1661–1683.

Laschi

, Mazzolai

, Cianchetti

. Soft robotics: Technologies and systems pushing the boundaries of robot abilities. Sci Robot, 2016; 1:eaah3690.

Wang

, Totaro

, Beccai

. Toward perceptive soft robots: Progress and challenges. Adv Sci, 2018; 5:1800541.

Hawkes

, Blumenschein

, Greer

, et al. A soft robot that navigates its environment through growth. Sci Robot, 2017; 2:eaan3028.

Runciman

, Darzi

, Mylonas

. Soft robotics in minimally invasive surgery. Soft Robot, 2019; 6:423–443.

Wang

, Nurzaman

, Iida

. Soft-material robotics. Found Trends Robot, 2017; 5:191–259.

Iida

, Nurzaman

. Adaptation of sensor morphology: an integrative view of perception from biologically inspired robotics perspective. Interf Focus, 2016; 6:20160016.

Culha

, Nurzaman

, Clemens

, et al. SVAS3: strain vector aided sensorization of soft structures. Sensors, 2014; 14:12748–12770.

10.

Ward-Cherrier

, Pestell

, Cramphorn

, et al. The TacTip Family: Soft optical tactile sensors with 3D-printed biomimetic morphologies. Soft Robot, 2018; 5:216–227.

11.

Larson

, Peele

, Li

, et al. Highly stretchable electroluminescent skin for optical signaling and tactile sensing. Science, 2016; 351:1071–1074.

12.

Muth

, Vogt

, Truby

, et al. 3D printing: Embedded 3D printing of Strain sensors within highly stretchable elastomers (Adv. Mater. 36/2014). Adv Mater, 2014; 26:6202–6202.

13.

Truby

, Wehner

, Grosskopf

, et al. Soft somatosensitive actuators via embedded 3D printing. Adv Mater, 2018; 30:1706383.

14.

Park

, Chen

, Wood

. Design and fabrication of soft artificial skin using embedded microchannels and liquid conductors. IEEE Sensors J, 2012; 12:2711–2718.

15.

Kramer

, Majidi

, Sahai

, et al. Soft curvature sensors for joint angle proprioception. In: 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), San Francisco, CA, September 25–30. IEEE,, 2011:1919–1926.

16.

Maiolino

, Galantini

, Mastrogiovanni

, et al. Soft dielectrics for capacitive sensing in robot skins: Performance of different elastomer types. Sensors Actuat A Phys, 2015; 226:37–47.

17.

Han

, Kim

, et al. Use of deep learning for characterization of microfluidic soft sensors. IEEE Robot Autom Lett, 2018; 3:873–880.

18.

Thuruthel

, Shih

, Laschi

, et al. Soft robot perception using embedded soft sensors and recurrent neural networks. Sci Robot, 2019; 4:eaav1488.

19.

Rus

, Tolley

. Design, fabrication and control of soft robots. Nature, 2015; 521:467.

20.

Laschi

, Cianchetti

. Soft robotics: New perspectives for robot bodyware and control. Front Bioeng Biotechnol, 2014; 2:3.

21.

Godage

, Medrano-Cerda

, Branson

, et al. Modal kinematics for multisection continuum arms. Bioinspirat Biomimet, 2015; 10:035002.

22.

Marchese

, Rus

. Design, kinematics, and control of a soft spatial fluidic elastomer manipulator. Int J Robot Res, 2016; 35:840–869.

23.

Thuruthel

, Falotico

, Renda

, et al. Learning dynamic models for open loop predictive control of soft robotic manipulators. Bioinspirat Biomimet, 2017; 12:066003.

24.

Godage

, Medrano-Cerda

, Branson

, et al. Dynamics for variable length multisection continuum arms. Int J Robot Res, 2016; 35:695–722.

25.

Marchese

, Tedrake

, Rus

. Dynamics and trajectory optimization for a soft spatial fluidic elastomer manipulator. Int J Robot Res, 2016; 35:1000–1019.

26.

Shiva

, Sadati

, Noh

, et al. Elasticity versus hyperelasticity considerations in quasistatic modeling of a soft finger-like robotic appendage for real-time position and force estimation. Soft Robot, 2019; 6:228–249.

27.

Wang

, Hirai

. Soft gripper dynamics using a line-segment model with an optimization-based parameter identification method. IEEE Robot Autom Lett, 2017; 2:624–631.

28.

Chen

, Wang

, Galloway

, et al. Modal-based kinematics and contact detection of soft robots. Soft Robot. [Epub ahead of print]; DOI: 10.1089/soro.2019.0095.

29.

Elgeneidy

, Lohse

, Jackson

. Bending angle prediction and control of soft pneumatic actuators with embedded flex sensors—a data-driven approach. Mechatronics, 2018; 50:234–247.

30.

Nakajima

, Hauser

, Li

, et al. Exploiting the dynamics of soft materials for machine learning. Soft Robot, 2018; 5:339–347.

31.

Van Meerbeek

, De Sa

, Shepherd

. Soft optoelectronic sensory foams with proprioception. Sci Robot, 2018; 3.

32.

Bruder

, Fu

, Gillespie

, et al. Data-driven control of soft robots using koopman operator theory. IEEE Trans Robot, 2021; 37:948–961.

33.

Hauser

, Ijspeert

, Füchslin

, et al. Towards a theoretical foundation for morphological computation with compliant bodies. Biol Cybernet, 2011; 105:355–370.

34.

Tanaka

, Yamane

, Hroux

, et al. Recent advances in physical reservoir computing: A review. Neural Netw, 2019; 115:100–123.

35.

Thieffry

, Kruszewski

, Duriez

, et al. Control design for soft robots based on reduced-order model. IEEE Robot Autom Lett, 2019; 4:25–32.

36.

Lunni

, Giordano

, Sinibaldi

, et al. Shape estimation based on Kalman filtering: Towards fully soft proprioception. In: 2018 IEEE International Conference on Soft Robotics (RoboSoft), Livorno, Italy, April 24–28. IEEE,, 2018:541–546.

37.

Loo

, Kong

, Tan

, et al. Non-linear system identification and state estimation in a pneumatic based soft continuum robot. In: 2019 IEEE Conference on Control Technology and Applications (CCTA), Hong Kong, China, August 19–21. IEEE,, 2019:39–46.

38.

Loo

, Tan

, Nurzaman

. H-infinity based Extended Kalman Filter for State Estimation in Highly Non-linear Soft Robotic System. In: 2019 American Control Conference (ACC), Philadelphia, PA, July 10–12. IEEE,, 2019:5154–5160.

39.

Huang

, Yang

, Zhou

. Adaptive quadratic sum-squares error with unknown inputs for damage identification of structures. Struct Contr Health Monitor, 2010; 17:404–426.

40.

Liu

, Su

, Zhu

, et al. Data fusion based EKF-UI for real-time simultaneous identification of structural systems and unknown external inputs. Measurement, 2016; 88:456–467.

41.

Hochreiter

, Schmidhuber

. Long short-term memory. Neural Comput, 1997; 9:17351780.

42.

Cho

, Van Merriënboer

, Gulcehre

, et al. Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078 2014.

43.

Julier

, Uhlmann

. Unscented filtering and nonlinear estimation. Proceed IEEE, 2004; 92:401–422.

44.

Ilievski

, Mazzeo

, Shepherd

, et al. Soft robotics for chemists. Angewandte Chem Int Edit, 2011; 50:1890–1895.

45.

Katzschmann

, Marchese

, Rus

. Autonomous object manipulation using a soft planar grasping manipulator. Soft Robot, 2015; 2:155–164.

46.

Jain

, Seera

, Lim

, et al. A review of online learning in supervised neural networks. Neural Comput Applicat, 2013; 25:491–509.

47.

Vlachas

, Pathak

, Hunt

, et al. Backpropagation algorithms and Reservoir Computing in Recurrent Neural Networks for the forecasting of complex spatiotemporal dynamics. Neural Netw, 2020; 126:191–217.

48.

Metia

, Ha

, Duc

, et al. Estimation of power plant emissions with unscented kalman filter. IEEE J Selected Top Appl Earth Observ Remote Sens, 2018; 11:2763–2772.

49.

Kingma

, Welling

. Auto-encoding variational bayes. In: 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14–16, 2014.

50.

Goodfellow

, Pouget-Abadie

, Mirza

, et al. Generative adversarial nets. In: Ghahramani Z, Welling M, Cortes C, Lawrence N, Weinberger KQ. (Eds). Proceedings of the Advances in Neural Information Processing Systems. Quebec, Canada: Curran Associates, Inc.,, 2014; 27:2672–2680.

51.

Kingma

, Mohamed

, Jimenez Rezende

, et al. Semi-supervised learning with deep generative models. In: Ghahramani Z, Welling M, Cortes C, Lawrence N, Weinberger KQ. (Eds). Proceedings of the Advances in Neural Information Processing Systems. Curran Associates, Inc.,, 2014; 27:3581–3589.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.88 MB

1.50 MB

0.01 MB

52.05 MB

0.03 MB

0.00 MB