Model-Based 3D Shape Reconstruction of Soft Robots via Distributed Strain Sensing

Abstract

Proprioception in soft robots is essential for enabling autonomous behaviors, allowing them to navigate and interact safely in unstructured environments. Previous sensorization-based shape reconstruction methods, which often rely on machine learning techniques, have limitations in their broad applicability for different robotic systems and environments. In this work, we present a shape reconstruction scheme enabled by sparsely distributed soft strain sensors on the surfaces of soft robots, combined with a model-based reconstruction framework. Our approach utilizes miniaturized stretchable capacitive strain sensors with large stretchability and low hysteresis, which can be easily attached to soft robot surfaces for accurate local strain measurements. These measurements are fed into an optimization algorithm with embedded mechanical constraints. Our approach can predict all deformation modes in a soft bar with a maximum displacement error of less than 4% of the bar length and accurately reconstruct the shapes of soft pneumatic grippers during grasping actions. Additional reconstructions of a bioinspired arm in complex contact scenarios further demonstrate the versatility of our approach. This shape reconstruction scheme using distributed strain sensors offers a convenient and broadly applicable solution for enhancing proprioception in soft robots.

Keywords

shape reconstruction distributed sensing model-based reconstruction stretchable electronics

Introduction

The mechanical compliance, dexterity, and fast prototyping capabilities of soft robots make them attractive for safe, delicate, and flexible interaction with soft materials and biological systems.^1,2 Applications of soft robots include object manipulation in industrial automation and agriculture,^3,4 marine exploration,^5,6 search and rescue,⁷ and rehabilitation and minimally invasive surgery.^8,9 Many of these applications require soft robots to operate accurately in dynamic and uncertain environments, necessitating closed-loop control. Proprioception provides the essential shape and orientation feedback needed for this adaptive control,^10,11 yet convenient and accurate proprioception of soft robots in real-world settings remains a fundamental challenge, mainly due to geometric and material nonlinearities.¹²

Existing proprioception methods of soft robots mainly rely on two basic strategies: vision-based tracking and embedded shape sensing.^11–13 Vision-based approaches, including camera-based motion capture^14,15 and color mapping,^16,17 enable fast and accurate shape reconstruction through geometrical computation. However, they may be limited by the need for a predefined multi-camera setup, controlled lighting, and susceptibility to optical occlusion outside laboratory settings. In environments with these constraints, integrating sensors directly with soft robots, or sensorization of soft robots, provides an alternative proprioception solution. These integrated sensors include inertial measurement units (IMUs),^18–22 fiber-based optical sensors,^23–27 and soft strain sensors.^28–32 IMUs are compact and capable of directly measuring motion-related data in various environments, but they typically suffer from drift due to sensor noise³³ and face challenges integrating with soft robots due to their rigidity.²⁰ Fiber-based optical sensors offer high precision in deformation measurements but often have limited stretchability and difficulties in spatial distribution.²⁴ Soft strain sensors, which can be integrated with soft robots, are particularly useful for three-dimensional (3D) shape sensing without significantly altering the robots’ intrinsic properties. Therefore, soft and stretchable strain sensors, which are well suited for distributed strain sensing, are used in this study.

Constructing 3D shapes of soft robots from measured strain has relied on various models, including Cosserat rod models,^14,34 Frenet-Serret formulas,^30,32 and finite element analysis (FEA).^29,35 However, many shape reconstructions are achieved using machine-learning-assisted or other data-driven methods.^15,36–38 These approaches often require large amounts of training data, which can be time-consuming and practically challenging. Despite their relatively high accuracy in real-time reconstruction and their capability as surrogate models for closed-loop control,²⁰ purely machine learning-based models, especially deep-learning models, do not provide sufficient physical understanding of soft robot deformations, limiting their applicability to different soft robotic systems in changing environments. In contrast, machine-learning-assisted reduced-order models account for both physical constraints and reconstruction efficiency.^39,40 Nevertheless, existing work mostly uses displacement to construct a mapping from sparse measurements to field reconstruction,⁴⁰ often relying on strong assumptions for reduced-order modeling.^39,41 This approach requires direct displacement measurement, limiting its general applicability. Furthermore, most previous work focuses on shape estimation of soft robots under omnidirectional bending,^{29,30,34,35,38} while few studies have reported reconstructing multimodal deformations that involve twisting and elongation modes.^12,23,37

In this work, we present a shape reconstruction scheme enabled by soft strain sensors sparsely distributed on soft robot surfaces and a model-based shape reconstruction framework. The centimeter-scale, low-cost capacitive strain sensors with large stretchability and low hysteresis provide accurate, orientation-dependent local strain measurements for both tensile and compressive strain, and they can be attached to soft robot surfaces with minimal mechanical constraints. Our model-based shape reconstruction framework incorporates local strain from distributed strain sensors into an optimization algorithm and leverages numerical techniques to reconstruct all deformation modes of soft robots with high accuracy. The shape reconstruction scheme is first applied to sensing multimodal deformations of a soft robotic bar, achieving a maximum displacement error of less than 4% of the bar length. Sensing the deformed shapes of soft pneumatic grippers during grasping actions highlights the capability of shape reconstruction under internal actuation and external contact forces. Shape reconstruction of a bioinspired soft robotic arm in complex contact scenarios further demonstrates the versatility of the scheme for soft robot shape sensing in unstructured environments.

Results

Shape reconstruction framework

The overall strategy of our shape reconstruction framework is to accurately estimate the equilibrium 3D shapes of soft robots using local strain measured by soft, miniaturized strain sensors strategically distributed over the soft robot surfaces. Figure 1a schematically shows multiple soft strain sensors attached to a soft robot. Mathematically, such a reconstruction scheme can be formulated as a constrained optimization problem that searches for the minimal difference between the experimentally measured local strain and the corresponding strain calculated from the reconstructed shape (Equation (1)). FEA is used as the embedded simulator to discretize the target soft robot and ensure the satisfaction of the mechanical equilibrium constraint, where the nodal forces are used as input variables.

\begin{array}{l} Find x (f) that minimizes Φ (x (f)) = \frac{1}{2} \sum_{i = 1}^{n} {(ε_{i} (x (f)) - {\bar{ε}}_{i})}^{2} + \frac{1}{2} w {‖ f ‖}^{2} \\ subject to {\begin{cases} \frac{\partial E}{\partial x} - \sum_{j = 1}^{m} p_{j} \frac{\partial V_{j}}{\partial x} - f^{T} = 0, where E (x (f)) = \sum \int Ψ (F (X, x, \bar{X})) d \bar{X} \\ f_{lower} < f < f_{upper} \end{cases} \end{array}

(1)

FIG. 1.

Shape reconstruction of soft robots via distributed strain sensing and an optimization-based framework. (a) Schematic illustration of a soft robotic bar with soft strain sensors distributed on its surface. (b) Flow chart of the shape reconstruction framework.

In the expression of the cost function $Φ$ , $f$ is the nodal force vector within a set range ( $f_{lower}$ , $f_{lower}$ ) and $x$ is the nodal position vector in FEA, ${\bar{ε}}_{i}$ is the strain measured by the distributed strain sensors, and $ε_{i}$ is the computed strain at the corresponding locations from FEA. $w$ is a small constant (ranging from 10⁻⁶ to 10⁻⁴) that acts as a regulation term to prevent the main data cost from being stuck in a potential local minimum. In the mechanical equilibrium condition, $E$ is the total strain energy function represented by the deformation gradient $Ψ (F (X, x, \bar{X}))$ , where $X$ and $x$ are the reference and deformed coordinates of nodes, and $\bar{X}$ is an arbitrary point used to define the deformation gradient. For pneumatic soft robots, the internal actuation pressure needs to be considered. $p_{j}$ is the applied pressure in the j^th pneumatic chamber (m chambers in total) and $V_{j}$ is the volume of the corresponding chamber. The local strain at the sensor locations can also be calculated from the FEA by tracking the length changes between deformed and undeformed configurations:

ε_{i} = \frac{\sum_{k = p, p + 1, . . .}^{q} || x_{k} - x_{k - 1} || - s_{0}}{s_{0}}

(2)

where

x_{k}

is the deformed position vector of an arbitrary node in a nodal set (nodal index from p to q in the FEA) to which the sensor is attached, and

s_{0}

is the initial length of the sensor.

Figure 1b shows the process flow of our reconstruction framework. Applying a set of initial nodal forces in the FEA leads to a deformed shape in equilibrium, from which the cost function and sensitivity function are computed using the measured strain from sensors (Supplementary Data S1). Numerical methods can be applied to decrease the cost function by updating the input nodal forces iteratively. The optimization process is completed when the convergence conditions are met, that is, both the cost function and its change after the previous interaction are smaller than set thresholds within a certain maximum number of iterations, which is set at 1,000 in this study.

Design, fabrication, and characterization of soft strain sensors

Achieving a combination of large stretchability, high linearity, and low hysteresis is critical for accurate strain measurement in soft robots but has been challenging in previous strain-sensing-based shape reconstruction studies. Here, we have developed soft capacitive strain sensors that can satisfy these requirements and measure local deformations accurately. The sensor features a thin (approximately 15 μm in thickness) layer of carbon nanotubes (CNTs) patterned into interdigitated electrodes (IDE) and sandwiched by two layers of silicone elastomer (Ecoflex 00-31, modulus at 100% strain: 69 kPa) as support and encapsulation (Fig. 2a and Supplementary Figs. S1–S2). The total sensor thickness is approximately 600 µm. Details of the sensor fabrication, modeling, and calibration appear in Supplementary Figure S3 and Supplementary Data S2–S4.

FIG. 2.

Design and characterization of soft IDE strain sensors. (a) Schematic illustration of strain sensing using IDE strain sensors. (b) Relative capacitance changes of individual IDE strain sensors during uniaxial loading (up to 100% strain) and unloading, measured from five sensors with linear fitting. (c) Relative capacitance changes of three-element (0°/45°/90°) strain sensor rosettes under local tensile strain from 0% to 38%. $C_{0^{°}}$ , $C_{45^{°}}$ , and $C_{90^{°}}$ represent the capacitance of the three IDE strain sensors aligned in the 0°, 45°, and 90° angle directions. The solid lines represent linear fits of measurements from three sensor rosettes, and the shaded areas represent the standard deviations. (d) Relative capacitance changes of three-element (0°/45°/90°) strain sensor rosettes under compressive strain from 0 to −25%. The solid lines represent the linear fits of measurements from three sensor rosettes, and the shaded areas represent the standard deviations. Scale bars, 1 cm in C and D. IDE, interdigitated electrodes.

Stretching an IDE capacitive strain sensor along the longitudinal direction of the electrodes causes an increase in the capacitance due to increased electrode length $L$ and reduced gap $a$ ; compressing the IDE strain sensor leads to opposite changes (Fig. 2a). Figure 2b presents the relative capacitance changes from five IDE strain sensors during loading and unloading in their longitudinal direction, exhibiting small hysteresis due to the capacitive sensing mechanism and the use of spray-coated CNT electrodes. The strain range can reach up to 200%, with a hysteresis of 1.10% at 200% nominal strain (Supplementary Fig. S4a), which is among the lowest for IDE strain sensors with a large (>100%) strain range (Supplementary Fig. S4b).^42–51 The gauge factor (GF), defined as $G F = \frac{Δ C / C_{0}}{ε}$ , is calculated using local strain to characterize sensor responses more accurately. The measured GF is 1.01 ± 0.08 from 10 sensors. Supplementary Figure S5 shows representative sensor responses to local tensile and compressive strain. The sensors also demonstrate stable performance over repeated stretching cycles, with a maximum relative capacitance change fluctuation of only 4.2% (<0.1% if normalized to the relative change in the first cycle) after 1,000 cycles at 100% tensile strain (Supplementary Fig. S6). In addition, the IDE strain sensors exhibit low sensitivity to normal and uniform shear stress from external contacts, with relative capacitance changes of 1.8% under 30 kPa normal pressure and 2.3% under 10 kPa uniform shear stress during contact. Supplementary Data S5 and Supplementary Tables S1–S3 provide more detailed data on the influences of normal and uniform shear stress, as well as temperature, on sensor responses.

Arranging multiple IDE strain sensors in a rosette allows the capture of the plane strain state locally for different modes of deformation. A three-element (0°/45°/90°) strain sensor rosette serves as an example (Fig. 2c). Uniaxial stretching (up to 40% strain) of the sensor rosette attached to an elastomeric substrate increases the capacitance of the sensor placed along the stretching direction (0°) $C_{0 °}$ and decreases the capacitance of the sensor orthogonal to it ( $C_{90 °}$ ). Uniaxial compression (up to −25% strain) of the sensor rosette causes opposite capacitance changes (Fig. 2d). $C_{0 °}$ and $C_{90 °}$ show slightly different $G F$ due to the simultaneous shrinking in the gap width and elongation of the electrodes⁵² (details appear in Supplementary Data S2 and Supplementary Fig. S7–S8).

Numerical methods and strain sensor configurations

Gradient-based optimizers, such as the successive linear programming and the family of quasi-Newton’s methods (e.g., trust-region method), are typically applied to nonlinearly constrained optimization problems such as the one in Equation (1). Here we use the method of moving asymptotes (MMA), which balances fast convergence and stability and integrates well with FEA in structural optimization.^53–55 It divides the global minimization problem into a set of convex subproblems without strict requirements on the convexity of the objective. With a simpler formulation than structural optimization problems, the MMA enables effective and efficient shape reconstruction.

Moreover, we apply dimension reduction (DR) to reduce the computational load and improve the accuracy of the optimization process. Instead of using the full-order nodal force vector, only the forces at the surface nodes or near the actuation regions, which are potentially subjected to external forces, are considered in the optimization process. Similarly, the corresponding rows in the Hessian matrix are selected to ensure consistency (Supplementary Data S1).

The combination of the MMA and DR offers high reconstruction accuracy with improved computational efficiency. Figure 3a shows the comparison of the computational reconstruction results of a soft silicone bar using several different methods. The bar is 200 mm in length and 40 mm in width and thickness, with one end fixed (blue-shaded surface) and the other end subjected to a prescribed biaxial bending (90°). A total of 8 three-element (0°/45°/90°) strain sensor rosettes are distributed on the four side surfaces of the bar (Supplementary Fig. S9). Details of the FEA appear in Supplementary Data S6. Supplementary Figure S10 compares the simulation results of the soft silicone (Ecoflex 00-31) bar under the same external loads with and without the strain sensor rosettes (1.2 mm in total thickness, including an Ecoflex 00-31 adhesive layer for stable attachment), with the maximum displacement difference within 0.3% normalized to the bar length. These results verify that the presence of the thin and soft IDE strain sensors has a negligible influence on the geometry and mechanical properties of target soft robots. This influence would be even smaller for soft robots made from materials with a higher modulus than Ecoflex 00–31. The MMA method yields a maximum normalized displacement error (MNDE, normalized to the bar length) of 45.0% after 1,000 iterations (Fig. 3a, left). An alternative quasi-Newton’s method, the trust region method, together with DR, reduces the MNDE to 28.4% (Fig. 3a, middle). A combination of the MMA and DR significantly increases the reconstruction accuracy, reaching as low as 1.3% MNDE after the same number of iterations (Fig. 3a, right). In addition, the use of DR together with the MMA optimizer can reduce the computation time by a factor of 40–60. The cost function during this reconstruction process is reduced by over three orders of magnitude after 1,000 interactions (Fig. 3b).

FIG. 3.

Numerical studies on the effects of different numerical methods, configurations, and number of strain sensors on the shape reconstruction. (a) Normalized displacement errors of shape reconstruction of a deformed soft bar using three different numerical methods: MMA only, trust region (TR) method and dimension reduction (DR), and MMA and DR. (b) Changes of cost function during an iterative optimization process for the shape reconstruction using MMA and DR shown in a. (c) Influence of different strain sensor configurations on the reconstruction accuracy for bending and hybrid deformations of a soft bar: single-element sensors (0°), two-element (0°/90°) strain sensor rosettes, and three-element (0°/45°/90°) strain sensor rosettes. (d) Influence of the number of three-element (0°/45°/90°) strain sensor rosettes on the reconstruction accuracy (measured in RMSE) for bending and hybrid deformations of a soft bar.

The configuration of IDE strain sensors has a strong influence on the reconstruction results. The use of a three-element (0°/45°/90°) strain sensor rosette allows the measurement of local strain states, facilitating accurate shape reconstruction. Supplementary Figure S11 schematically shows a 12-sensor rosette configuration (three-element sensor rosettes on each side surface) on the same soft bar. When only the 12 single-element (0°) IDE strain sensors in each rosette are used for strain sensing, reconstruction of the same bending deformation yields a 1.77 mm root square mean error (RSME), defined as $RSME = \frac{\sqrt{\sum_{i = 1}^{n} (x_{i} - \bar{x_{i}})^{2}}}{n}$ . This RMSE is reduced to 0.51 mm if two IDE strain sensors (0° and 90°) in each sensor rosettes are used for strain sensing, and the RMSE is the smallest (0.12 mm) for three-element (0°/45°/90°) strain sensor rosettes (Fig. 3c). The IDE strain sensors in the 45° direction in the sensor rosettes are particularly useful in measuring shear strain during twisting deformations. For a hybrid deformation (90° biaxial bending with 30° twisting) of the bar, no convergence is reached without this 45° IDE strain sensor, while convergence occurs with an RMSE of 0.82 mm after 1,000 iterations. This drastic difference shows the advantages of the strain sensor rosette for measuring deformations that involve twisting, where significant shear strain is present.

In addition to the local strain sensor configuration, it is essential to investigate the number of strain sensor rosettes. Figure 3d presents the reconstruction accuracy of 90° biaxial bending of the bar, with three-element (0°/45°/90°) strain sensor rosettes incrementally added or removed from the initial 12 sensor rosettes. Supplementary Figure S11c shows the locations of specific sensor rosettes for a certain number of rosettes used in Figure 3d. The RSME drops markedly with the increasing number of strain rosettes, from 1.8 mm to 0.1 mm for bidirectional bending, and from 3.4 mm to 0.9 mm for bending coupled with twisting. While increasing the number/density of sensors typically leads to higher accuracy, it increases the difficulty of sensorization and may slow down the convergence process. Further increases in the number of sensor rosettes beyond 9 or 10 do not result in a notable decrease in the RMSE.

For the spatial arrangement of sensor rosettes for a certain number of rosettes, a general strategy is to distribute the rosettes over the soft robot surface regions with large strain. Without prior knowledge of the strain distribution, a relatively uniform distribution of sensor rosettes is typically adopted.

Reconstruction of multimodal deformations of a soft robotic bar

Experimental validation of the shape reconstruction framework starts with the reconstruction of a passive silicone (Ecoflex 00-31) bar under different deformation modes. The specifications of the bar and the configuration of the eight strain sensor rosettes in the experiments are the same as those in the computational studies in Figure 3a. The 3D poses of the bar, captured by two depth cameras with the aid of RGB markers (Fig. 4a and Supplementary Data S7), serve as the ground truth. The capacitance changes of the distributed IDE strain sensors are measured using a custom multiplexing circuit (Supplementary Data S8 and Supplementary Fig. S12). All the IDE strain sensors are calibrated to ensure strain measurement accuracy (see Supplementary Data S4 and Supplementary Table S4). Figure 4b shows the relative capacitance changes from four strain sensor rosettes on two surfaces under 90° in-plane bending. The capacitance changes from all other surfaces appear in Supplementary Figure S13. The measured capacitance changes from the initial upright (undeformed) shape to a manually deformed equilibrium shape. The time-averaged, stable capacitance readings are then converted to local strains based on the IDE strain sensor characterization.

FIG. 4.

Experimental setup and shape reconstruction results of a soft bar under multimodal deformations. (a) Optical image of the experiment setup for the shape reconstruction of a soft bar. (b) Capacitance changes from selected IDE strain sensors under in-plane bending of the soft bar. (c–h) Optical images of a soft bar under different deformation modes overlaid with reconstructed shapes (in blue): (c) in-plane bending, (d) biaxial bending, (e) clockwise twisting, (f) counter-clockwise twisting, (g) hybrid mode 1 combining biaxial bending and clockwise twisting, and (h) hybrid mode 2 combining biaxial bending and counter-clockwise twisting. Scale bar, 3 cm. IDE, interdigitated electrodes.

Figure 4c–h shows the reconstructed shapes (in blue color) overlaid with experimental images in six different deformation modes. The reconstruction accuracy for multimodal deformations, measured in the RSME, and the MNDEs, is summarized in Table 1. The MNDE is within 4% in all these deformations, indicating a high level of reconstruction accuracy. Notably, the MNDEs for the twisting modes are within 1.5%. This level of reconstruction accuracy is among the highest compared with the limited studies on shape reconstruction of soft robots under twisting deformations.^14,23,37 The reconstruction errors are below 3% for the two hybrid deformations, highlighting the benefits of using three-element strain sensor rosettes for complex, multimodal deformations.

Table 1.

Shape Reconstruction Errors of a Soft Bar Under Multimodal Deformations

Deformation mode	Deformation magnitude	RSME (mm)	Max. normalized displacement error (%)
In-plane bending	60°	2.55	3.4
Biaxial bending	60°	2.78	3.7
Clockwise twisting	50°	1.41	1.3
Counter-clockwise twisting	50°	1.24	0.9
Hybrid 1	60° biaxial bending with 30° twisting	2.80	2.7
Hybrid 2	60° biaxial bending with 30° twisting	3.57	2.3

RSME, root square mean error.

The experimental measurements of biaxial bending (Fig. 4d) and hybrid deformation 1 (Fig. 4g) are also used to verify the effects of sensor configuration and the number of sensor rosettes on shape reconstruction, which are numerically studied and presented in Figure 2c–d. The detailed results in Supplementary Tables S5 and S6 show that a single-element IDE strain sensor configuration does not support reconstruction of twisting deformation and that the RSME decreases with an increasing number of sensor rosettes—both are consistent with the numerical results.

In addition to the examples shown in Figure 4, a set of numerical simulations extends the type and range of deformations for testing the shape reconstruction performances, some of which are challenging to achieve experimentally. In these tests, the deformed configurations are first simulated using FEA and used as the reference shape. With 12 three-element strain sensor rosettes distributed on the side surfaces (same configuration as in Supplementary Fig. S11a–b), our framework is capable of reconstructing large (90°) uniaxial and biaxial bending with RSME less than 0.01 mm (Supplementary Fig. S14 and Supplementary Table S7). Supplementary Figure S14d highlights the capability of the framework to reconstruct twisting deformations (50°) with 2% MNDEs, which is consistent with the experimental results. In addition, the framework can also deliver accurate shape sensing with a 1% MNDE for 25% elongation (Supplementary Fig. S14e). It also reconstructs shapes effectively in hybrid modes, such as biaxial bending coupled with twisting (Supplementary Fig. S14c) and a combination of bending, twisting, and elongation (Supplementary Fig. S14f). The reconstruction error increases with the magnitude of displacement. As a result, the maximum error always occurs at the tip of the beam, consistent with experimental measurements.

The reconstruction errors originate from multiple sources. In modeling, discrete strain measurements from sparse sensor rosettes, numerical errors, and potential local minimums can contribute to the errors. In experiments, inaccuracies in the geometries, material properties, and sensor positioning, imperfections in sensor fabrication, and measurement noises are potential sources. These factors typically result in larger reconstruction errors in experiments compared with numerical results.

Shape reconstruction of pneumatic soft robotic grippers

The reconstruction framework is applicable to soft robots under internal actuation and external contact forces. Internal actuation, such as pneumatic pressure, is considered in the mechanical equilibrium (see Equation (1)). Pressure from surface contacts is inherently accounted for in the reconstruction framework, as surface nodal forces are variables in the optimization process. The shape reconstruction capability for soft robots under internal and external loadings is demonstrated in bi-material PneuNet actuators,⁵⁶ a widely used type of pneumatic actuators. Details of the design, fabrication, actuation, and sensorization of the actuators appear in Supplementary Data S9 and Supplementary Figure S15. Supplementary Figure S15c shows the relative capacitance changes measured from the four IDE strain sensors and simulated strain from FEA. Integrating these two actuators onto the end effector of an industrial robotic arm and actuating them via a pneumatic control circuit (Supplementary Data S10 and Supplementary Fig. S16) enable object grasping (Fig. 5a). The grasping process begins as the grippers approach a glass vial freely, followed by the application of actuation pressure to bend the grippers. After contacting the vial, the grippers are lifted to complete the grasping process (Supplementary Movie S1).

FIG. 5.

Shape reconstruction of soft robotic grippers. (a) Optical image of two pneumatic soft robotic grippers mounted at the end effector of a robotic manipulator. (b) Optical images overlaid with reconstructed shapes (in blue) in three states during a grasping action. Scale bar, 3 cm.

Figure 5b shows three frames (states 0, 1, and 2) representing three distinct stages of the grasping process with simulated or reconstructed shapes overlaid on the experimental images. The static equilibrium state of the actuator without actuation is simulated directly by FEA. The shapes of the two actuators are inversely extracted from the sensor characterization curves in Supplementary Figure S15c, as the sensors’ readings are monotonic with increasing pressure when no external interactions are present. After the grippers contact the vial, our reconstruction framework is applied to estimate the deformed shapes. While the pneumatic actuation and the unpredictable contacts between the grippers and the vial significantly increase the level of difficulty compared with the bar in Figure 4, our reconstruction framework exhibits high reconstruction accuracy, with MNDEs within 3% (Supplementary Table S8).

It is worth noting that the positioning of the IDE strain sensors on the expansion-limited sides of the PneuNet actuators is due to the accurate sensing capabilities of our strain sensors for uniaxial strain, compared with biaxial surface expansion. To illustrate this, separate strain sensing experiments are conducted on a four-chamber soft actuator integrated with 12 single-element IDE strain sensors (Supplementary Fig. S17a). A strain sensor experiences 4% relative capacitance changes when the soft actuator is manually bent to 30°, consistent with sensor characterization. In contrast, the sensor’s relative capacitance decreases under volumetric expansion when the soft actuator is bent to the same bending angle through pneumatic actuation (Supplementary Fig. S17b). This can be explained by biaxial tensile strain applied to the sensor during air chamber expansion, which increases the gaps between the IDEs and results in a coupled sensor response to longitudinal and transverse stretching. While the current IDE strain sensor design is not suitable for biaxial deformations such as expansion, our shape reconstruction scheme remains applicable to soft robots undergoing volumetric expansion. A reconstruction of the four-chamber actuator shown in Supplementary Figure S17a is performed numerically under pneumatic actuation and external interaction forces, assuming known surface strain states at the sensor locations (Supplementary Fig. S17c). The reconstruction yields an MNDE of 0.2%, suggesting that our framework is applicable to soft robots with complex deformations such as non-uniform expansion, provided that surface strain states can be measured by appropriate strain sensors.

Shape reconstruction of a bioinspired arm in contact with objects

Our framework’s shape reconstruction capability in contact scenarios is further demonstrated in shape reconstructions of a bioinspired arm interacting with the environment. A tapered silicone arm (40 mm and 10 mm in diameter for the large and small ends, respectively; 200 mm in length), inspired by the tentacles of cephalopods, represents an example of a bioinspired dexterous soft manipulator. Twelve single-element IDE strain sensors are distributed evenly longitudinally and circumferentially (Supplementary Fig. S18a). The choice of the single-element sensor pattern is due to the bending-dominated deformations in this example. The large end is fixed and the entire external surface of the tapered arm is set as the interacting zone in the DR process of reconstruction. Figure 6a illustrates a scenario in which the bioinspired arm comes into contact with two obstacles (fixed rods), resulting in a shape reconstruction with an MNDE of 5.4% (Supplementary Table S9). The arm may also experience additional external loads, as shown in Figure 6b, where the MNDE of the reconstructed shape increases to 7.0% due to relatively large stress concentration near the contact region. Direct contact with the IDE strain sensor surfaces can also lead to slightly increased reconstruction errors (Supplementary Fig. S19 and Supplementary Table S9). Supplementary Figure S20 presents the raw sensor data for these experiments.

FIG. 6.

Shape reconstruction of a bioinspired soft arm in contact with objects. (a–b) Optical images of a bioinspired soft arm (a) in contact with obstacles and (b) with external load applied near the tip, overlaid with the reconstructed shape (in blue). Inset in (a) shows the side-view image of the obstacles. Scale bars, 5 mm. (c) Shape reconstruction of a simulated octopus arm-inspired soft robot in contact with different objects. The reference shapes generated by the simulation are in gray, and the reconstructed shapes are color-coded using the normalized displacement error. Poses 1–4 represent the arm wrapping around a rod, wrapping around a rod with another rod in contact, bending with the tip in contact with a rigid plate, and bending with the side surface in contact with a rigid plate, respectively.

Additional numerical studies further demonstrate the shape reconstruction of bioinspired arms interacting with the environment. Supplementary Movie S2 illustrates four numerical examples of a simulated tapered arm (20 mm and 5 mm in diameter for the large and small ends, respectively; 200 mm in length) integrated with 20 IDE strain sensors (Supplementary Fig. S18b). Figure 6c presents the final reconstructed shapes at the end of each interaction. In pose 1, the arm wraps around an elastic cylinder, mimicking the grasping action of an octopus’ arm, while an additional contact from another elastic cylinder interrupts the movement in pose 2. In poses 3 and 4, the arm touches the ground (a rigid plate) with different contact regions. The RSME of these four numerical examples is as low as 0.02 mm (Supplementary Table S10). The error increases when contact is involved due to significant stress concentration around the contact area and the reduced number of elements used to lower the computational load.

Conclusion

We present a proprioception solution for soft robots that combines low-cost and easily attachable capacitive IDE strain sensors sparsely distributed on the surfaces of soft robots and a model-based optimization framework. Our approach is broadly applicable to various soft robotic systems for high-accuracy reconstruction of 3D shapes in equilibrium states, as demonstrated experimentally through shape sensing of multimodal deformations of a soft bar and grasping actions of pneumatic grippers, as well as shape reconstruction of bioinspired soft arms interacting with their environment. The shape reconstruction in these examples requires fewer than 12 single-element IDE strain sensors or sensor rosettes and can achieve experimental reconstruction accuracies of less than 7% (4% in contact-free environments). Notably, the reconstruction accuracies of twisting-related deformations can achieve the same level as in some model-free reconstructions.^23,37

The strain sensors feature an IDE design with low hysteresis (∼1%) for large strain ranges (up to 200%), and they respond to both tensile and compressive strain, which are accurately modeled. The sensor rosettes provide local, multidirectional strain sensing capabilities, which are critical for deformations involving twisting. The sensor fabrication process is lithography-free, using low-cost materials to achieve low hysteresis with sufficient stretchability, offering significant advantages over strain sensors based on conductive composites^42,57 or liquid metals.³⁷ The sensors are centimeter-scale and approximately 600 µm thick, allowing them to be easily attached to existing soft robots with minimal alterations to their intrinsic structural and material properties. This is in contrast to the sensorization of the entire robot body^28,37 or embedding sensors during the soft robot fabrication process.²⁹ The directional strain sensing capabilities and strategic configurations of sensors enable multimodal shape reconstruction, especially for deformations involving twisting. The current IDE strain sensor design has limitations in measuring volumetric expansion or other complex biaxial stretching or compression, which may be more accurately captured using advanced strain sensors, such as omnidirectional strain sensors. Another limitation of the current sensor integration is the wiring required for each sensor, which can restrict the workspace of soft robots and increase noise levels due to wire movement. The use of miniaturized wireless strain sensors may help mitigate this issue.

The sparsity of strain sensors is a major source of reconstruction errors. Using fewer sensors simplifies the sensorization process but can also increase reconstruction errors. Future studies on optimizing the sensor placement based on prior knowledge of deformation modes can mitigate this tradeoff by positioning more sensors in regions experiencing significant or characteristic deformations.

Our shape reconstruction framework uses FEA as the embedded simulator, providing high geometric resolution in 3D proprioception with high accuracy, although at higher computational costs (see Supplementary Table S11 for the computational time of each reconstruction case presented in this study) compared with reduced-order models. The use of MMA and DR in the optimization algorithm significantly enhances the convergence speed, balancing reconstruction accuracy and efficiency.

Another unique feature of our framework is its capability to sense shapes in contact-rich environments. This capability is critical for closed-loop control of soft robots in unstructured environments and is enabled by incorporating interaction forces in the optimization. However, real-time reconstruction using our approach remains challenging due to the high computational cost of the FEA simulator. In addition, the sim-to-real gap in model-based methods for soft robot shape reconstruction may increase error levels when large deformations are involved.^58,59 Future work may include developing a hybrid approach that integrates physical constraints with data-driven methods to realize real-time, accurate, and physically feasible shape reconstruction. In addition, certain types of neural networks⁶⁰ could be explored to better bridge the sim-to-real gap.

The sensor hardware, distribution, and reconstruction framework in this study provides a potentially general and practical solution to high-accuracy proprioception for various types of soft robots, which represents a significant advancement over previous attempts in this area. Our work may be applied to shape sensing in wearable or implantable stretchable devices and even shape reconstruction of soft-bodied biological systems, such as octopus arms⁶¹ and elephant trunks.⁶² Furthermore, integrating other types of sensors, such as pressure sensors,^63,64 temperature sensors,⁶⁵ and chemical sensors,⁶⁶ can enable exteroception and multimodal perception in soft robots for real-world applications.

Authors’ Contributions

L.L. and H.Z. conceived the concept and designed the research. L.L. developed the shape reconstruction framework and performed numerical studies. X.H., X.Z., H.X., and L.L. performed sensor design, fabrication, and characterization. L.L., X.H., X.Z., B.Z., V.M.T., K.L., and Z.S. performed shape reconstruction experiments. L.L., X.H., and H.Z. analyzed the data. L.L. and H.Z. wrote the paper, with input from all co-authors. H.Z. supervised the project.

Footnotes

Acknowledgments

The authors thank Dr. S.K. Gupta and H. Ye for their assistance on the gripper experiments.

Author Disclosure Statement

The authors declare that they have no competing financial interests.

Funding Information

This work was supported by the Office of Naval Research Young Investigator Award (ONR YIP; award number N00014-24-1-2030).

Data Availability

The data that support the findings of this study are available within the main text and the Supplementary Information. All data are also available from the corresponding authors upon reasonable request.

Supplemental Material

References

Rus

, Tolley

. Design, fabrication and control of soft robots. Nature, 2015; 521(7553):467–475.

Cianchetti

, Laschi

, Menciassi

, et al. Biomedical applications of soft robotics. Nat Rev Mater, 2018; 3(6):143–153.

Shintake

, Cacucciolo

, Floreano

, et al. Soft robotic grippers. Adv Mater, 2018; 30(29):1707035.

Liu

, Hou

, Li

, et al. Intelligent soft robotic grippers for agricultural and food product handling: A brief review with a focus on design and control. Adv Intell Syst, 2023; 5(12):2300233.

Galloway

, Becker

, Phillips

, et al. Soft robotic grippers for biological sampling on deep reefs. Soft Robot, 2016; 3(1):23–33.

Aracri

, Giorgio-Serchi

, Suaria

, et al. Soft robots for ocean exploration and offshore operations: A perspective. Soft Robot, 2021; 8(6):625–639.

der Maur

, Djambazi

, Haberthür

, et al. RoBoa: Construction and evaluation of a steerable vine robot for search and rescue applications. In: 2021 IEEE 4th International Conference on Soft Robotics (RoboSoft) New Haven, CT, USA; 2021; pp. 15–20.

Runciman

, Darzi

, Mylonas

. Soft robotics in minimally invasive surgery. Soft Robot, 2019; 6(4):423–443.

Gorissen

, De Volder

, Reynaerts

. Chip-on-tip endoscope incorporating a soft robotic pneumatic bending microactuator. Biomed Microdevices, 2018; 20(3):73.

10.

Lin

, Wang

, Zhao

, et al. Recent advances in perceptive intelligence for soft robotics. Adv Intell Syst, 2023; 5(5):2200329.

11.

Hegde

, Su

, Tan

JMR

, et al. Sensing in soft robotics. ACS Nano, 2023; 17(16):15277–15307.

12.

Wang

, Totaro

, Beccai

. Toward perceptive soft robots: Progress and challenges. Adv Sci (Weinh), 2018; 5(9):1800541.

13.

Dou

, Zhong

, Cao

, et al. Soft robotic manipulators: Designs, actuation, stiffness tuning, and sensing. Adv Mater Technol, 2021; 6(9):2100018.

14.

Kim

, Chang

H-S

, Shih

C-H

, et al. A physics-Informed, vision-based method to reconstruct all deformation modes in slender bodies. In: 2022 International Conference on Robotics and Automation (ICRA). IEEE: Philadelphia, PA, USA; 2022; pp. 4810–4817.

15.

Wang

, Wang

, Du

, et al. Real-Time soft body 3D proprioception via deep vision-based sensing. IEEE Robot Autom Lett, 2020; 5(2):3382–3389.

16.

Scharff

RBN

, Fang

, Tian

, et al. Sensing and reconstruction of 3-D deformation on pneumatic soft robots. IEEE/ASME Trans Mechatron, 2021; 26(4):1877–1885.

17.

Scharff

RBN

, Doornbusch

, Doubrovski

, et al. Color-based proprioception of soft actuators interacting with objects. IEEE/ASME Trans Mechatron, 2019; 24(5):1964–1973.

18.

Martin

, Bruder

, Wood

. A proprioceptive method for soft robots using inertial measurement units. In: 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE: Kyoto, Japan; 2022; pp. 9379–9384.

19.

Hughes

, Stella

, Santina

, et al. Sensing soft robot shape using IMUs: an experimental investigation. In: Experimental Robotics. ( Siciliano

, Laschi

, Khatib

eds) Springer International Publishing: Floriana, Malta; 2021; pp. 543–552.

20.

Meng

, Fang

, Yang

, et al. Spring-IMU fusion-based proprioception for feedback control of soft manipulators. IEEE/ASME Trans Mechatron, 2024; 29(2):832–842.

21.

, Fu

, Zhang

, et al. Inverse kinematics solving of a novel soft manipulator with vision and IMU data. In: 2023 International Conference on Service Robotics (ICoSR). IEEE: Shanghai, China; 2023; pp. 112–117.

22.

Pei

, Stella

, Meebed

, et al. IMU based pose reconstruction and closed-loop control for soft robotic arms. In: 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE: Abu Dhabi, United Arab Emirates; 2024; pp. 1847–1852.

23.

Van Meerbeek

, De Sa

, Shepherd

. Soft optoelectronic sensory foams with proprioception. Sci Robot, 2018; 3(24):eaau2489.

24.

Galloway

, Chen

, Templeton

, et al. Fiber optic shape sensing for soft robotics. Soft Robot, 2019; 6(5):671–684.

25.

Lun

TLT

, Wang

, Ho

JDL

, et al. Real-time surface shape sensing for soft and flexible structures using fiber bragg gratings. IEEE Robot Autom Lett, 2019; 4(2):1454–1461.

26.

, Cui

, Dong

, et al. Shape reconstruction of soft continuum robots via the fusion of local strains and global poses. Cell Rep Phys Sci, 2024; 5(10):102224.

27.

Zhuang

, Sun

, Li

, et al. FBG based shape sensing of a silicone octopus tentacle model for soft robotics. Optik, 2018; 165:7–15.

28.

Wall

, Zoller

, Brock

. A Method for sensorizing soft actuators and its application to the RBO hand 2. In: 2017 IEEE International Conference on Robotics and Automation (ICRA). IEEE: Singapore; 2017; pp. 4965–4970.

29.

Tapia

, Knoop

, Mutný

, et al. MakeSense: Automated sensor design for proprioceptive soft robots. Soft Robot, 2020; 7(3):332–345.

30.

Cianchetti

, Renda

, Licofonte

, et al. Sensorization of continuum soft robots for reconstructing their spatial configuration. In: 2012 4th IEEE RAS & EMBS International Conference on Biomedical Robotics and Biomechatronics (BioRob). IEEE: Rome, Italy; 2012; pp. 634–639.

31.

Wang

, Xie

, Xin

, et al. Sensing expectation enables simultaneous proprioception and contact detection in an intelligent soft continuum robot. Nat Commun, 2024; 15(1):9978.

32.

, Kim

, et al. Shape estimation of soft manipulator using stretchable sensor. Cyborg Bionic Syst, 2021; 2021:9843894.

33.

Stella

, Santina

, Hughes

. Soft robot shape estimation with IMUs leveraging PCC kinematics for drift filtering. IEEE Robot Autom Lett, 2024; 9(2):1945–1952.

34.

Lilge

, Barfoot

, Burgner-Kahrs

. Continuum robot state estimation using Gaussian process regression on SE(3). Int J Robot Res, 2022; 41(13–14):1099–1120.

35.

Cangan

, Navarro

, Yang

, et al. Model-based disturbance estimation for a fiber-reinforced soft manipulator using orientation sensing. In: 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE: Kyoto, Japan; 2022; pp. 9424–9430.

36.

Shih

, Shah

, Li

, et al. Electronic skins and machine learning for intelligent soft robots. Sci Robot, 2020; 5(41):eaaz9239.

37.

, Giorgio-Serchi

, Zhang

, et al. Stretchable e-skin and transformer enable high-resolution morphological reconstruction for soft robots. Nat Mach Intell, 2023; 5(3):261–272.

38.

Truby

, Santina

, Rus

. Distributed proprioception of 3D configuration in soft, sensorized robots via deep learning. IEEE Robot Autom Lett, 2020; 5(2):3299–3306.

39.

Sharma

, Najera-Flores

, Todd

, et al. Lagrangian operator inference enhanced with structure-preserving machine learning for nonintrusive model reduction of mechanical systems. Comput Methods Appl Mech Eng, 2024; 423:116865.

40.

Alora

, Cenedese

, Schmerling

, et al. Data-driven spectral submanifold reduction for nonlinear optimal control of high-dimensional robots. In: 2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE: London, United Kingdom; 2023; pp. 2627–2633.

41.

Armanini

, Boyer

, Mathew

, et al. Soft robots modeling: A structured overview. IEEE Trans Robot, 2023; 39(3):1728–1748.

42.

Huang

, Liu

, Lin

, et al. High-stretchability and low-hysteresis strain sensors using origami-inspired 3D mesostructures. Sci Adv, 2023; 9(34):eadh9799.

43.

Chen

, Zhang

, Luo

, et al. Superelastic, sensitive, and low hysteresis flexible strain sensor based on wave-patterned liquid metal for human activity monitoring. ACS Appl Mater Interfaces, 2020; 12(19):22200–22211.

44.

Kurian

, Souri

, Mohan

, et al. Highly stretchable strain sensors based on polypyrrole-silicone rubber composites for human motion detection. Sensor Actuat A-Phys, 2020; 312:112131.

45.

Xia

, Wang

, Zhai

, et al. Highly linear and low hysteresis porous strain sensor for wearable electronic skins. Compos Commun, 2021; 26:100809.

46.

J-H

, Zhao

, Zha

X-J

, et al. Multilayer structured AgNW/WPU-MXene fiber strain sensors with ultrahigh sensitivity and a wide operating range for wearable monitoring and healthcare. J Mater Chem A, 2019; 7(26):15913–15923.

47.

, Chhetry

, Kim

, et al. Hysteresis-free double-network hydrogel-based strain sensor for wearable smart bioelectronics. ACS Appl Mater Interfaces, 2022; 14(27):31363–31372.

48.

Zhang

, Zhang

, Wu

, et al. Liquid metal interdigitated capacitive strain sensor with normal stress insensitivity. Adv Intell Syst, 2022; 4(4):2100201.

49.

Wang

, Deng

, Jiang

, et al. Low-hysteresis, pressure-insensitive, and transparent capacitive strain sensor for human activity monitoring. Microsyst Nanoeng, 2022; 8(1):113.

50.

Cao

P-J

, Liu

, Asghar

, et al. A stretchable capacitive strain sensor having adjustable elastic modulus capability for wide‐range force detection. Adv Eng Mater, 2020; 22(3):1901239.

51.

Yoshida

, Wang

Y-F

, Sekine

, et al. Printed low-hysteresis stretchable strain sensor based on a self-segregating conductive composite. ACS Appl Eng Mater, 2023; 1(1):50–58.

52.

Yuan

, Zhao

, Li

, et al. Flexible, anisotropic strain sensor based on interdigital capacitance for multi-direction discrimination. Sensor Actuat A-Phys, 2023; 359:114459.

53.

Svanberg

. The method of moving asymptotes—a new method for structural optimization. Numerical Meth Engineering, 1987; 24(2):359–373.

54.

Liu

, Lee

. A design of auxetic metamaterial with locally resonant bandgaps using topological optimization. Int J Appl Mechanics, 2023; 15(06):2350048.

55.

, Luo

, Li

, et al. Level-set topology optimization for mechanical metamaterials under hybrid uncertainties. Comput Methods Appl Mech Eng, 2017; 319:414–441.

56.

Mosadegh

, Polygerinos

, Keplinger

, et al. Pneumatic networks for soft robotics that actuate rapidly. Adv Funct Mater, 2014; 24(15):2163–2170.

57.

Souri

, Banerjee

, Jusufi

, et al. Wearable and stretchable strain sensors: Materials, sensing mechanisms, and applications. Adv Intell Syst, 2020; 2(8):2000039.

58.

Fang

, Tian

, Yang

Z-X

, et al. Efficient Jacobian-based inverse kinematics with sim-to-real transfer of soft robots by learning. IEEE/ASME Trans Mechatron, 2022; 27(6):5296–5306.

59.

Tian

, Fang

, Su

, et al. Function based sim-to-real learning for shape control of deformable free-form surfaces. In: Robotics. Science and Systems Robotics Science and Systems Foundation: Delft, Netherlands; 2024; p. 98.

60.

Gao

, Michelis

, Spielberg

, et al. Sim-to-real of soft robots with learned residual physics. IEEE Robot Autom Lett, 2024; 9(10):8523–8530.

61.

Wang

, Halder

, Gribkova

, et al. Modeling the Neuromuscular Control System of an Octopus Arm. 2022. Available from: https://arxiv.org/abs/2211.06767v1

62.

Dagenais

, Hensman

, Haechler

, et al. Elephants evolved strategies reducing the biomechanical complexity of their trunk. Curr Biol, 2021; 31(21):4727–4737.

63.

, Zhang

, Jang

, et al. Highly sensitive capacitive pressure sensors over a wide pressure range enabled by the hybrid responses of a highly porous nanocomposite. Adv Mater, 2021; 33(48):2103320.

64.

Roberts

, Zadan

, Majidi

. Soft tactile sensing skins for robotics. Curr Robot Rep, 2021; 2(3):343–354.

65.

Zou

, Wang

, Yang

. An intrinsically embedded pressure-temperature dual-mode soft sensor towards soft robotics. Sensor Actuat A-Phys, 2021; 332:113084.

66.

, Li

, Solomon

, et al. All-printed soft human-machine interface for robotic physicochemical sensing. Sci Robot, 2022; 7(67):eabn0495.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

48.06 MB

19.58 MB

6.20 MB

0.00 MB