A Deep Learning Framework for Soft Robots with Synthetic Data

Abstract

Data-driven methods with deep neural networks demonstrate promising results for accurate modeling in soft robots. However, deep neural network models rely on voluminous data in discovering the complex and nonlinear representations inherent in soft robots. Consequently, while it is not always possible, a substantial amount of effort is required for data acquisition, labeling, and annotation. This article introduces a data-driven learning framework based on synthetic data to circumvent the exhaustive data collection process. More specifically, we propose a novel time series generative adversarial network with a self-attention mechanism, Transformer TimeGAN (TTGAN) to precisely learn the complex dynamics of a soft robot. On top of that, the TTGAN is incorporated with a conditioning network that enables it to produce synthetic data for specific soft robot behaviors. The proposed framework is verified on a widely used pneumatic-based soft gripper as an exemplary experimental setup. Experimental results demonstrate that the TTGAN generates synthetic time series data with realistic soft robot dynamics. Critically, a combination of the synthetic and only partially available original data produces a data-driven model with estimation accuracy comparable to models obtained from using complete original data.

Introduction

Soft robots aim to transcend traditional robots for use in unstructured or unpredictable environments.¹ This is achieved by designing flexible and compliant robots that are able to conform to their surrounding structure.² However, obtaining a kinematic or dynamic model of a soft robot is analytically challenging due to the diverse behaviors, inherent nonlinearities, and theoretically infinite degrees of freedom.³ Taking these difficulties into account, previous efforts on analytical modeling typically include simplifications and assumptions in their formulations.⁴

For example, kinematic models in previous studies^5–9 are derived using the piecewise constant curvature simplifications, whereas the dynamic models in previous studies^10–15 are obtained via Euler–Lagrangian formulation or Cosserat rod model with quasi-static assumptions. Based on these approximations, the acquired analytical models may not accurately capture the complex nonlinear dynamics of a soft robot and thus do not necessarily guarantee a good estimation performance.¹⁶ A more complex model may solve this problem, but the process of deriving such models is tedious considering that soft robots composed of low Young's modulus materials manifest highly nonlinear characteristics.

To mitigate the difficulties in modeling soft robots analytically, data-driven modeling has been extensively studied and successfully applied for soft robots. For instance, Elgeneidy et al.¹⁷ applied a feed-forward network to estimate the curvature angles of a pneumatic actuator. Thuruthel et al.¹⁸ demonstrated the significant potentials of deep neural networks as an empirical model to estimate the tip position and contact force of a soft finger. Wang et al.¹⁹ paired a convolutional neural network with an autoencoder to reconstruct the three-dimensional (3D) shape of soft bodies under deformation. However, these data-driven approaches assume the availability of high-quality data representative of the soft robots.²⁰

Unfortunately, data collection is often time-consuming as the soft robot would need to be exhaustively actuated to sufficiently cover its task space.^21,22 Furthermore, for some potential soft robot applications such as underwater exploration or surgical robotics, data collection processes can be expensive and limited. For instance, in underwater grasping²³ or exploration,²⁴ it can be tedious and costly to send the robot underwater to collect data for behavioral modeling.²⁵ Besides this, collecting medical soft robot data poses its own challenges due to legal and ethical concerns.²⁶ Evidently, data-driven models become difficult to implement when there are insufficient data. Models trained on structurally incomplete data fall short at capturing the nonlinear full-range dynamics of the complex soft robots (underfitting).²¹ On the contrary, models could also fit abnormally well to the limited training data (overfitting).²⁷

Existing approaches alleviate this requirement of voluminous data using simple or nonparametric models. For example, regression analysis is used in Elgeneidy et al.¹⁷ for bending angle prediction. Similarly, a local Gaussian process is utilized in Fang et al.²⁸ for online control. These methods rely less on data availability and thus are less prone to overfitting at the cost of having less expressive models. An alternative approach is to use empirical simulation models or environments. For example, Runge et al.²⁹ and Massari et al.³⁰ generated data for neural network training via the finite element method (FEM). Nevertheless, both data-driven and FEM approaches lack generalization ability in addressing factors such as altering material properties, inconsistent fabrication, and changes in environment. A slight variation in these factors could result in substantial differences between the developed models and the actual soft robots.²⁹ Therefore, data would have to be collected regularly to fine-tune these models via costly and laborious data collection process.

To generalize empirical models trained on simulated environments to real-world scenarios, recent works incorporate sim-to-real transfer strategies to reduce the sim-to-real gap^31,32 in FEM-simulated data for soft robot design,³³ modeling and control,^33–35 and state estimation.^36,37 Alternatively, previous studies^38–40 employ domain randomization, whereby the dynamics of a simulated environment are randomly changed, thus allowing a trained model to generalize better to unseen real-world scenarios. These methods attempt to bridge the sim-to-real gap by modeling discrepancies between the simulated and real-world domains as variability in the simulated domain. Besides this, the works^41,42 have used differentiable simulators to generate data for data-driven models.

These models provide an avenue for reducing the sim-to-real gap by developing efficient, gradient-based optimization algorithms to find the simulation parameters that best fit the observed sensor readings. However, these analytical models can only predict the dynamical behavior of systems for which they have been designed. Apart from that, the underlying mathematical formulation of high-fidelity simulation models on highly complex environments is time-consuming to develop, due to the nonlinear characteristics of soft robots.^43,44 Taking this into consideration, data-driven frameworks offer a straightforward alternative to model nonlinear, complex soft robot behaviors and interactions by omitting the mathematical rigor of simulation models, although at the expense of model explainability.

Apart from the aforementioned techniques, there are also methods or techniques that generate synthetic data by extrapolating the original data. For example, data augmentation applies random transformations to the original training data by introducing random noise also known as jittering,⁴⁵ scaling,⁴⁶ rotation or flipping,⁴⁷ and time-warping.⁴⁶ Several works have adopted data augmentation techniques in the robotics field, for instance, Su et al.⁴⁸ used data augmentation techniques to improve image segmentation in agricultural robots. Liu and Li⁴⁹ used perspective transformation to expand existing data sets for estimating the motion of an eye-in-hand robotic system. Similarly, data augmentation can also be applied to time series robotic data.

For example, Wang and Majewicz Fey⁵⁰ and Lakomkin et al.⁵¹ used data augmentation techniques to prevent overfitting on their inference models for robot-assisted surgery and human–robot speech interaction, respectively. However, the above-mentioned methods rely on random transformations of the original time series, which might deviate from the true underlying distribution of the soft robots. In addition, these transformations are applicable to only specific data sets (i.e., vision data). Alternatively, meta-learning is a machine learning technique that attempts to learn from a small set of data by exploiting previously learned knowledge. However, meta-learning in robotics is primarily based on vision data, which may not be directly applicable for soft robots with complex dynamics in a time series domain.²¹

Recently, generative neural network models have gained immense traction due to their ability to accurately transfer from a random distribution (typically uniform noise) to a desired distribution, allowing explicit generation from the true underlying data distribution.⁵² In addition, these models are able to extrapolate beyond the distribution of the original data, which can potentially resolve the sim-to-real gap.⁵³ Generative adversarial nets (GANs)⁵⁴ are a class of deep generative models capable of generating realistic synthetic data by learning the dynamics of the original data through an adversarial learning process.

Several works have employed GAN models for domain adaptability by transferring simulated images to realistic ones to reduce the real-to-sim gap in rigid robots for tasks, such as grasping^55–57 and object interaction.⁵⁷ In contrast, for our study, we exploit the generative capability of GANs in expanding the volume of a limited time series data set to improve the training of deep neural networks for soft robot application,^58,59 at a reduced effort and cost of data collection.⁶⁰ In addition, GANs provide a data-driven framework for generating synthetic data, thus extricating dependency on the expert knowledge required to construct high-fidelity analytical and FEM models.

Commonly, data from the domain of soft robots are composed of real-valued time series readings sourced from sensors, such as flex sensors,^17,18 pressure readings^20,61 and force magnitudes.^62,63 Therefore, it is imperative that the synthetic samples generated using GANs resemble the nonlinear and multivariate time series of soft robotic data, that is, time series that will most likely be nonlinear and consist of more than one time-dependent variables.¹⁸ Recently, a few pioneering works have successfully adapted GANs for the time series domain.^64–66 However, these works were not specifically designed to address the complexities of the soft robotics domain.

To this end, we propose a novel time series GAN model, Transformer-TimeGAN (TTGAN), for realistic soft robotic data generation. Concretely, we aim to capture the nonlinear multivariate properties of the data by proposing a GAN architecture for handling time series, integrated with self-attention networks to accurately model the possible temporal dependencies in the data.^66,67 We then show that our proposed GAN model can be used to compensate for the large data requirement in deploying data-driven models for soft robotics. We test our proposed methodology on a widely used soft robotic platform—a pneumatic soft gripper (PSG).^15,20,68 This platform facilitates the collection of a full data set needed for adequate result validation while also being sufficiently representative of a complex soft robotic time series. In detail, we collect a soft robotic data set composed of pressure, flex, force, and motion variables sourced from sensors within our pneumatic gripper platform. We then use the proposed GAN model to generate synthetic samples of data, which we combine with a small amount of original data to train a data-driven model for a multimodal sensing task.

Furthermore, considering that soft robots intrinsically have high behavioral diversity due to their soft and compliant body,^69–71 it would be beneficial to isolate data generation to select desired behaviors for more accurate representation learning. Therefore, we introduce a conditioning vector as in Dai et al.⁷² to our proposed architecture producing Conditional TTGAN (CTTGAN), enabling data generation of specific robot behaviors based on the corresponding conditional vector.

To the best of our knowledge, this is the first work that investigates the feasibility of GANs for synthetic data generation, as an alternative to the costly and laborious data collection process in soft robotics. In summary, our contributions in this article are highlighted as follows: 1.

We propose a novel TTGAN, a time series generative model for synthetic data generation. Here, we forgo the commonly used autoregressive networks and opt instead for attention networks to capture the nonlinear and multivariate dynamics of soft robotic data in a more succinct latent space to aid the learning of the GAN network. Experimental results show that our model is able to better capture the dynamics of soft robotic data and generate more realistic time series compared with the relevant state of the art.

We adopt the conditional framework into our model, namely CTTGAN, by introducing conditional labeled vectors when training and generating data. Conditional vectors afford control over the distribution of the synthetic data generated, which allow data generation of specific robot behaviors. Experimental results show that the CTTGAN model is able to complement data sets skewed to specific soft robot behavior.

To demonstrate the validity of our proposed approach, we design, fabricate, and collect data on three different soft robot platforms. Experimental results show that by combining synthetic data with a small subset of original data, we achieved comparable accuracy to that of a model trained using full-length original data on all three soft robot platforms.

The results presented in this work fundamentally depict a viable alternative to costly and exhaustive data collection required for data-driven methods in soft robotics.

Materials

Design and fabrication

We use a popular class of soft actuators known as PneuNet⁶⁸ to design and develop a pneumatic-based soft gripper used in our experiments. Our PSG is composed of three individual pneumatic soft fingers (PSFs) held onto a 3D-printed holder, with each PSF being 120° apart from the other, as shown in the top left box in Figure 1. The PSFs are constructed following identical fabrication processes and are composed of two sections. The first section consists of a main body made up of a series of channels and chambers. The second section consists of a base layer embedded with a flex sensor (4.5″; SparkFun) and an inelastic material (paper) to make the layer inextensible. Both the main body and the base layer of the PSF are formed using 3D-printed silicone (EcoFlex 0050; Smooth-On, Inc.) mold.

FIG. 1.

Experimental setup for PSG data collection. Motion capture cameras are focused on the PSG platform to capture its movement by tracking the reflective markers placed along the body of each of the PSFs. Red arrows on the table show the three axes (X, Y, and Z) of the resultant forces. Shown in the top left, the D-D plane shows the top view of the PSG relative to the contact bulb. Blue arrows indicate two-axis (X, Z) PSF contact forces exerted by a single PSF, whereas red arrows indicate the three-axis (X, Y, Z) resultant grasping forces exerted by the compounded PSG. Data are collected for two soft robotic behaviors under two actuation patterns. In total, four scenarios were captured as shown on the right. PSF, pneumatic soft fingers; PSG, pneumatic soft gripper.

The connections of the flex sensor are soldered, and the PSF is fixed with a sharp-end pneumatic pipe as an air inlet. Finally, the base layer and the main body are joined together using additional silicone. Although the fabrication processes are identical, the act of manually replicating the PSFs along with the use of soft materials resulted in slight variations such as asymmetrical air columns. In addition, the flex sensor embedded in each of the PSFs exhibited inconsistent (out-of-distribution) sensor dynamics, resulting in different sensor responses to the same PSF bending configuration. Such variability is not uncommon as the task of achieving consistent dynamics for sensors would typically require a precise automated fabrication process in a highly controlled environment, which may not be economically practical. Nevertheless, with sufficient data, a data-driven approach in modeling the gripper would be able to account for inconsistent dynamics of the individual PSFs.

Experimental procedure

An electropneumatic regulator (ITV1030; SMC Corporation) connected to a pneumatic supply is used to modulate the pressure of our pneumatic system using pulse width modulation (PWM) signals.⁷³ The modulated pressure is supplied to each PSF inlet and a pressure sensor (MPXH6400A; NXP) in parallel, where the pressure inside the PSFs is assumed to be the same as the measured pressure. The PWM signal is controlled using a microcontroller (PSoC^®5LP) that manipulates the duty cycle (control input) of the PWM signal.

Motion of the PSG is captured by five cameras (OptiTrack Flex13; NaturalPoint, Inc.) that continuously measure the coordinates of 10 reflective ball-markers placed along the inextensible base layer of each PSF. Furthermore, a sensing circuit is used to convert the change in resistance of the embedded flex sensor to voltage readings. For simplicity, the sensor values are collected directly without any filtering or amplification. Contact and grasping force measurements are collected using a multi-axis load cell (Axia80; ATI Industrial Automation, Inc.) with a contact bulb attached on top of it. Here, we consider the forces applied on the X, Y, and Z axes of the load cell, as shown in Figure 1.

The PSG is actuated using a series of pseudorandom pressure levels, within a predetermined pressure range. Two different actuation patterns, namely Oscillatory actuation and Random actuation, are used to verify the generality of our approach to different input signals. In the former, two types of input voltage signals are fed to the electropneumatic regulator to generate a gradual oscillatory pattern, and each PSF is actuated with the same input pressure. In the latter, a faster random pressure actuation pattern is used, and each PSF is independently actuated with random input pressure.

For each actuation pattern, data were collected for two experimental behaviors. In the first behavior, the PSG is actuated without any obstructions in its task space (Free Bending), whereas in the second behavior, the PSG comes in contact with the contact bulb near its fingertips to mimic surface contact (Tip Contact). The contact bulb is placed in the center of the three PSFs to measure the resultant grasping force of the PSG.

Data collection

Accounting for the two actuation patterns and experimental behaviors, data were collected for a total of four scenarios, namely Oscillatory Free Bending, Oscillatory Tip Contact, Random Free Bending, and Random Tip Contact, as depicted in Figure 1 (right). Each data sample contains measurements for the actuation pressure and flex sensor voltage for the three PSFs. Furthermore, the 10 markers placed on each of the PSF are recorded as 3D-Cartesian coordinates to capture the full motion of the PSG (30 markers total). Finally, contact force is captured as the resultant force applied by the PSG along its X, Y, and Z axes totaling to 99 measurements per sample. Data were collected at a rate of 10 Hz using MATLAB.¹⁸ We partitioned the data into training (9000 samples per scenario) and testing sets (3000 samples per scenario).

Methods

Synthetic data generation

Figure 2 illustrates the proposed architectures, training, and data generation process. To generate realistic synthetic data using the proposed TTGAN model, we first describe how we prepare our data to be fed into the model. Let X_t be a vector of features (flex, pressure, force, position markers) that makes up the sequence $X_{1 : T}$ , where T represents the sequence length. $X_{1 : T}$ is then divided into N groups of overlapping time windows, $x_{t, 1 : N} = X_{t : t + n - 1}$ of length n. For simplicity, subscripts N are omitted unless explicitly required. The model then takes randomly shuffled batches of x_t as input and updates the parameters of the network appropriately.

FIG. 2.

Architecture diagram for the training and usage of the proposed TTGAN and CTTGAN generative models. The numbers inside the brackets shown in the figure correspond to the equations explained in the text. TTGAN, Transformer TimeGAN; CTTGAN, Conditional Transformer TimeGAN.

We describe how each component of the network is used and trained as follows. The autoencoder network is composed of an encoder and recovery network, which are used to provide a reversible mapping between feature and latent spaces. The former maps the input, x_t, to its corresponding latent code, h_t, whereas the latter works in the opposite direction, mapping the latent codes back to their feature representation, ${\tilde{x}}_{t}$ . In this work, we use the term, recovery, instead of the conventional term, decoder, to avoid confusing it with the discriminator in TTGAN. $h_{t} = E (x_{t} + P E) .$ (1)

{\tilde{x}}_{t} = R (h_{t} + P E) .

(2)

These are shown formally in Equations (1) and (2) where both E and R are parameterized by Transformer encoders, and PE are the position embeddings proposed in Vaswani et al.⁶⁷ Overall, the autoencoder network aims to provide accurate reconstructions, ${\tilde{x}}_{1 : T}$ , of $x_{1 : T}$ from its latent representation $h_{1 : T}$ . This is carried out by optimizing the reconstruction loss, $ℒ_{R}$ , as shown in Equation (3), where $d (.)$ is an appropriate distance function. In this work, we use mean squared error (MSE) as follows: $ℒ_{R} = \sum_{t = 1}^{T} d (x_{t}, {\tilde{x}}_{t}) .$ (3) $ĥ_{t} = G (z_{t}) o r ĥ_{t} = G (z_{t}, c_{t}) .$ (4)

The GAN network is composed of a Generator, G, and Discriminator, D, which operate within the latent space produced by the autoencoder network. G is fed in a random noise vector sampled independently of a uniform distribution, $z_{t} \sim U (0, 1)$ , and produces synthetic latent codes, $ĥ_{t}$ . In essence, G learns a mapping function of $z_{t} \to ĥ_{t}$ , as shown in Equation (4). In the case of CTTGAN, G is conditioned on auxiliary information in the form of a conditional vector, c_t, which is concatenated with the input of G. In this work, we generate four distinctive one-hot encoded vectors as c_t to represent each of the captured behaviors (Oscillatory Free Bending, Oscillatory Tip Contact, Random Free Bending, and Random Tip Contact). With one-hot encoding, each categorical value in the data is converted into a new categorical column and assigned a binary value of 1 or 0 such that each encoding is unique to a behavior. Nonetheless, in general, c_t can be represented by any categorical encoding method. $ỹ_{t} = D ({\tilde{h}}_{t}) o r ỹ_{t} = D ({\tilde{h}}_{t}, c_{t}) .$ (5)

Conversely, the Discriminator, D, is fed with real or synthetic data points and attempts to provide a binary classification feedback. ${\tilde{h}}_{t}$ is either a real or synthetic data point, and $ỹ_{t}$ is the subsequent classification of ${\tilde{h}}_{t}$ . Similar to G, D is additionally fed with c_t in the case of CTTGAN, as shown in Equation (5). $ℒ_{U} = \sum_{t = 1}^{T} d (y_{t}, ỹ_{t}) + \sum_{t = 1}^{T} d (ŷ_{t}, ỹ_{t}) .$ (6)

ℒ_{S} = \sum_{t = 1}^{T} d (h_{t}, ĥ_{t}) .

(7)

Both G and D are parameterized by a 3-layered long short-term memory (LSTM) network, each with 24 hidden dimensions. To train G and D, the GAN framework is utilized where the networks are trained with differing objectives of the same loss. This loss, $ℒ_{U}$ , is calculated by using the binary classification of a sample $ĥ_{t}$ and calculating its distance from the ground truth. This loss can be calculated in an unsupervised manner, meaning that no labels are needed, and is shown formally in Equation (6), where y_t represents the classification for a real data point and $ŷ_{t}$ represents the classification for a synthetic data point.

We also used the supervised loss proposed in Yoon et al.⁶⁶ to aid the model in synchronizing the latent dynamics of the real and synthetic data. Using maximum likelihoods, this yields the supervised loss, $ℒ_{S}$ , as shown in Equation (7). Both TTGAN and CTTGAN are trained for 50 epochs, and the Adam optimizer is used for back-propagation with a learning rate of 0.0005. We set the value of n, the time windows, at 50. Once the TTGAN model has been trained, synthetic data can be generated as shown by the orange arrows in Figure 2, where only the trained Generator, G, and Recovery, R, networks are utilized.

Evaluation of synthetic data

To quantitatively analyze the generated data, we measure how the synthetic data affects the representational learning of an empirical model used to estimate perceptive variables of our soft robotic system, that is, a multimodal sensing task. We devise two experiments that reflect realistic scenarios, which are referred to as Experiments A and B. In both experiments, we use LSTM networks as the backbone of our inference model.⁷⁴ Note that this backbone is model-agnostic, and any suitable time series network can be used to parameterize the inference model. In this work, we use a three-layered LSTM model with each layer having ${128, 64, 32}$ , where each entry of the curly brackets is a hidden layer and the value denotes layer width.

Using flex and pressure readings as the input sequence, the inference model is used to estimate the target sequence composed of the (1) bending state of the PSG via the 30 position markers, $P (t)$ , and (2) resultant contact force applied by the PSG, $F (t)$ . This mapping is described in Equations (8) and (9) for force estimation and position markers estimation, where $f (t)$ represents the flex sensor reading, $ρ (t)$ represents the input pressure to the actuator, $F (t)$ is force measurement, $P (t)$ are position markers, $c_{M} (t)$ is the current state of the inference model network, and $\tilde{P} (t)$ and $\tilde{F} (t)$ are vectors representing the estimated position markers and force, respectively. $\tilde{P} (t)$ and $\tilde{F} (t)$ are then compared with the target sequence, $P (t)$ and $F (t)$ , and a loss function (MSE in this work) is used to back-propagate and train the inference model. This is highlighted by the blue arrows in Figure 2. $[f (t) + ρ (t) + c_{M} (t)] \to \tilde{F} (t) .$ (8)

[f (t) + ρ (t) + c_{M} (t)] \to \tilde{P} (t) .

(9)

The proposed TTGAN model can be useful in conditions where data collection is limited (i.e., due to physical, cost, or time constraints) and can be used to uniformly increase the volume and variety of a data set. We verify this through Experiment A, as also explained in Figure 3. We simulate this condition by arbitrarily removing half of the collected training data from each of the collected scenarios (resulting in the Half data set). We train the TTGAN model using only the Half data set and produce an equal amount of synthetic data. We also generate synthetic data using popular time series data augmentation techniques,⁵³ such as jittering, scaling, rotation, time warping,⁴⁶ random guided warping,⁷⁵ and suboptimal warped time series generator (Spawner),⁷⁶ as a fair comparison to the proposed method.

FIG. 3.

Experimental scenarios investigated. In Experiment A, each scenario as well as the concatenation of all the scenarios (Overall) are alternatively used as the Complete data set. In contrast, Experiment B only uses Overall as the Complete data set. Testing for both experiments follows the same methodology, whereby the trained inference models are evaluated using an unseen test set.

The inference model is then individually trained on four sets of data, namely the full training data (Complete), the halved training data (Half), the TTGAN-generated synthetic only data (Synthetic), and a mixture of the halved training data with the synthetic data sets generated (Mixed). Once trained, we expose the inference model to the test set for the first time and compute the root-mean-square error (RMSE) by comparing the estimated values with the ground truth. We then normalize the RMSE by dividing the maximum and minimum values producing the normalized root-mean-square error (NRMSE), which we use to quantitatively compare the information learned by the inference model trained on the different data sets.

However, uniformly removing data from each scenario might not reflect a real-world context. Such conditions are prevalent when data collection is costly for certain behaviors but inexpensive for others.²⁵ Similarly, instrument malfunction or software errors midway through data collection can lead to skewed data sets. Therefore, we investigate this condition using Experiment B, as also shown in Figure 3. Here, we skew our data set by randomly removing half of our training data from a target behavior (i.e., either Free Bending or Tip Contact) while keeping the other behavior intact. In contrast to the vanilla TTGAN, the CTTGAN model is able to dictate the distribution of data generated to match only the removed behavior, hence balancing the data set.

To evaluate the CTTGAN model, we compute the NRMSE of the inference model after training on four data sets: Complete—as above, the skewed data set with a target behavior removed (Skewed), a mixture of the Skewed data set and synthetic data generated with CTTGAN (Mixed-CTTGAN), and a mixture of Skewed and synthetic data generated with TTGAN (Mixed-TTGAN). Both the TTGAN and CTTGAN models are trained using only the Skewed data set. The following section analyzes the quality and usability of the synthetic data on the PSG platform. To further ascertain the adaptability of the proposed method, we included additional experiments in the Supplementary Section to first analyze the model's performance on more nonlinear behaviors using a single PSF; and second, to carry out a similar analysis on a multichannel pneumatic soft continuum (PSC) body capable of nonplanar bending.

Results and Discussion

Comparison of synthetic and original data

In this subsection, we analyze the quality of the synthetic data in retrospect to the original data. The analysis here focuses on the ability of TTGAN to capture the relationship between the input (i.e., flex and pressure) and output (i.e., force and position markers) variables for each of the scenarios collected. We also analyze the usability of the synthetic data by calculating a predictive score for each synthetic data set.

First, we carried out a t-distributed stochastic neighbor embedding (t-SNE) analysis on the original and synthetic data set. t-SNE is a widely used dimensionality reduction technique for visualizing high-dimensional data in a low-dimensional space. This assessment depicts how closely the distribution of synthetic samples matches that of the original samples in a two-dimensional space and is used as qualitative analysis. We run the t-SNE analysis for 300 iterations, setting the perplexity to 40. From Figure 4, we can observe that the generated data set bears a striking resemblance (in a lower dimensional space) to the original, based on their similarly evaluated t-SNE plots for all the scenarios. This result implies that the synthetic data generated are of high quality, in the sense that they are semantically similar to the original. Critically, this shows that the generative model is able to successfully capture the relationship between the input and output variables.

FIG. 4.

t-SNE visualization of the distribution of synthetic and original data variables (i.e., flex, pressure, force, and position markers) for each scenario is shown on the left where red dots denote original data and blue denotes synthetic data. On the right, we reconstruct the closest synthetic and original t-SNE variables back to the feature space using the k-means nearest-neighbor algorithm. Trajectories of each PSF (across time) are depicted as moving lines on the reconstructed position markers. A video demonstrating the real-time comparison of the original and synthetic data is provided as a Supplementary Movie S1. t-SNE, t-distributed stochastic neighbor embedding.

To determine the usability of the synthetic data, we measure how well the generated data capture the predictive characteristics of the original data by calculating a predictive score.²⁰ This is calculated by using the synthetic data set to train a post hoc sequence-prediction model, which predicts the next-step temporal vectors over each input sequence. This trained model is then evaluated on the original data set, producing the predictive score in terms of a mean absolute error (lower the better). The predictive scores computed for each of the synthetic data sets are shown in Table 1. The low predictive scores produced by the proposed TTGAN further exemplify the ability of the generative model in producing a usable synthetic data set for representational learning. Apart from that, we observe that TTGAN consistently generates synthetic data sets with better (lower) predictive scores than the baseline TimeGAN (i.e., a well-known GAN framework for time series as proposed in Yoon et al.⁶⁶) across all scenarios.

Table 1.

Mean Absolute Error Comparison of Predictive Scores of TimeGAN Versus Transformer TimeGAN

Model	Oscillatory actuation		Random actuation
Model	Free Bending	Tip Contact	Free Bending	Tip Contact
TimeGAN	0.0491 ± 0.0026	0.0442 ± 0.0020	0.0586 ± 0.0016	0.0623 ± 0.0011
TTGAN	0.0478 ± 0.0019	0.0430 ± 0.0015	0.0434 ± 0.0013	0.0573 ± 0.0009

Bold text indicates best performance.

TTGAN, Transformer TimeGAN.

To understand why TTGAN is able to produce synthetic data sets with better predictive scores, Figure 5 illustrates the t-SNE plots of the latent space produced by the autoencoder network of each model. We note that from the original distribution shown in Figure 5 (top), the Tip Contact scenarios have similar distributions and are clustered together in all three t-SNE plots. Both TimeGAN and TTGAN are able to cluster and segment Random Free Bending from other scenarios. However, while the TTGAN is able to cluster and segment the Oscillatory Free Bending scenario, TimeGAN fails to make a clear segregation. Evidently, the autoencoder in TTGAN is more equipped to capture the dynamics of the different scenarios, which allows the GAN network to learn the underlying dynamics of each scenario in greater detail.

FIG. 5.

t-SNE visualization of the latent space produced by the encoder component of TimeGAN (middle) and TTGAN (bottom) after training. Colored circles on the right represent the corresponding actuation state space of the PSG for each scenario. Note that Oscillatory Tip Contact and Random Tip Contact share the same state space.

Significance of synthetic data in compensating for the lack of original data

We analyze the significance of the synthetic data based on results of Experiment A, as shown in Table 2. Focusing on the Overall scenario, we note that although training on Complete provides the lowest NRMSE, we are able to get comparable performance, with a difference of only −0.311% for Force NRMSE and −0.062% for Position Markers NRMSE when training on the Mixed data set. In addition, we note that the inference model trained on the Mixed data set with synthetic data generated by TTGAN significantly produces a lower NRMSE than all the data augmentation comparison data sets. Likewise, the Mixed data set performs 2.271%/0.009% and 6.076%/1.003% better than the Half and Synthetic data sets for Force/Marker estimations. This implies that the synthetic data set is able to add useful diversity and volume to the halved original data set.

Table 2.

Comparison of Training Data Set and Corresponding Normalized Root-Mean-Square Error

Data set	Force (%)				Overall	Markers (%)				Overall
	Oscillatory actuation		Random actuation			Oscillatory actuation		Random actuation
	Free Bending	Tip Contact	Free Bending	Tip Contact		Free Bending	Tip Contact	Free Bending	Tip Contact
Complete	0.021	10.533	0.018	7.087	8.516	4.293	5.981	6.226	6.129	4.358
Half	0.041	10.656	0.052	7.686	11.098	4.346	6.022	6.292	6.280	4.429
Synthetic	0.050	14.206	0.151	9.270	14.903	5.484	8.006	6.722	6.744	5.423
Mixed
Jittering	0.266	10.554	0.175	7.557	10.643	4.304	6.003	6.305	6.250	4.550
RGW	0.008	10.563	0.026	7.470	10.002	4.266	6.043	6.300	6.207	5.167
Rotation	0.694	11.003	0.730	7.449	11.215	4.329	6.074	6.318	6.240	4.948
Scaling	0.014	10.677	0.042	7.411	10.956	4.307	5.996	6.357	6.327	4.781
Spawner	0.017	10.584	0.025	7.421	9.821	4.386	6.041	6.371	6.341	4.455
Time warping	0.020	10.638	0.011	7.468	9.976	4.304	6.020	6.138	6.293	4.529
TTGAN (proposed)	0.031	10.528	0.122	7.395	8.827	4.294	5.955	6.119	6.188	4.420

The inference model is independently trained on four data sets of different contexts, namely Complete, Half, Synthetic, and Mixed, as explained above. Time series estimation is performed for each of the scenarios collected as well as Overall where all scenarios are concatenated and estimated serially. The NRMSE results depicted are the result of averaging over five independent runs.

RGW, random guided warping.

To further analyze the performance of the synthetic data set for representational learning, Figure 6 illustrates the time series estimation of resultant force, and Figure 7 compares the position markers against the ground truth. These figures illustrate that the inference model suffers from overfitting when trained on the Half and Synthetic data sets. This is notably pronounced in the Free Bending behavior, where the model predicts high force values even when there is no contact. Attributed to the lack of data samples in the aforementioned data sets, this leads to weaker model generalization on the unseen test set.

FIG. 6.

Time series estimation of resultant force by the inference model compared with the ground truth when individually trained on each of the different training data sets—Complete, Half, Synthetic, and Mixed. (a) Resultant gripper force for Oscillatory Tip Contact and Random Tip Contact scenarios. (b) Resultant gripper force for Oscillatory Free Bending and Random Free Bending scenarios.

FIG. 7.

Comparison of reconstructed (ground truth) and estimated position markers (left) by the inference model when trained on the different data sets for each of the collected scenarios. The corresponding real gripper is shown on the right. Furthermore, the estimated and real trajectories of the tip marker coordinates (across time) are also illustrated. A video demonstrating the real-time reconstruction of the time series is provided as a Supplementary Movie S1.

In comparison, the Mixed data set circumvents this problem due to its higher volume of data, achieving performance comparable to the Complete data set and the ground truth. We also confirm this by comparing the training and validation loss of the inference model when trained on the different data sets. As shown in Supplementary Figure S5, the Half and Synthetic data sets either suffer from overfitting, where the training loss is low but the validation loss is high, or underfitting, where the training loss does not converge (i.e., for Force when trained on Half).

To verify the generality of our method, this experiment is repeated with smaller amounts of original data to train the generative model and the inference model. We plot the performance of the inference model when trained on five data sets of varying size, namely:

Complete: which is the full original data (9000 samples per scenario).

Original: which is the original data but only containing as many samples as shown on the x axis in Figure 8.

Synth 1: which contains an equal amount of synthetic data as Original.

Synth 2: which contains an equal amount of synthetic data as Complete.

Compensated: which compensates for the lack of data samples in Original by generating a complementary amount of synthetic data, resulting in a data set equal in length to Complete.

FIG. 8.

NRMSE of the inference model when trained on varying amounts of training samples. The x axis represents the amount of original data used to train the inference model and the generative model. The dashed red horizontal line shows the performance of the model when trained on the full original data set and serves as a lower bound. NRMSE, normalized root-mean-square error.

For Synth 1, Synth 2 and Compensated, we use the same TTGAN model to generate the synthetic data, which is trained only on the Original data set. The full plot for each of the four collected scenarios is shown in Figure 8. By using the Complete data set as a lower bound (best) performance, we are able to conclude when we can stop the collection of original data. Although this generally differs for each scenario, in all cases, the Compensated data set (blue line in Fig. 8) is able to achieve comparable (within −2.5%/–1% NRMSE for Force/Marker estimations) as the Complete data set (dashed red line in Fig. 8) with only 12.5% (1125 data samples) of the amount of original data in Complete. These results suggest that the synthetic data are able to consistently complement the lack of original data. More importantly, these results demonstrate the feasibility of the proposed generative model in producing synthetic data, which complements the lack of original data in soft robotics.

Addressing skewed data sets with CTTGAN

In Experiment B, we analyze the impact of a skewed data set and the performance of the proposed CTTGAN model based on tabulated results in Table 3. We note that the inference model trained with the Mixed-CTTGAN data set is able to perform comparatively with only a −0.01% Force NRMSE and −0.006% Marker NRMSE difference compared with the Complete data set. It also performs 1.009% and 0.156% better for Force and Marker estimation than the Skewed data set. In addition, Figure 9 illustrates the data distribution between the complete and synthetic data sets for both the CTTGAN and TTGAN models.

FIG. 9.

Data distribution of generated synthetic data where (a) illustrates the distribution of original data versus synthetic data generated with TTGAN and (b) illustrates synthetic data generated with CTTGAN, which has been conditioned to generate data for the Free Bending behavior.

Table 3.

Normalized Root-Mean-Square Error of Transformer TimeGAN Versus Conditional Transformer TimeGAN on Skewed Data Set

Data set	Free Bending		Tip Contact
Data set	Force	Markers	Force	Markers
Original	6.691 ± 0.166	4.397 ± 0.052	6.606 ± 0.161	4.340 ± 0.042
Skewed	7.548 ± 1.154	4.513 ± 0.123	7.751 ± 0.916	4.591 ± 0.329
TTGAN-Mixed	8.225 ± 1.316	4.519 ± 0.174	7.033 ± 0.384	4.470 ± 0.046
CTTGAN-Mixed	6.712 ± 0.198	4.409 ± 0.046	6.605 ± 0.158	4.382 ± 0.021

The inference model is independently trained on four data sets where each data set only contains 25% of the removed behavior (either Free Bending or Tip Contact), except for the Complete data set. Results are evaluated for the Overall scenario and have been averaged over five independent runs.

CTTGAN, conditional TTGAN.

From Figure 9a, we note that the model generates data randomly within the data distributions of all the experimental behaviors, whereas Figure 9b shows that we are able to control generation of data to a specific data distribution, in this case, the Free Bending behavior (i.e., Oscillatory Free Bending and Random Free Bending). In this study, we conclude that CTTGAN demonstrates a better attention to the target behavior (i.e., Free Bending). This exemplifies the notion that that CTTGAN is beneficial in behaviors where data recollection is not an option when the data set is skewed.

Limitations and future works

An error analysis is carried out by examining the prediction error plots of the inference model trained on the Synthetic and Half data sets, as shown in Figure 10. Although the errors are generally small, we note that negative performance of the synthetic data is heavily driven by specific components of the soft robot motion (i.e., when it is fully actuated in Tip Contact). This is to be expected as TTGAN aims to learn and generalize from the original data distribution. This results in the model generating data close to the mean at a high likelihood, and hence, extreme points such as full actuation are less likely to be generated. In future works, we look to promote the generation of more variety by encouraging the model to generate more out-of-distribution samples.

FIG. 10.

Prediction error plot over time. Extreme points (models weakest predictions) are circled and labeled based on the extent of the PSG actuation at that time.

In addition, we could exploit more efficient Transformer architectures such as sparse Transformers⁷⁷ to improve the efficiency of the generative model. In regard to experiments carried out, we planned to apply our framework on a more complex multisegment soft robot composed of a series of identical PSC bodies, where each of the PneuNet channels are actuated independently. Our current pneumatic control setup allowed us to concurrently control the pressures of not more than three PneuNet channels. In addition, we could explore 3D printing soft strain sensor grid⁷⁸ directly into the PSC, as an alternative to the current laborious soft sensor fabrication process in Supplementary Section S.2.1. On top of that, the current load cell designated for point load measurements has prohibited us from modeling contact force distribution applied on the PSF. Inspired by the state-of-the-art physic-informed neural networks⁷⁹ and sim-to-real transfer learning,³⁴ there are new opportunities to model the contact force distribution via hybrid models that leverage both the explainability of rigorous analytical or FEM methods and the flexibility of deep learning frameworks.

Conclusions

This study is the first to propose a data-driven framework to capture the complex dynamics of soft robots aided by synthetic data. In a multimodal sensing task, the proposed TTGAN model is able to successfully generate high-quality synthetic data that can be combined with just half of an original sequence to train a data-driven inference model and obtain performance comparable to using the entire original sequence. This is impactful as it implies that the process of costly data collection can be replaced by gathering a smaller set of data and compensating with synthetic data. In addition, by utilizing CTTGAN, we are able to control the distribution of the generated synthetic data to focus on a specific behavior of a soft robot and also aids in the case of skewed data sets. The outcomes of this research make an important contribution toward enabling complex and data-driven based soft robot modeling, without relying on huge amounts of data.

Footnotes

Acknowledgments

The authors acknowledge the resources provided by the advanced computing platform at Monash University Malaysia to train the neural network models.

Authors' Contributions

S.S. developed the methodology, performed the experiments, and prepared the article. Z.Y.D. and J.Y.L. developed the system. V.M.B. and C.P.T. supervised the project. S.G.N. conceived, supervised, and funded the project. All authors read and provided feedback for the article draft and approved the final article.

Data and Materials Availability

All code and data needed to verify the results in this article are uploaded to the GitHub repository ().

Author Disclosure Statement

No competing financial interests exist.

Funding Information

No funding was received for this article.

Supplementary Material

References

Kim

, Laschi

, Trimmer

. Soft robotics: A bioinspired evolution in robotics. Trends Biotechnol, 2013; 31(5):287–294.

Laschi

, Mazzolai

, Cianchetti

. Soft robotics: Technologies and systems pushing the boundaries of robot abilities. Sci Robot, 2016; 1(1):eaah3690.

Nurzaman

, Iida

, Laschi

, Ishiguro

, Wood

. Soft robotics [tc spotlight]. IEEE Robot Autom Mag, 2013; 20(3):24–95.

Laschi

, Cianchetti

. Soft robotics: New perspectives for robot bodyware and control. Front Bioeng Biotechnol, 2014; 2:3.

Webster

, Jones

. Design and kinematic modeling of constant curvature continuum robots: A review. Int J Robot Res, 2010; 29(13):1661–1683.

Godage

, Medrano-Cerda

, Branson

, et al. Modal kinematics for multisection continuum arms. Bioinspir Biomim, 2015; 10(3):035002.

Marchese

, Rus

. Design, kinematics, and control of a soft spatial fluidic elastomer manipulator. Int J Robot Res, 2016; 35(7):840–869.

Chen

, Wang

, Galloway

, et al. Modal-based kinematics and contact detection of soft robots. Soft Robot, 2021; 8(3):298–309.

Connolly

, Walsh

, Bertoldi

. Automatic design of fiber-reinforced soft actuators for trajectory matching. Proc Natl Acad Sci USA, 2016; 114(1):51–56.

10.

Thuruthel

, Falotico

, Renda

, et al. Learning dynamic models for open loop predictive control of soft robotic manipulators. Bioinspir Biomim, 2017; 12(6):066003.

11.

Godage

, Medrano-Cerda

, Branson

, et al. Dynamics for variable length multisection continuum arms. Int J Robot Res, 2016; 35(6):695–722.

12.

Marchese

, Tedrake

, Rus

. Dynamics and trajectory optimization for a soft spatial fluidic elastomer manipulator. Int J Robot Res, 2016; 35(8):1000–1019.

13.

Shiva

, Sadati

, Noh

, et al. Elasticity versus hyperelasticity considerations in quasistatic modeling of a soft finger-like robotic appendage for real-time position and force estimation. Soft Robot, 2019; 6(2):228–249.

14.

Renda

, Giorelli

, Calisti

, et al. Dynamic model of a multibending soft robot arm driven by cables. IEEE Trans Robot, 2014; 30(5):1109–1122.

15.

Wang

, Hirai

. Soft gripper dynamics using a line-segment model with an optimization-based parameter identification method. IEEE Robot Autom Lett, 2017; 2(2):624–631.

16.

Wang

, Nurzaman

, Iida

. Soft-material robotics. Found Trends Robot, 2017; 5(3):191–259.

17.

Elgeneidy

, Lohse

, Jackson

. Bending angle prediction and control of soft pneumatic actuators with embedded flex sensors—A data-driven approach. Mechatronics, 2018; 50:234–247.

18.

Thuruthel

, Shih

, Laschi

, et al. Soft robot perception using embedded soft sensors and recurrent neural networks. Sci Robot, 2019; 4(26):eaav1488.

19.

Wang

, Wang

, Du

, et al. Real-time soft body 3D proprioception via deep vision-based sensing. IEEE Robot Autom Lett, 2020; 5(2):3382–3389.

20.

Loo

, Ding

, Baskaran

, et al. Robust multimodal indirect sensing for soft robots via neural network-aided filter-based estimation. Soft Robot, 2022; 9(3):591–612.

21.

Kim

, Kim

, et al. Review of machine learning methods in soft robotics. PLoS One, 2021; 16(2):e0246102.

22.

Chin

, Hellebrekers

, Majidi

. Machine learning for soft robotic sensing and control. Adv Intell Syst, 2020; 2(6):1900171.

23.

Cianchetti

, Calisti

, Margheri

, et al. Bioinspired locomotion and grasping in water: The soft eight-arm OCTOPUS robot. Bioinspir Biomim, 2015; 10(3):035003.

24.

Marchese

, Onal

, Rus

. Autonomous soft robotic fish capable of escape maneuvers using fluidic elastomer actuators. Soft Robot, 2014; 1(1):75–87.

25.

Khan

, Liquan

, Gang

, et al. Concept design of the underwater manned seabed walking robot. J Marine Sci Eng, 2019; 7(10):366.

26.

Cios

, Moore

. Uniqueness of medical data mining. Artif Intell Med, 2002; 26(1–2):1–24.

27.

Schramm

, Sintov

, Boularias

. Learning to Transfer Dynamic Models of Underactuated Soft Robotic Hands. In: 2020. IEEE International Conference on Robotics and Automation (ICRA); 2010; pp. 4579–4585.

28.

Fang

, Wang

, et al. Vision-based online learning kinematic control for soft robots using Local Gaussian Process Regression. IEEE Robot Autom Lett, 2019; 4(2):1194–1201.

29.

Runge

, Wiese

, Raatz

. FEM-Based Training of Artificial Neural Networks for Modular Soft Robots. In: 2017 IEEE International Conference on Robotics and Biomimetics (ROBIO), Macau; 2017; pp. 385–392.

30.

Massari

, Schena

, Massaroni

, et al. A machine-learning-based approach to solve both contact location and force in soft material tactile sensors. Soft Robot, 2020; 7(4):409–420.

31.

Sui

, Shang

, Li

. Transfer of robot perception module with adversarial learning. IEEE Access, 2019; 7:79726–79736.

32.

Chen

, Hu

, Jin

, et al. Understanding Domain Randomization for Sim-to-Real Transfer. In: 2022. 10th International Conference on Learning Representations (ICLR), Virtual.

33.

Kriegman

, Nasab

, Shah

, et al. Scalable Sim-to-Real Transfer of Soft Robot Designs. In: 2020 3rd IEEE International Conference on Soft Robotics (RoboSoft), Virtual; 2020; pp. 359–366.

34.

Park

, Cho

, Park

, et al. Sim-to-real transfer learning approach for tracking multi-DOF ankle motions using soft strain sensors. IEEE Robot Autom Lett, 2020; 5(2):3525–3532.

35.

Wiese

, Runge-Borchert

, Cao

, et al. Transfer Learning for Accurate Modeling and Control of Soft Actuators. In: 2021 IEEE 4th International Conference on Soft Robotics (RoboSoft), Connecticut, USA; 2021; pp. 51–57.

36.

Donat

, Lilge

, Burgner-Kahrs

, et al. Estimating tip contact forces for concentric tube continuum robots based on backbone deflection. IEEE Trans Med Robot Bionics, 2020; 2(4):619–630.

37.

Fang

, Tian

, Yang

, et al. Efficient Jacobian-based inverse kinematics with sim-to-real transfer of soft robots by learning. IEEE/ASME Trans Mechatron, 2022; 27(6);5926–5306.

38.

Dong

, Zhou

, Qiu

, et al. Robotic manipulations of cylinders and ellipsoids by ellipse detection with domain randomization. IEEE/ASME Trans Mechatron, 2022; 28(1):302–313.

39.

Tremblay

, To

, Sundaralingam

, et al. Deep Object Pose Estimation for Semantic Robotic Grasping of Household Objects. In: 2018 Conference on Robot Learning (CORL), Zurich, Switzerland; 2018; pp. 306–316.

40.

Tobin

, Fong

, Ray

, et al. Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World. In: 2017 IEEE/RSJ International Conference On Intelligent Robots And Systems (IROS), Vancouver, Canada; 2017; pp. 23–30.

41.

, Liu

, Spielberg

, et al. Chainqueen: A Real-Time Differentiable Physical Simulator for Soft Robotics. In: 2019 International Conference on Robotics and Automation (ICRA); 2019; pp. 6265–6271.

42.

Zhang

, Zhang

, Ma

, et al. Sim2Real for soft robotic fish via differentiable simulation. In: 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Kyoto, Japan; 2022; pp. 12598–12605.

43.

Lipson

Challenges and opportunities for design, simulation, and fabrication of soft robots. Soft Robot, 2014; 1(1):21–27.

44.

Schegg

, Duriez

. Review on generic methods for mechanical modeling, simulation and control of soft robots. PLoS One, 2022; 17(1):e0251059

45.

Fields

, Hsieh

, Chenou

. Mitigating Drift in Time Series Data with Noise Augmentation. In: 2019 International Conference on Computational Science and Computational Intelligence (CSCI), Las Vegas, USA; 2019; pp. 227–230.

46.

, Pfister

, Pichler

, et al. Data Augmentation of Wearable Sensor Data for Parkinsons Disease Monitoring Using Convolutional Neural Networks. In: Proceedings of the 19th ACM International Conference on Multimodal Interaction, Glasgow, Scotland; 2017; pp. 216–220.

47.

Le Guennec

, Malinowski

, Tavenard

Data Augmentation for Time Series Classification Using Convolutional Neural Networks. In: ECML/PKDD Workshop on Advanced Analytics and Learning on Temporal Data, Riva del Garda, Italy; 2016.

48.

, Kong

, Qiao

, Sukkarieh

. Data augmentation for deep learning based semantic segmentation and crop-weed classification in agricultural robotics. Comput Electron Agric, 2021; 190:106418.

49.

Liu

, Li

. Visual servoing with deep learning and data augmentation for robotic manipulation. J Adv Comput Intell Intell Inform, 2020; 24:953–962.

50.

Wang

, Majewicz Fey

. Deep learning with convolutional neural network for objective skill evaluation in robot-assisted surgery. Int J Comput Assist Radiol Surg, 2018; 13:1959–1970.

51.

Lakomkin

, Zamani

, Weber

, et al. On the Robustness of Speech Emotion Recognition for Human-Robot Interaction with Deep Neural Networks. In: 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Madrid, Spain; 2018; pp. 854–860.

52.

Forestier

, Petitjean

, Dau

, et al. Generating Synthetic Time Series to Augment Sparse Datasets. In: 2017 IEEE international conference on data mining (ICDM), Los Angeles, USA; 2017; pp. 865–870.

53.

Iwana

, Uchida

. An empirical survey of data augmentation for time series classification with neural networks. PLos One, 2021; 16(7):e0254841.

54.

Goodfellow

, Pouget-Abadie

, Mirza

, et al. Generative Adversarial Nets. In: NeurIPS, Quebec, Canada; 2014; pp. 2672–2680.

55.

Rao

, Harris

, Irpan

, et al. Rl-cyclegan: Reinforcement Learning Aware Simulation-to-Real. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, USA; 2020; pp. 11157–11166.

56.

Bousmalis

, Irpan

, Wohlhart

, et al. Using Simulation and Domain Adaptation to Improve Efficiency of Deep Robotic Grasping. In: 2018 IEEE International Conference on Robotics and Automation (ICRA), Brisbane, Australia; 2018; pp. 4243–4250.

57.

, Rao

, Xu

, et al. Retinagan: An Object-Aware Approach to Sim-to-Real Transfer. In: 2021 IEEE International Conference on Robotics and Automation (ICRA), Xi'an, China; 2021; pp. 10920–10926.

58.

Shin

, Tenenholtz

, Rogers

, et al. Medical Image Synthesis for Data Augmentation and Anonymization Using Generative Adversarial Networks. In: International Workshop on Simulation and Synthesis in Medical Imaging. Springer: Granada, Spain; 2018; pp. 1–11.

59.

Frid-Adar

, Diamant

, Klang

, et al. GAN-based synthetic medical image augmentation for increased CNN performance in liver lesion classification. Neurocomputing, 2018; 321:321–331.

60.

James

, Harbron

, Branson

, et al. Synthetic data use: Exploring use cases to optimise data utility. Discov Artif Intell, 2021; 1(1):1–13.

61.

Tolley

, Shepherd

, Mosadegh

, et al. A resilient, untethered soft robot. Soft Robot, 2014; 1(3):213–223.

62.

Alici

, Canty

, Mutlu

, et al. Modeling and experimental evaluation of bending behavior of soft pneumatic actuators made of discrete actuation chambers. Soft Robot, 2018; 5(1): 24–35.

63.

Gao

, Liu

, Shen

, et al. Laser-profiled continuum robot with integrated tension sensing for simultaneous shape and tip force estimation. Soft Robot, 2020; 7(4):421–443.

64.

Mogren

. C-RNN-GAN: Continuous Recurrent Neural Networks with Adversarial Training. 2016; arXiv preprint arXiv:1611.09904.

65.

Esteban

, Hyland

, Rätsch

. Real-Valued (Medical) Time Series Generation with Recurrent Conditional Gans. 2017; arXiv preprint arXiv:1706.02633.

66.

Yoon

, Jarrett

, Schaar v. dM. Time-Series Generative Adversarial Networks. In: NeurIPS, Vancouver, Canada; 2019; pp. 5508–5518.

67.

Vaswani

, Shazeer

, Parmar

, et al. Attention is All You Need. In: Advances in Neural Information Processing Systems, Long Beach, USA; 2017; pp. 5998–6008.

68.

Ilievski

, Mazzeo

, Shepherd

, et al. Soft robotics for chemists. Angew Chem, 2011; 123(8):1930–1935.

69.

Alici

Softer is harder: What differentiates soft robotics from hard robotics?. MRS Adv, 2018; 3(28):1557–1568.

70.

, Nurzaman

, Culha

, Iida

. Soft robotics education. Soft Robot, 2014; 1(3):202–212.

71.

Umedachi

, Ito

, Ishiguro

. Soft-bodied amoeba-inspired robot that switches between qualitatively different behaviors with decentralized stiffness control. Adap Behav, 2015; 23(2):97–108.

72.

Dai

, Fidler

, Urtasun

, et al. Towards Diverse and Natural Image Descriptions Via a Conditional Gan. In: Proceedings of the IEEE International Conference on Computer Vision; 2017; pp. 2970–2979.

73.

Memarian

, Gorbet

, Kulić

. Control of Soft Pneumatic Finger-Like Actuators for Affective Motion Generation. In: IROS, Hamburg, Germany; 2015; pp. 1691–1697.

74.

Hochreiter

, Schmidhuber

. Long short-term memory. Neural Comput, 1997; 9(8):1735–1780.

75.

Iwana

, Uchida

. Time Series Data Augmentation for Neural Networks by Time Warping with a Discriminative Teacher. In: 2020 25th International Conference on Pattern Recognition (ICPR), Virtual, 2021; pp. 3558–3565.

76.

Kamycki

, Kapuscinski

, Oszust

. Data augmentation with suboptimal warping for time-series classification. Sensors, 2019; 20(1):98.

77.

Zhou

, Zhang

, Peng

, et al. Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting. In: Proceedings of AAAI, Vancouver, Canada; 2021.

78.

Muth

, Vogt

, Truby

, et al. Embedded 3D printing of strain sensors within highly stretchable elastomers. Adv Mater, 2014; 26(36):6307–6312.

79.

Sun

, Akashi

, Kuniyoshi

, et al. Physics-informed recurrent neural networks for soft pneumatic actuators. IEEE Robot Autom Lett, 2022; 7(3):6862–6869.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

22.69 MB

13.34 MB

0.00 MB