Perceptually Inspired C 0 -Continuity Haptic Shape Display with Trichamber Soft Actuators

Abstract

Shape display devices composed of actuation pixels enable dynamic rendering of surface morphological features, which have important roles in virtual reality and metaverse applications. The traditional pin-array solution produces sidestep-like structures between neighboring pins and normally relies on high-density pins to obtain curved surfaces. It remains a challenge to achieve continuous curved surfaces using a small number of actuated units. To address the challenge, we resort to the concept of surface continuity in computational geometry and develop a C⁰-continuity shape display device with trichamber fiber-reinforced soft actuators. Each trichamber unit produces three-dimensional (3D) deformation consisting of elongation, pitch, and yaw rotation, thus ensuring rendered surface continuity using low-resolution actuation units. Inspired by human tactile discrimination threshold on height and angle gradients between adjacent units, we proposed the mathematical criteria of C⁰-continuity shape display and compared the maximal number of distinguishable shapes using the proposed device in comparison with typical pin-array. We then established a shape control model considering the nonlinearity of soft materials to characterize and control the soft device to display C⁰-continuity shapes. Experimental results showed that the proposed device with nine trichamber units could render typical sets of distinguishable C⁰-continuity shape sequence changes. We envision that the concept of C⁰-continuity shape display with 3D deformation capability could improve the fidelity of the rendered shapes in many metaverse scenarios such as touching human organs in medical palpation simulations.

Introduction

Haptic devices are used in the metaverse to provide the user with touch feedback, such as texture, shape, softness, temperature, roughness, and other features to enhance immersion. Shape displays, in particular, are essential in delivering haptic information to humans by reproducing and simulating shapes and surfaces. A haptic shape display is an encountered-type haptic device that allows users to interact with their bare hands.^1,2 Shape displays usually need to present a sizeable continuous area and multiple possible shapes covering the entire hand.

Such devices would greatly expand the channels of haptic human–machine interaction and further increase the rich sense of realism in virtual reality (VR) and augmented reality. The continuous shape display device may be promising for bare hand interaction applications, including medical training, fast prototype design, e-commerce, and virtual games. In medical palpation applications, the device can restore the shape sensation of touching different organs or tissues to train doctors' palpation ability. Doctors can touch and feel the difference between normal and abnormal organs, such as liver enlargement. The use of haptic shape display is a solution to the problem of lack of patients in medical palpation.

Realistic shape reproduction that requires continuous accurate shape display needs to form changeable surface using limited portfolio change of actuation units. The most typical type of shape display device is through the pin-array approach.^3,4 It means that dozens or even hundreds of pins can move vertically on the plane. The shape can be represented by controlling the vertical height of the pins.^3,5–9 Siu et al. designed a pin-array^4,10 and equipped with a mobile device that can move in real time according to the hand position to achieve a wide range of shape display. A shape display device consisting of the auxetic structure has been developed to display a surface of specific curvature.¹¹ In addition, some pin-arrays used new types of actuators, such as electrorheological and magnetorheological, to achieve millimeter level and high spatial resolution.^12–17

Okamura and colleagues adopted particle jamming cells as the flexible tactile array to control each part of the interface to achieve 2.5-dimensional shape display.^18–22 On this basis, they produced a three-dimensional (3D) shape display device²³ and designed an automatic design algorithm to display an ideal range of shapes with a small number of internal actuators. In addition, the simulation of the surface shape can also be realized through the hinge structure.^24,25

In computer graphics, C⁰-continuity surface is a surface consisting of multiple surface slices, and the zero-order derivatives between surface slices are equal (position continuous) at the junction.²⁶ The objects we encounter in daily life or in medical palpation, such as balls, saddles, and other objects, typically satisfy the surface C⁰-continuity. Using 1 degree of freedom (DOF) actuator array suffers from sharp edges due to the discrete rigid element arrangement, which does not satisfy the display of C⁰-continuous shapes; the height difference between units reduces its ability to render a large continuous surface. The steps generated by the one-dimensional (1D) unit need to be smaller than the human discrimination threshold to eliminate the discontinuity. It is difficult to manufacture an actuator unit with a width that meets this fingertip two-point discrimination threshold of <2.5 mm.²⁷ Therefore, we increased the DOF of the actuator and ensured that each unit's edge was collinear with its neighboring unit's edge (Fig. 1) to display C⁰-continuity shapes.

FIG. 1.

The shape display device based on the modular trichamber fiber-reinforced soft actuators can realize the representation of multiple categories of C⁰-continuity shapes. Trichamber fiber-reinforced soft actuation units capable of generating three-dimensional deformation consisting of elongation, pitch, and yaw rotation were proposed and fabricated. The fast and low-cost manufacturing of units and their connections can be accomplished through modular combination. DOF, degree of freedom.

The maximum number of distinguishable shapes that the devices can render varies with the form of actuation units and the connection between units.^3,4,11,18,24 It is an essential indicator of the shape display performance in generating shapes. The ability of a structure to form shapes was quantitatively characterized by DOF, and two measures were presented.²⁸ The two measures were used to compare and analyze several typical shape display structures. Considering the complexity of the human perception threshold,^29,30 the use of discrete quantities to characterize the maximal number of distinguishable haptic shapes is still a problem. If the height or angle gradient change of neighboring units is less than the human body's perception threshold, such shape change will not be perceived by human touch. Therefore, we proposed a general classification of shapes based on the discretization of continuous workspace with human perception threshold.

Traditionally, the shape control model focuses on the rigid and linear elastic materials shape display device.^3,4,21,23 The shape display control model is used to achieve the mapping of the input parameter and the target shape. The mass-spring model was used to model the input parameters and output shape.²³ For the shape rendering device, which is entirely composed of soft materials, it is still a challenge to build a mapping model due to the nonlinear effect of the material. The infinite DOF and compliant nature of soft actuators make it difficult to establish an accurate model that can satisfy the explicit control of a predefined shape, unlike the limited and discrete rendered shapes of rigid devices. Therefore, we proposed a C⁰-continuity shape control model to solve these issues.

In this study, we developed a C⁰-continuity shape display device with trichamber fiber-reinforced soft actuators. The main contributions of this study are as follows: 1.

We introduced the concept of C⁰-continuity from computational geometry into shape display to achieve continuous surface by using a trichamber soft actuator. Compared with traditional shape display using array of 1D translating pins, our proposed unit can produce 3D deformation and thus can eliminate sidestep-like structures between neighboring units and significantly improve the rendered surface continuity.

We proposed a model for quantifying the criteria of C⁰-continuity based on the human tactile discrimination threshold on height and angle gradient between adjacent units. Based on the proposed model, the maximal number of distinguishable shapes was calculated and compared with typical pin-array, and the effect of the rigidity of connection joint was analyzed. We found that it is more effective to increase the DOF of each unit than to increase the number of units.

We established an analytical model for shape control, which quantifies the mapping function of the design parameters among three spaces, that is, actuation space, joint space, and operation space of the soft trichamber unit. Based on the shape control model, the shape display experiments were conducted, and the results showed that the proposed C⁰-continuity shape display device can render various complex shapes.

Principle, Shape Number, and Design of Shape Display

To achieve a realistic simulation of 3D shapes, a discrete mesh can be applied to the object's surface. In addition to ensuring its shape display properties, the manufacturing cost and time needs to be controlled.²⁸ The 3D shape after meshing division can be regarded as multiple independent units.

Shape discretization principle

For each unit, its shape is particularly critical for the design of shape display device. With the different choices of the unit's shape, the 3D shape simulation algorithms are different, and the actuation methods are also different. The meshing division is generally divided into triangular mesh and quadrilateral mesh. Triangular mesh as the object of geometric processing is more in-depth studied, which is easy to implement and can adapt to complex shapes. Combined with some control methods, it can generate a high-quality mesh.^31–33 Quadrilateral mesh research started later, but the quadrilateral mesh is more regular.³⁴ Most of them are aligned to the direction of principal curvature, which can realize regular discretization for natural shapes.³⁵ We mainly focus on the human perception of shape, so the quadrilateral mesh is chosen for meshing division, and therefore, the units are chosen to be the same square units.

For the simulation of 3D shapes, if only the position of each unit is controlled, the angle between the units is a fixed angle. However, most 3D shapes always have a large difference in the angle between the quadrilateral units of the surface after discretization.³⁶ Controlling only the position of the units and ignoring the angle between the units will lead to coarse shape display. Therefore, the units' 6 DOF need to be considered. The coordinate system is defined by the z-axis perpendicular to the unit. For the surface formed by joining several units, the unit center's x- and y-axis coordinates are fixed, and the unit cannot rotate along the z-axis. Therefore, 3 DOF of motion are required for each unit, that is, the movement along the z-axis and rotation along the x- and y-axes.

When the unit has 3 DOF, if there is no connection between the units, the human hand can only feel the discontinuous shape. Different connections of the units will also lead to differences in the shape of the person's touch. The connection joints can be divided into three types, as shown in Figure 2b, namely units with no joint, units with rigid joint, and units with flexible joint. For the units with no joint, the unit can move freely, and there is no interference between the units. For the units with rigid joint, only relative rotation can occur between the units. For the units with flexible joint, depending on the flexibility of the flexible joint, the two units can not only have a relative rotation but also have a relative movement of different degrees.

FIG. 2.

Modeling the number of distinguishable shapes. (a) For the unit with 3 DOF, the output level of each DOF based on height and angle gradient discrimination thresholds between adjacent units of the human body. (b) Design principles for shape display devices consisting of rigid units and different types of connection joints. (c) Unrealized singularity shapes are produced when units with rigid joint.

Maximal number of distinguishable shapes

For the shape display device, the number of distinguishable shapes is an important indicator to evaluate a shape display device. It indicates the actual application scope and scenario of the device. In specific VR scenarios, a shape display device is expected to render a greater number of shapes, which avoids the hassle of frequently changing shape display device to simulate different types of objects.

The number of distinguishable shapes depends on the DOF of the device, as well as to the workspace of the device and the human perception threshold. The workspace describes the set of locations that can be reached with the end of a device. The number of distinguishable shapes increases when the workspace is larger. However, not all shape changes in the workspace can be perceived by manual touch. Due to the human perception threshold, only shapes with a certain range of changes can be perceived. Therefore, we treat the continuous workspace as discrete robot motion poses to reflect the number of human perceptible shapes.

The perception of the shape of objects beyond the finger scale is influenced by various factors, including the concavity of the shape, the direction of movement along the surface, the position of the contact point on the hand, and the shape's own features.²⁹ We simplify the perception of shapes by human fingers to be mainly related to human kinesthetic perception. For C⁰-continuity shape, the differences of shape are mainly caused by the different heights or angles between units. According to the experiment,¹¹ the height perception threshold of the human fingers for shape is <20 mm, and the device width in this article is 220 mm, so it can be calculated that the angle perception threshold for shape is <10.3°. Therefore, the shape can be perceived as different when the unit moves 20 mm along the z-axis or rotates 10.3° around the x- and y-axes (Fig. 2a), which can be expressed as $\{\begin{matrix} θ_{b e n d} > θ_{t h r e s h o l d} \\ h_{e l o n g a t e} > h_{t h r e s h o l d} \end{matrix}$ (1)

$θ_{b e n d}$ and $h_{e l o n g a t e}$ represent the actual deformation of the unit. $θ_{t h r e s h o l d}$ and $h_{t h r e s h o l d}$ represent the haptic shape perception threshold of the human fingers. If two equations are satisfied as above, poses 1, 2 can be considered as different shapes. The actual deformation is calculated as follows: $\{\begin{matrix} θ_{b e n d} = |arccos (\frac{\vec{n_{1}} \cdot \vec{n_{2}}}{|\vec{n_{1}}| |\vec{n_{2}}|})| \\ h_{e l o n g a t e} = |z_{1} - z_{2}| \end{matrix}$ (2)

$\vec{n_{1}}$ , $\vec{n_{2}}$ represent the normal vectors of a unit in two different poses; z₁, z₂ represent the height difference between the centroids of the unit in two different poses. The elastic deformation of the unit's own shape is small compared with the joint and can be neglected; the unit is treated as a nonstretchable rigid unit. We have modeled the distinguishable shapes under three different connection joints, and the detailed models are shown in Supplementary Appendix SA1.

The distinguishable shapes are compared in Table 1. For the units with rigid joint, it can be easy to present C⁰-continuity shapes. This also leads to the limited shapes. Singularity shapes that will occur is that one unit can take any position, while the other element can only take part of the limited position. Some singularity shapes are shown in Figure 2c. For the units with no joint, there are more distinguishable shapes without singularity loss. However, some discontinuous shapes will be generated. Users will feel the edges and corners of the unit. For the units with flexible joint, the deformation between the units is continuous. Therefore, we chose the flexible joint as the connection structure for the subsequent design.

Table 1.

Maximal Number of Distinguishable Shapes in Different Cases

Number of units	Graphic representation	1 DOF	3 DOF
Number of units	Graphic representation	No joint	No joint	Rigid joint	Flexible joint
1		$2^{1}$	$1 8^{1}$	18	$1 8^{1}$
2		$2^{2}$	$1 8^{2}$	54	$1 8^{2}$
4		$2^{4}$	$1 8^{4}$	108	$1 8^{4}$
9		$2^{9}$	$1 8^{9}$	324	$1 8^{9}$
n	…	$2^{n}$	$1 8^{n}$	4 $\times 3^{\sqrt{n} + 1}$	$1 8^{n}$

DOF, degree of freedom.

For the array with 1 DOF units connected by flexible joints, if the number of units is n and each unit can produce a distinguishable stroke output level of 2, it can display $2^{n}$ distinguishable shapes according to Equation A5 in Supplementary Appendix SA1. While the unit's DOF becomes 3, and the stroke output levels of each DOF are 2, 3, and 3, as shown in Figure 2a, the number of distinguishable shapes is ${(2 \times 3 \times 3)}^{n}$ . Therefore, by increasing the DOF of each unit from 1 to 3, the number of distinguishable shapes increases $1 8^{n} ∕ 2^{n} = 9^{n}$ times.

Contrastively, if the unit's DOF is 1 and the number of units is tripled to 3n, the number of distinguishable shapes is $2^{3 n}$ . Therefore, by increasing the number of each unit from n to 3n, the number of distinguishable shapes increases $2^{3 n} ∕ 2^{n} = 4^{n}$ times, which is significantly smaller than the benefit of increasing the unit's DOF ( $i . e ., 9^{n}$ ).

Therefore, to increasing the distinguishable shape of the shape display device, it is more effective to increase the DOF of each unit to 3 than increasing the number of units to 3 times. The underlying reason is that for the kinematic performance of the device, it is easier to realize the curved shape changes through angular changes rather than height changes.

Design and components of shape display

Our actuator needs to have 3 DOF and requires to be as small as possible, with a suitable output motion performance and a safer human–machine interaction. Therefore, we used a trichamber soft actuator^37–41 to control the motion of each unit. Moreover, to ensure that the unit can simulate different stiffnesses to satisfy, for example, the simulation of diverse soft and hard organs in medical palpation,⁴² a variable stiffness layer jamming was placed at the tip of the actuator. The overall structure of the unit is shown in Figure 3.

FIG. 3.

Components of shape display device. (a) The composition of a single unit includes the upper layer jamming and the lower trichamber fiber-reinforced soft actuator. The actuator consists of an outer wall and three internal chambers, and the overall is made of silicone rubber material. The double helical fibers were oriented at angles of 10° and −10° with respect to the actuator cross section. (b) The physical prototype of a single unit. The units are the same unit with modular design, which ensures the convenience of subsequent processing and manufacturing. (c) Interaction between human hands and the physical shape display device composed of nine units. (d) Distribution of nine units and chambers of the device. The surrounding eight actuators were initially tilted. (e) Planar state of the device after prepressurization and the corresponding chamber air pressure. The planar state is based on the modified shape control model to obtain target pressure, adjusted for actual installation errors.

The three chambers are symmetrically distributed in the cross section. Double helical fibers were embedded inside each chamber of the soft actuator. The fibers have a high elastic modulus and can be regarded as nonstretchable compared with silicone rubber, which constrained the expansion of the actuator along the radial direction. When the air pressure in the three chambers changes, the actuator will bend along the horizontal plane or elongate along the axial direction. The layer jamming on the top of the unit ensures that sufficient support is provided to maintain the shape without changing when the person presses it. The structure and performance of the layer jamming is referenced from the article^20,43–45 and is designed according to the human interaction force requirements.⁴⁶ The structural parameters of the device are shown in Table 2.

Table 2.

Structural Parameters of the Device

L/mm	r₁/mm	r₂/mm	r₃/mm	r₄/mm	r₅/mm
60	1.75	2.5	4.5	6.5	7.5

d₁/mm	d₂/mm	x₁/mm	b/mm	h/mm
0.5	1	1	30	2.6

For the connection between units, the top of the unit was connected by a silicone rubber material whose elastic modulus is smaller than the unit itself to achieve a flexible hinge effect. The top of the unit was fixed to the base through a ball kinematic pair to ensure that the curvature of the actuator is as close as possible during the bending process to ensure the subsequent modeling and control.

The device covering a person's palm with nine trichamber units was designed and manufactured, and the distribution of the chambers is shown in Figure 3d. For the surrounding eight actuators, two of the three chambers of the actuator are distributed away from the center of the device. The outer two chambers allow the outside of the unit to be lifted more than the inside, where there is one chamber. The ability of the unit to render a concave shape is greater than a convex shape. Therefore, the actuators were designed to be adjacent to each other at the top of the device. The surrounding eight actuators were initially tilted to balance the device's ability to render convex and concave shapes. The device was initially planar by presetting the air pressure, and the air pressure in each chamber is shown in Figure 3e.

Actuator and Shape Control Modeling

Actuator analytical modeling

We developed an analytical model for the trichamber fiber-reinforced soft actuators.^47–50 By establishing the relationship between input pressure, bending angle, output position, and force, as shown in Figure 4a, the control of the actuator can be achieved. Considering the hyperelastic material property of silicone rubber and the fiber reinforcement of the actuator, the Neo-Hookean (NH) method⁵¹ was applied for the modeling analysis.

FIG. 4.

Analytical model of actuator and layer jamming and shape control model of the device. (a) Actuation space, joint space, and operation space mapping relationship of the unit. The actuation space is the input air pressure and drive torque; the joint space is the length of the actuator chamber; the operation space is the end position of the actuator. (b) The photos of the actuator at different bending angles are superimposed and the structural parameters of the trichamber actuator under external force. (c) The structural parameters of the layer jamming. (d) The analysis of the workspace about nine units soft array shape display device.

The trichamber actuator's structural parameters are shown in Figure 4b. When the air is inflated into a chamber, it is at the position of the neutral layer, where the axial length remains constant. The position is the farthest point from the chamber's center in the cross section. There is a balance between the driving torque $M_{d r i v e}$ , the elastic restoring torque $M_{r e s t o r e}$ , and the external torque $M_{o u t}$ at this position as $M_{d r i v e} = M_{r e s t o r e} + M_{o u t}$ (3)

where the driving torque is generated by the input air pressure. Due to the complex shape of the chamber, the chamber is first supplemented as a sector ring passing through the center of the circle. Then calculate the torque generated by the sector ring and subtract the torque generated by the supplementary part, as $\begin{matrix} M_{d r i v e} = p \int_{θ_{1} + \frac{π}{6}}^{π - (θ_{1} + \frac{π}{6})} \int_{r_{2}}^{r_{3}} r (r_{5} + r sin θ) d θ d r \\ - 2 p \int_{θ_{1} + \frac{π}{6}}^{θ_{2} + \frac{π}{6}} \int_{r_{2}}^{\frac{d_{2}}{sin (θ - π ∕ 6)}} r (r_{5} + r sin θ) d θ d r \end{matrix}$ (4)

The restoring torque is produced by the deformation of the soft silicone rubber material. Since the actuator is made of a variety of soft materials and has a complex structure, the detailed solution formulas for each torque are given in Supplementary Appendix SA1.

The relationship between the axial strain $λ_{1}$ and the bending angle $δ$ as $λ_{1} = 1 + \frac{δ (r_{5} + r sin θ)}{L}$ (5)

When air is inflated into two or three chambers, we assume that there exists a point in the cross section, along which the axial length elongation is minimal. Its position is the vertical direction of the resultant torque of the trichamber, and it is located on the outermost side of the actuator section. Therefore, for the simultaneous ventilation of the three chambers, the calculation was divided into two parts. First, we calculated the elongation of the actuator when the three chambers were input with the same minimum air pressure (air pressure among the three chambers) and used it as the new actuator length (L). Second, we calculated the driving and restoring torque under the three-chamber air pressure. Finally, the relationship can be obtained as $(R, φ, δ) = G (P_{1}, P_{2}, P_{3}, M_{o u t})$ (6)

Flexible joint modeling

The connecting structure was made of silicone materials. Here, the connecting structure is a flexible joint, and its material is silicone Ecoflex 00–30. Its structural parameters are shown in Figure 4c. The NH method was used for analysis modeling.

For uniaxial tension of silicone rubber material, the three principal stretches ratios and stresses are as $λ_{1} = λ, λ_{2}^{2} = λ_{3}^{2} = \frac{1}{λ}$ (7)

σ_{1} = 2 C (λ_{1} - \frac{1}{λ_{1}^{2}}), σ_{2} = σ_{3} = 0

(8)

According to the above formula, the relationship between the output torque and the deformation of the flexible joint is as follows: $M_{o u t} = H (x) = F L = 2 b h C (\frac{x^{3} - x_{0}^{3}}{x_{0} x^{2}})$ (9)

where F is the tensile force of the flexible joint to the unit's edge, and the force's direction is parallel to the line connecting the edges of the two units. And x is the length of the flexible joint after stretching; b and h are the width and thickness of the flexible joint, respectively. The x₀ is the initial length of the flexible joint, and the L is the length of the actuator.

Shape control modeling

We constructed a shape control model for the multiunit soft array shape display device to solve the inverse problem, in which we get the input air pressure of each chamber when the tip position of each actuator is known.

Flexible joint extensions <2 mm is required to satisfy C⁰-continuity due to the discrimination threshold of two-point resolution of the human hand.²⁷ The kinematic workspace of the device with nine trichamber units for flexible joint extensions <2 mm, namely the continuous shape workspace, was analyzed in Figure 4d.

For the shape display device, the spatial coordinates of all the actuators and the relative position relationship of the actuator centers can be expressed as $S = (\begin{matrix} R \\ φ \\ δ \end{matrix}) = (\begin{matrix} R_{1}, R_{2}, \dots, R_{n} \\ φ_{1}, φ_{2}, \dots, φ_{n} \\ δ_{1}, δ_{2}, \dots, δ_{n} \end{matrix}), X = (X_{1}, X_{2}, \dots, X_{m})$ (10)

where m is the number of flexible joints. Substituting the above matrix into the equations for the elastic restoring torque and joint tensile force, the expressions for the flexible joint tensile torque $M_{o u t} = H (X)$ can be obtained. According to the balance equation of the external torque, elastic restoring torque, and drive torque of the actuator, the drive torque of the actuator can be solved. Substituting the above matrix into the equation for solving the drive torque of the actuator, the input air pressure of each chamber can be obtained as $P = (\begin{matrix} P_{1} \\ P_{2} \\ P_{3} \end{matrix}) = (\begin{matrix} p_{11}, p_{12}, \dots, p_{1 n} \\ p_{21}, p_{22}, \dots, p_{2 n} \\ p_{31}, p_{32}, \dots, p_{3 n} \end{matrix}) = G^{- 1} (S, M_{o u t})$ (11)

Experiment and Validation

Performance specification experiment

The elongation and bending angle of single-chamber, two-chamber, and three-chamber under different air pressure were measured, and the theoretical calculation values were calculated by the analytical model. The results are shown in Figure 5a–c.

FIG. 5.

The relationship between the input actuator pressure, bending angle, elongation, bending force, and elongation force obtained from the analytical model and experimental data. The experimental data were collected through a unit's measurement, which was repeated three times in each experiment. A force sensor (ATI-Nano17, USA) was used to measure the bending force in the radial direction and the elongation force in the axial direction. (a–c) The relationship between input pressure and bending angle or elongation during single-chamber, two-chamber, three-chamber pressurization. (d–f) The relationship between input pressure and bending force or elongation force during single-chamber, two-chamber, three-chamber pressurization. (g–i) The response of air pressure and actuator end to step reference pressure. (j–l) The repeatability testing results of the actuator. (m–o) The hysteresis of the actuator for different maximum pressure pressurization and depressurization.

The results show that the analytical model can capture the overall motion trend of the actuator, and the theoretical and experimental data can be well matched (the maximum bending angle error is 3.5°, and the maximum elongation error is 1.25 mm) in the range of 0–80 kPa. The actuator can bend over 60° and elongate over 20 mm at 125 kPa, which meets the current requirement for the perception threshold of the distinguishable shape.

For the analytical model, two factors could account for these errors. When the air pressure is <80 kPa, the material deformation is small and the gravitational force is the main influencing factor. The analytical model is greater than the experimental results. When the air pressure is 80–125 kPa, the main factors are the linearization of the NH model under large material deformation and the radial bulging effects. The nonlinear behavior of the material leads to a gradually increasing error.

When the air pressure is <80 kPa, the experiment and analytical model fit well. We use the analytical model for control. When the air pressure is 80–125 kPa, we use the least squares method to modify the material property parameter C to minimize the error between the analytical model and the experimental results.

The maximum output force at the top of the actuator was tested. The results are shown in Figure 5d–f. At the air pressure of 120 kPa, the bending force of the actuator can reach 0.30 N, and the elongation force can reach 4.41 N, which can satisfy the typical interaction requirements of human pressing the device and sliding on the device surface, such as touching the ball and sliding on the table. There are two main reasons for the error in the triple-chamber elongation force test. When the air pressure is <30 kPa, the contact between the actuator and the force sensor is not tight, and the actuator will still undergo a tiny displacement, and the restoring torque is not zero. When the air pressure is >100 kPa, the actuator is buckling as the buckling of a compression bar is under the action of the force, which leads to a decrease in the measured force.

Furthermore, the dynamic performance including response time, repeatability, and hysteresis of the actuator during single-chamber, dual-chamber, and triple-chamber pressurization was measured. The results are shown in Figure 5.

The results of the actuator response time to different control signals are shown in Figure 5g–i. The measured pressure and actuator tip position can follow the reference signal. The average convergence time of the actuator tip under the three pressurization states are 0.63, 0.62, and 0.68 s, proving that the system can change the shape quickly. The response time of the pressurization is obviously faster than the depressurization, which is due to the pressure difference between the actuator and the external air source.

The long-term stability of the actuator is validated under an input pressure of 100 kPa, as shown in Figure 5j–l. We measured 1000 pressurization cycles in the same condition. The results show that no significant performance changes in the three states, which indicates the long-term usage capability of the actuator.

The hysteresis of the actuator under different maximum pressures was tested, as shown in Figure 5m–o. The actuator was pressurized from 0 to the maximum pressure and then depressurized to 0, and the change of the tip position of the actuator was recorded. The maximum errors of the actuator due to hysteresis were 9.7°, 14.82°, and 2.64 mm. The hysteresis increases as the maximum pressure increases, which is due to the stress relaxation of the silicone rubber material. In the future, we will add position closed-loop control to minimize the effect of hysteresis in the dynamic shape change control.

Shape display validation

To verify that this shape display device can present a variety of shapes, we built a prototype experimental verification platform. Three processes of deformation from the flat shape to the target shape were included, and two representative intermediate shapes were selected for comparison, as shown in Figure 6. The detailed experiments are shown in Supplementary Video S1. The initial state of the device is a planar shape. We used three shape sets, that is, convex ball, hillsides, and saddle. The air pressure in the soft actuator chambers was controlled to display the corresponding shapes. We recorded the chamber air pressure, actual shape, and target shape for the three shape sets. All shape deformations are based on the proposed modified shape control model to obtain target pressure, adjusted for actual installation errors. The 20 Hz closed-loop control of the air pressure is maintained through the air pressure sensor feedback. We will place sensors inside the device to realize closed-loop control of the device's position in the future work.

FIG. 6.

The contrast between the target shape and the actual shape of the device. The computer sent the target shape to the control system, and the camera recorded the shape. The camera's position relative to the device was the same as in the target shape. The camera-shot shapes and the target shapes were compared. (a) In changing from a plane to a convex spherical shape, the first step is to tilt the three units on the left side, the second is to tilt the two units on the front side, and the third is to tilt the three units on the back side and the right side. (b) In changing from a plane to a hillsides shape, the first step is to tilt the left front unit, the second is to tilt the left rear unit, and the third is to tilt the right three units. (c) In changing from a plane to a saddle shape, the first step repeats the steps in (a) to become a convex spherical shape. In the second step, tilt the left three units, and tilt the right three units in the third step. (d) The user experimentation scenarios. (e) The results of user experiments simulating three sets of shapes.

The experiment shows the potential and limitations of the device to present the C⁰-continuity shapes. We found that when the angle change between the units is small, flexible joint stretch little, which is not perceived by the hand. The device consisting of multiple units can reasonably achieve the representation of multiple complex shapes in the continuous shape workspace by combining different air pressures in 27 chambers.

However, when the angle between the units continues to increase in the convex ball state of Figure 6a, the device deforms out of the continuous shape workspace. The larger stretch of the flexible joint could be perceived by the human hand, which does not satisfy the C⁰-continuity. The experimental results show that the device is suitable for simulating specific shapes in which the height gradient of adjacent units is <15 mm, the angle gradient is <30°, and the tensile deformation of the flexible hinges of adjacent units is <2 mm. The relationship between the tensile properties of flexible joints and the continuous working space of the device will be investigated to render other types of shapes in our future work.

To evaluate the haptic effectiveness of the device, eight student participants (age 21.88 ± 1.36 years, seven males and one female) from Beihang University were recruited for the experiment. They are all right-handed and reported no history of neurological or psychiatric diseases. Written informed consent was obtained from all participants.

For each participant, three sets of experiments were conducted, with four shapes per set, which used the types in Figure 6. In each set, a total of 32 trials were performed, with 8 trials for each shape. One of the four shapes was randomly displayed for each set, and the participants were required to distinguish between the four shapes.

Before the formal experiments began, we introduced our device to the participants and allowed users to interact with the device. The experimenter presented the four shapes within the current set, and users experienced and familiarized with the four shapes. Until the users thought that they were ready, we started the formal experiment. The experimental scenario is shown in Figure 6d. Participants wore earplugs to block the faint sounds of the pneumatic system and were not allowed to watch the device.

In the formal experiments, users were required to take their hands off the device surface before and after each trial. During the trial, users were free to explore the surface of the device to obtain the shape experience. If users failed to perceive the relevant shapes, they could choose to give up reporting.

The experimental results are shown as the confusion matrix in Figure 6e. The values of the units along the diagonal line are significantly greater than those of the other units. The average recognition accuracy for the three sets of experiments is 92.45%. This result illustrates that the proposed device can support participants to correctly distinguish several different C⁰-continuity shapes by free touch. The recognition accuracy of states 2, 3 of set 1 and state 4 of set 2 is lower. We conclude that the reason is the user's interaction habits during the experiment. Participants were used to perceiving the motion of the g, h, i units through the ring finger and pinky and the motion of the f, i units through the palm. The perception thresholds in these places are lower compared with the thumb, index finger, and middle finger, which leads to lower recognition accuracy.^52,53

Conclusions

In this study, we proposed a C⁰-continuity shape display and established a model of the maximal number of distinguishable shapes. Based on this, the actuator unit was designed with 3 DOF (elongation, pitch, and yaw), and the joint was designed to be flexible. A novel shape display device with trichamber fiber-reinforced soft actuators was designed, modeled, fabricated, controlled, and tested.

The model of the maximal number of distinguishable shapes was established to analyze the shape discretization process and unit connection method. With the DOF of the actuator increasing from 1 to 3, the device resolved discontinuities between neighboring units combined with flexible joints. We developed an analytical model of the soft actuator unit and the connection joint and a shape control model of the multiunit soft array shape display device. The performance specification experiments show the potential of analytical model for accurately modeling of the device. The shape display validation experiment demonstrated the variation of three sequences of typical continuous shapes (convex ball, hillsides, and saddle), which shows the ability of model for accurate control to target shape. Such a device can satisfy multiple distinguishable C⁰-continuity shape representation in a small deformation range.

In the future, some experiments involving active human interaction considering the dynamics of the device (different organs or different shapes of the same organ) will be conducted for fast and diverse organ simulation in medical training. Eventually, we will combine this device with more immersive VR technology. Through the overall movement of this device, users can freely explore different shapes in different areas to meet the haptic requirements of full-body medical training, large-scale VR interaction, and even metaverse development.

Footnotes

Author Disclosure Statement

No competing financial interests exist.

Funding Information

This work is supported by the National Natural Science Foundation of China under Grant 62373021.

Supplementary Material

References

McNeely

WA.

Robotic graphics: A new approach to force feedback for virtual reality. In: Proceedings of IEEE Virtual Reality Annual International Symposium, Seattle, WA, USA; 1993; pp. 336–341.

Yokokohji

, Hollis

, Kanade

. What you can see is what you can feel-development of a visual/haptic interface to virtual environment. In: Proceedings of the IEEE 1996 Virtual Reality Annual International Symposium, Santa Clara, CA, USA; 1996; pp. 46–53.

Nakagaki

, Vink

, Counts

, et al. Materiable: Rendering dynamic material properties in response to direct physical touch with shape changing interfaces. In: CHI '16: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems; May 2016; pp. 2764–2772.

Siu

, Gonzalez

, Yuan

, et al. Shapeshift: 2D spatial manipulation and self-actuation of tabletop shape displays for tangible and haptic interaction. In: CHI '18: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems; April, 2018; Paper No.: 291; pp. 1–13.

Iwata

, Yano

, Nakaizumi

, et al. Project FEELEX: Adding haptic surface to graphics. In: SIGGRAPH '01: Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques; August 2001; pp. 469–476.

Follmer

, Leithinger

, Olwal

, et al. inFORM: Dynamic physical affordances and constraints through shape and object actuation. In: UIST '13: Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology; October 2013; pp. 417–426.

Leithinger

, Lakatos

, Devincenzi

, et al. Direct and gestural interaction with relief: A 2.5 D shape display. In: UIST '11: Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology; October 2011; pp. 541–548.

Nakagaki

, Fitzgerald

, Ma

, et al. inFORCE: Bi-directionalforce’ shape display for haptic interaction. ACM; 2019. In: Proceedings of the Thirteenth International Conference on Tangible, Embedded, and Embodied Interaction Tempe, Arizona, USA; 2019.

Leithinger

, Ishii

Relief: A Scalable Actuated Shape Display. MIT Media Lab: Cambridge, MA, USA; 2010.

10.

Siu

, Gonzalez

, Yuan

, et al. shapeShift: A mobile tabletop shape display for tangible and haptic interaction. In: UIST '17 Adjunct: Adjunct Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology; October 2017; pp. 77–79.

11.

Steed

, Ofek

, Sinclair

, et al. A mechatronic shape display based on auxetic materials. Nat Commun, 2021; 12(1):4758; doi: 10.1038/s41467-021-24974-0

12.

Taylor

, Pollet

, Hosseini-Sianaki

, et al. Advances in an electrorheological fluid based tactile array. Displays, 1998; 18(3):135–141; doi: 10.1016/s0141-9382(98)00014-6

13.

Liu

, Davidson

, Taylor

, et al. Single cell magnetorheological fluid based tactile display. Displays, 2005; 26(1):29–35; doi: 10.1016/j.displa.2004.10.002

14.

Wagner

, Lederman

, Howe

. A Tactile Shape Display Using RC Servomotors. Orlando, FL, USA; 2002.

15.

Yang

T-H

, Kim

S-Y

, Kim

, et al. Development of a miniature pin-array tactile module using elastic and electromagnetic force for mobile devices. In: World Haptics 2009—Third Joint EuroHaptics conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, Salt Lake City, UT, USA; 2009; pp. 13–17.

16.

Summers

, Chanter

, Southall

, et al. Results from a Tactile Array on the Fingertip. In: Proceedings of Eurohaptics, Birmingham, UK; 2001.

17.

Benko

, Holz

, Sinclair

, et al. Normaltouch and texturetouch: High-fidelity 3d haptic shape rendering on handheld virtual reality controllers. In: UIST '16: Proceedings of the 29th Annual Symposium on User Interface Software and Technology; October 2016; pp. 717–728.

18.

Stanley

, Okamura

. Controllable surface haptics via particle jamming and pneumatics. IEEE Trans Hapt, 2015; 8(1):20–30; doi: 10.1109/toh.2015.2391093

19.

Genecov

, Stanley

, Okamura

. Perception of a Haptic Jamming display: Just noticeable differences in stiffness and geometry. In: 2014 IEEE Haptics Symposium (HAPTICS), Houston, TX, USA; 2014; pp. 333–338.

20.

Stanley

, Gwilliam

, Okamura

. Haptic jamming: A deformable geometry, variable stiffness tactile display using pneumatics and particle jamming. In: 2013 World Haptics Conference (WHC), Daejeon, Korea (South); 2013; pp. 25–30.

21.

Stanley

, Hata

, Okamura

. Closed-loop shape control of a haptic jamming deformable surface. In: 2016 IEEE International Conference on Robotics and Automation (ICRA), Stockholm, Sweden; 2016; pp. 2718–2724.

22.

Stanley

, Okamura

. Deformable model-based methods for shape control of a haptic jamming surface. IEEE Trans Visual Comp Graph, 2017; 23(2):1029–1041; doi: 10.1109/tvcg.2016.2525788

23.

Koehler

, Usevitch

, Okamura

. Model-based design of a soft 3-D haptic shape display. IEEE Trans Robot, 2020; 36(3):613–628; doi: 10.1109/tro.2020.2980114

24.

Huang

, Zhakypov

, Sonar

, et al. A reconfigurable interactive interface for controlling robotic origami in virtual environments. Int J Robot Res, 2018; 37(6):629–647; doi: 10.1177/0278364918769157

25.

Chen

, Peng

, You

. Origami of thick panels. Science, 2015; 349(6246):396–400; doi: 10.1126/science.aab2870

26.

Foley

JD.

Computer Graphics: Principles and Practice. Addison-Wesley Professional: Reading, MA, USA; 1996.

27.

Sherrick

, Craig

. The psychophysics of touch. In: Tactual Perception: A Sourcebook. Cambridge University Press: New York, NY, USA; 1982; pp. 55–81.

28.

Bosscher

, Ebert-Uphoff

. Digital clay: Architecture designs for shape-generating mechanisms. In: 2003 IEEE International Conference on Robotics and Automation (Cat. No. 03CH37422), Taipei, Taiwan; 2003; pp. 834–841.

29.

Lederman

, Klatzky

. Haptic perception: A tutorial. Attent Percept Psychophys, 2009; 71(7):1439–1459; doi: 10.3758/app.71.7.1439

30.

Gaißert

, Waterkamp

, Fleming

, et al. Haptic categorical perception of shape. PLoS One, 2012; 7(8):e43062; doi: 10.1371/journal.pone.0043062

31.

Lazarus

, Verroust

. Three-dimensional metamorphosis: A survey. Visual Comp, 1998; 14(8–9):373–389; doi: 10.1007/s003710050149

32.

Zhang

YJ.

Geometric Modeling and Mesh Generation from Scanned Images. CRC Press: Boca Raton, FL, USA; 2016.

33.

Alliez

, Ucelli

, Gotsman

, et al. Recent Advances in Remeshing of Surfaces. In: Shape Analysis and Structuring. ( De Floriani

, Spagnuolo

eds.) Springer: Berlin, Heidelberg; 2008; pp. 53–82; doi: 10.1007/978-3-540-33265-7_2

34.

Campen

Partitioning surfaces into quadrilateral patches: A survey. Comp Graph Forum, 2017; 36(8):567–588; doi: 10.1111/cgf.13153

35.

Bommes

, Lévy

, Pietroni

, et al. Quad-mesh generation and processing: A survey. Comp Graph Forum, 2013; 32(6):51–76; doi: 10.1111/cgf.12014

36.

Kent

, Parent

, Carlson

. Establishing correspondences by topological merging: A new approach to 3-D shape transformation. In: Proceedings of Graphics Interface '91: Calgary, Alberta, Canada, June 3–7, 1991; pp. 271–278.

37.

Suzumori

, Maeda

, Wantabe

, et al. Fiberless flexible microactuator designed by finite-element method. IEEE ASME Trans Mech, 1997; 2(4):281–286; doi: 10.1109/3516.653052

38.

Rus

, Tolley

. Design, fabrication and control of soft robots. Nature, 2015; 521(7553):467–475; doi: 10.1038/nature14543

39.

Martinez

, Branch

, Fish

, et al. Robotic tentacles with three-dimensional mobility based on flexible elastomers. Adv Mater, 2013; 25(2):205–212; doi: 10.1002/adma.201203002

40.

Suzumori

, Iikura

, Tanaka

. Flexible microactuator for miniature robots. In: Proceedings of the IEEE Micro Electro Mechanical Systems, Nara, Japan; 1991; pp. 204–209.

41.

Marchese

, Rus

. Design, kinematics, and control of a soft spatial fluidic elastomer manipulator. Int J Robot Res, 2016; 35(7):840–869; doi: 10.1177/0278364915587925

42.

Ullrich

, Kuhlen

. Haptic palpation for medical simulation in virtual environments. IEEE Trans Visual Comp Graph, 2012; 18(4):617–625; doi: 10.1109/tvcg.2012.46

43.

Corwin

, Jaeger

, Nagel

. Structural signature of jamming in granular media. Nature, 2005; 435(7045):1075–1078.

44.

, Ranzani

, Sareh

, et al. Multi-fingered haptic palpation utilizing granular jamming stiffness feedback actuators. Smart Mater Struct, 2014; 23(9):095007.

45.

Brown

, Rodenberg

, Amend

, et al. Universal robotic gripper based on the jamming of granular material. Proc Natl Acad Sci USA, 2010; 107(44):18809–18814; doi: 10.1073/pnas.1003250107

46.

Tan

, Srinivasan

, Eberman

, et al. Human factors for the design of force-reflecting haptic interfaces. Dynam Syst Control, 1994; 55(1):353–359.

47.

Connolly

, Walsh

, Bertoldi

. Automatic design of fiber-reinforced soft actuators for trajectory matching. Proc Natl Acad Sci USA, 2017; 114(1):51–56.

48.

Polygerinos

, Wang

, Overvelde

JTB

, et al. Modeling of soft fiber-reinforced bending actuators. IEEE Trans Robot, 2015; 31(3):778–789; doi: 10.1109/tro.2015.2428504

49.

Singh

, Krishnan

. Designing fiber-reinforced soft actuators for planar curvilinear shape matching. Soft Robot, 2020; 7(1):109–121.

50.

Zhang

, Yang

, Yan

, et al. Inchworm inspired multimodal soft robots with crawling, climbing, and transitioning locomotion. IEEE Trans Robot, 2022; 38(3):1806–1819; doi: 10.1109/tro.2021.3115257

51.

Ogden

RW.

Non-linear Elastic Deformations. Courier Dover Publications: Mineola, NY, USA; 1997.

52.

Zoeller

, Drewing

. A systematic comparison of perceptual performance in softness discrimination with different fingers. Attent Percept Psychophys, 2020; 82:3696–3709.

53.

Craig

, Lyle

. A comparison of tactile spatial sensitivity on the palm and fingerpad. Percept Psychophys, 2001; 63(2):337–347.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

2.80 MB

38.52 MB

0.00 MB