Global Vision-Based Formation Control of Soft Robotic Fish Swarm

Abstract

Possessing the attributes of high adaptability and low cost, soft robotic individuals can further coordinate and form into a swarming system, enhancing the performances as well as functions in practical applications. However, the formation control of soft robotic swarm remains challenging mainly due to the limitation in relatively low precision and slow response of the soft actuators. In this work, a soft robotic fish swarm system with global vision positioning was studied. The soft robotic fish used in the project is driven by a hybrid power-control system, in which the soft dielectric elastomers and the rigid electrical servo provide forward propulsion and controllable steering function, respectively. Results show that soft robotic fish swarm can quickly shift their formations, mimicking three typical swarming behaviors of natural creatures: highly parallel group, encircling, and torus. The system design and controlling principles of the soft robotic fish swarm may guide the future research of soft robots and robotic swarms, specifically for underwater applications.

Introduction

Robots with rigid bodies are widely used for effective manufacturing, fast transportation, and exploration, but have challenges in achieving high environmental adaptability and excellent man/machine safety features. Soft robots with intrinsically soft or extensible materials provide an opportunity to bridge the gap between machine and human.^1–5 Natural creatures provide great biomimetic prototypes for soft robots, inspiring various designs mimicking fish, snakes, and insects.⁶ As one of the most demanding regions for exploration, the aquatic world, for example, the ocean has always been the pursuit of people.⁷ Compared with typical underwater robots (e.g., remote operated vehicle and autonomous underwater vehicle) driven by electrical motors, soft aquatic robots have low noise and high adaptability in complex environments. Various types of actuators are used in soft aquatic robots such as dielectric elastomer, shape memory alloy,⁸ ionic polymer metal composites,⁹ and pneumatics.¹⁰ Dielectric elastomer actuators (DEAs) have advantages for aquatic robots, possessing the attributes of fast response, large deformation, and high-energy density.¹¹ Recently, several DEA-driven soft aquatic robots (jellyfish,¹² manta-ray,¹³ leptocephali,¹⁴ fish,¹⁵ and frog¹⁶) have been reported, presenting excellent mobility and adaptability.

Although the soft aquatic robots show promising advantages in future application, relatively low power, small capacity, and slow speed limit their performance as exploring robots compared with conventional remote operated vehicle and autonomous underwater vehicle with rigid motors and structures.^17,18 Inspired by the animal swarms during migration and predation,¹⁹ multiple soft robots can group up a swarm to overcome the limitations of individually low power and capacity.^20,21 A soft robotic fish swarm can outperform complex underwater tasks in a more efficient way, gaining the advantages of robust, cheap, and versatile features. Inspired by the formation behavior of creatures in nature, the formation control of multiple cooperative robots has been recently well studied.^22–24 The three common animal swarm formation behaviors are as follows: highly parallel group (Fig. 1A),²⁵ encircling (Fig. 1B),²⁶ and torus (Fig. 1C).¹⁹ The highly parallel group behavior characterizes the phenomenon that the combined group moves toward a certain target with a fixed formation; the encircling behavior represents the phenomenon that the combined group reaches the aggregate movement in the bounded region; the torus behavior means that the combined group rotates around a certain aggregation center.²⁷

FIG. 1.

Three typical collective behaviors of natural creatures, robotic fish swarm, and practical experiments: (A) highly parallel group, (B) encircling, and (C) torus. The corresponding schematics of the robotic fish swarm and robots in practical experiments: highly parallel group (D, G), encircling (E, H), and torus (F, I).

Although there are many achievements in the swarming control of robots (driven by rigid motors or electrical servos) such as robotic cars,²² quadrotors,²³ and robotic fish,²⁴ few studies have been reported on the formation control of soft robotic swarm. The soft robotic swarm has the same advantages as soft robots and will do well in some special cases such as exploration of quiet or complex environments. Li et al. have studied a soft active origami robot and developed a theoretical model to interpret the actuation mechanism offering inexpensive solutions to swarm robots.²⁸ Mintchev et al. have designed a soft spherical robot to form swarm for random exploration of terrestrial environments.²⁰ Won et al. have proposed a magnetic soft robot swarm that is capable of transportation of cargo with swimming functionality.²¹

The formation control of soft robotic swarm remains challenging due to the low precision and slow response of the soft actuators. To overcome the difficulties of swarming control with solely soft actuators, a hybrid soft robotic fish driven by both soft actuator (dielectric elastomer) and steering electrical servo was designed in this work. The dielectric elastomers actuate the flapping wings to provide forward propulsion. Steering electrical servo drives the steering tail to control the heading-angle, and assisting the propulsion of the robotic fish. We built up a multiple robotic fish visual locating and control system to operate the soft robotic fish swarm, imitating three animal swarm formation behaviors: highly parallel group (Fig. 1D, G), encircling (Fig. 1E, H), and torus (Fig. 1F, I). In this work, we mainly want to investigate the principle and possible control methods for the soft robotic fish swarm. We choose three as the typical and smallest number to better illustrate the mechanism of soft robotic swarm control.

Materials and Methods

Materials

The soft robotic fish consists of three parts: the body, the flapping wing, and the steering tail. The body part contains a highly compact electric system, which is sealed in a centrifuge tube. The tail is laser cut from acrylic sheet (thickness of 0.1 mm) into a hydrodynamic shape. The wing of the fish is composed of the following seven components (Fig. 2B). The reinforcing frame is made by polymethyl methacrylate (PMMA) with thickness of 1 mm. The dielectric elastomer (DE) membranes are made by the stacks of VHB 4910 [3M] film, VHB 4905 [3M] film, and the sandwiched carbon grease layer (MG Chemicals Carbon Conductive Grease 846-80G). The feed line is a strip of tin foil (thickness of 0.08 mm) with the width of 3 mm. The elastic frame is made by a 65 Mn steel thin slide with the thickness of 0.1 mm. The insulating sheet is made by polyethylene terephthalate (PET) with the thickness of 0.1 mm. The passive waving fin (attached on the flapping wing frame) is made by silicone film with the thickness of 0.1 mm.

FIG. 2.

Fabrication and actuation. (A) The mechanism composition of the soft robotic fish. The soft fish consists of the Epod, a steering electrical servo, a steering tail, and two flapping wings. (B) Two DE membranes sandwich the electrodes to form artificial muscle. The precut frame and the precut reinforcement stick to each side of the muscle. The insulating sheet aims to prevent the feed line from coming into contact with the support frame. The flexible wave fin is to provide impetus when the wings flap. (C) Prestretched state and actuated state of the wing. (D) When applying an alternating voltage, the wings change back and forth between prestretched state and actuated state providing forward force. (E) Steering electrical servo turns to a deflection angle and oscillates at a small angle to provide steering force. DE, dielectric elastomer.

Fabrication

The soft robotic fish is integrated by four parts: (1) flapping wings, (2) the body (the Epod), (3) steering electrical servo, and (4) steering tail (Fig. 2A). Two DE membranes in the flapping wings function as the artificial muscle. The DE membranes have the initial thickness of 1.5 mm, which is made by stacking a VHB 4910 [3M] film (1 mm thick) and a VHB 4905 [3M] film (0.5 mm thick). The strip feed line is drawn from the DE membrane-sandwiched carbon grease electrode to the edge of the wing. As the material of the elastic frame is conductive, we design an insulating sheet near the edge of the wing to where the strip feed line is drawn. The reinforcing frame (PMMA), the elastic frame (65 Mn), and the insulating sheet stack together with the DE membranes, integrating the flapping wing (Fig. 2B).

The body of the robotic fish contains a wireless module, an eight-pin microcontroller unit, and a high-voltage supply. It can provide an output alternating voltage of 0–10 kV when powered by a 3.7 V lithium-ion battery. A high-voltage output line, a ground line, and three servo lines are drawn out from the body, connecting to the flapping wings, the surrounding water, and the steering electrical servo (Fig. 2A).

Actuation of the soft robot

The power of the hybrid fish is composed of the forward propulsion from the wings and the steering force from the tail. The prestretched DE membranes attach on and deform the elastic frame into a flapping shape with the bending angle up to 90° (Fig. 2C). When a high voltage is applied on the DE membranes, the flapping wings relax with decreased bending angle. The alternating voltage drives the flapping wing up and down, generating forward propulsion¹³ (Fig. 2D). The steering servo drives the tail (turn to a deflection angle or oscillate at a small angle), controlling the moving direction of the robotic fish (Fig. 2E). Based on previous work,¹³ the swimming speed of the fish is positively correlated with the amplitude of the applied voltage (6–9 kV) with a fixed alternating frequency (0.6 Hz).

Control method

We use a multiple robotic fish visual locating and control system as our platform, which consists of swimming area, digital camera, wireless module, soft robotic fish, and host computer (Fig. 3).^22,29,30 Based on the visual images from the overhead camera, the system is able to track the position and direction of the fish and the target. The global vision-based system can avoid the need of extra sensors and better microcontroller, making the robot fish small in size and light in weight. A lightweight soft robotic fish is needed because of the flapping wing's small propulsion. Due to the condition that the robotic fish reaches the final speed (relatively low speed of ∼3.0 cm/s) in a short time (∼500 ms), we hereby assume the acceleration process of the robotic fish can be ignored. A control algorithm was developed, which converts the position (x, y) and orientation (θ) into the applied voltage V on dielectric elastomer and the steering angle λ on the electrical servo of the robotic fish. This strategy can be easily adjusted for various soft robot units and can be applied in the case of low-coupling effects of forward propulsion and steering force.

FIG. 3.

Multiple robotic fish visual locating and control system.

Individual robot approaches a target

We have first investigated the control of an individual robot approaching a target, which is the basic method for the formation control of highly parallel group and encircling. Located in Cartesian coordinates (Fig. 4A), the current state of the robotic fish is described by the position (x, y) and orientation (θ). Based on the current position of target (x_T, y_T), the tilting angle (Γ) and the distance (d) between the robotic fish and the target are calculated. We design the control algorithm such that the applied voltage V is proportional to d, and the steering angle of the servo λ is proportional to (Γ) (Fig. 4).

FIG. 4.

(A) The schematic diagram illustrates the current pose of the fish (x, y, θ), the target (x_T, y_T), the distance d, and the tilting angle Γ. (B) The flowchart of control algorithm (β is calculated by atan2 function in Python).

Highly parallel group

We use the leader/follower method (leading fish, follower 1, and follower 2) to control the formation of three robotic fish into a highly parallel group (Fig. 5). We apply invariable voltage and steering angle to the leader fish. The leader fish moved freely and circled in the same place. To ensure that the followers can follow the leader fish at once, the voltage of the leader fish is smaller than the other two followers' voltage. According to the current state of the leading fish [the position (x₀, y₀) and orientation (θ₀)] and the preset formation, we can obtain the current targeting positions (x₃, y₃) and (x₄, y₄) of the two followers. Then the previous algorithm (individual robot approaches a target) is used for follower 1 and follower 2, and thus, they reach the predetermined targets to achieve a triangle formation (Fig. 5). During the process of grouping, the swimming speed of the leading fish decreases. Based on the current state of the leading fish, the targeting positions for the followers continue to be updated until the two followers catch up and keep the preset formation (the blue dashed triangle in Fig. 5A).

FIG. 5.

(A) The schematic diagram illustrates the three robotic fish for highly parallel group. The targeting positions of the two followers are continually updated. (B) The flowchart of control algorithm.

Encircling

To demonstrate the encircling behavior, we control five robotic fish simultaneously approaching a target (a fixed red ball) (Fig. 6).

FIG. 6.

(A) The schematic diagram illustrates the five robotic fish for encircling. (B) The flowchart of control algorithm.

Torus

We first investigated the control of individual robot swimming in circle, which is the basic method for the formation control of torus (Fig. 7). During the circling control, the applied voltage of the fish remains constant, while the control signal for the electrical servo varies. Each robotic fish has the coordinate of (x, y, θ) and the center point has the coordinate of (x₀, y₀). The angle between the fish-target circular tangent and the fish's direction is β. The distance between fish and center point is l. According to the position of the fish and the target circle, we adopt two control strategies. When the fish is outside the circle, we tune the angular velocity of the fish so that the fish moves toward the circular tangent. When the fish is inside the target circle, the fish is controlled to swim in the direction perpendicular to the center point.

FIG. 7.

(A) The schematic diagram illustrates individual robot swimming in circle. The positions and orientation of fish are (x, y, θ). The position of the center point is (x₀, y₀). There are two methods for the fish inside and outside the circle, respectively. (B) The flowchart of control algorithm (β is calculated by atan2 function in Python).

Now we control three robotic fish to imitate the torus behavior. Each robotic fish is controlled with the circling algorithm to keep the angle of 120° from the center point (Fig. 8). When multiple robotic fish form a torus group from the beginning of random distribution, there is a colliding problem. Applying a repulsive force to each other that is positively correlated to the distance will be a useful solution. We will delve into this problem in future studies.

FIG. 8.

(A) The schematic diagram illustrates the three robotic fish for torus. The positions and orientation of three fish are (x₁, y₁, θ₁), (x₂, y₂, θ₂), and (x₃, y₃, θ₃). The position of the center point is (x₀, y₀). (B) The flowchart of control algorithm.

Results and Discussion

We have designed the hybrid-power soft robotic fish, with the control of both the soft actuator and the steering electrical servo. When the steering servo on the tail is squared, the robotic fish swims straight forward by the flapping wings. The swimming speed is regulated by applied voltage on dielectric elastomer. The steering direction is regulated by the control of electrical servo. Based on the multiple robotic fish visual locating and control system, the robotic fish swarm imitates three typical formations of natural creatures: highly parallel group, encircling, and torus.

We characterized the electromechanical performances of the flapping wing by the quasistatic tests. The amplitude of the bending angle for the flapping wing increases when the applied voltage ramps up from 6 kV. When the applied voltage reaches 9 kV, the prebended wing can deform into an almost flat shape (with the flapping amplitude of 45.22°). To avoid the electrical break down of dielectric elastomer during the driving process, the applied voltage on the robotic fish is regulated in the range of 6–9 kV. We scan the frequencies of applied voltage from 0.4 to 2.0 Hz to drive the robotic fish. The robot fish reaches its maximum speed of 6.2 cm/s at the voltage of 9 kV with the frequency of 0.6 Hz. Based on the varied speed and swimming states during the tests, we fix the driving frequency at 0.6 Hz during the formation control. The steering servo has the turning angle of −50 to 50°. The fish has the smallest turning radius of 0.234 m when the fish's tail steering angle is maximized.

Figure 9 shows the experimental results of formation control of the robotic fish corresponding to various formations and strategies described in the previous section (details of the formation control can be found in the Supplementary Video S1). The colored lines show the swimming trajectories of the robotic fish during formation control.

FIG. 9.

Experimental results of formation control. (A) Individual robot approaches a target. (B) Highly parallel group. (C) Encircling. (D) Individual fish swims in circle. (E) Torus. Color images are available online.

From the experimental results, the control algorithms are verified with good stability in practice. To investigate the accuracy of the control, we measured the real-time radius of the three fish during torus formation (with the preset circling radius of 541.67 mm for three fish). As the other two behaviors do not have a predetermined trajectory, we do not provide their errors. When all three robotic fish have entered the circling trajectory, the real circling radius per frame for fish No. 1–3 is recorded (Fig. 10). Table 1 shows their average and the maximum error between the preset and the real circling radius. During the tests, the orientation of robotic fish No. 1 was mistakenly identified in a few frames (the light red zone in Fig. 10A), resulting in a peaked radius with the maximum error (54.79%), which was then quickly corrected by the visual identifying system. All the robotic fish have average errors less than 10%, while the maximum errors exceed 20%. The errors mainly come from two aspects: (1) the fluctuated propulsion of the flapping wings, resulting from both the low actuating precision of dielectric elastomer and the structural defects from the fabrication; and (2) the error from the control algorithm and the visual tracking system.

FIG. 10.

The real-time radius per frame in torus behavior. (A) The first fish. (B) The second fish. (C) The third fish.

Table 1.

Motion Trajectory Error of Three Fish in Torus Behavior

Fish	The average error (%)	The maximum error (%)
No.1	9.23	54.78
No.2	7.81	20.08
No.3	9.37	24.82

Conclusions

In summary, we have developed a hybrid robot fish by combining servo drive and DE drive. The hybrid design of the robotic fish can provide servo-driven propulsion for the soft-actuated robotic fish with enhanced mobility. While the included motors in the system may cause a compromise of soft robots such as low noise, high adaptability, and human/machine safety. The future research may focus on how to enhance the reliability and robustness of the hybrid system and improve the control algorithm. Based on the multiple robotic fish visual locating and control system, the robotic fish swarm can imitate three typical formation behaviors: highly parallel group, encircling, and torus. We will extend this method to more formation patterns that can be customized in the future. The control algorithm of the formation control has been verified with good stability, while the control accuracy requires further improvements. The robustness of the whole system still needs to be tested in other environments. In the future design of soft robotic fish swarm, to improve the control accuracy, more powerful actuators and standardized fabrication methods shall be used to improve the stabilization and driving precision of the robot. In addition, dynamic model for the robotic fish can be embed into the control algorithm, improving the control accuracy. Modeling the soft wings in water is very difficult, so a data-driven model may be a practical method. Because we have not modeled the dynamics of our robotic fish and there are also differences between individual fish for handmade reasons, we did not show a lot of control data. In future research, after we improve the fabrication process, we will analyze the control data and model the dynamics of our robotic fish.

Footnotes

Author Disclosure Statement

No competing financial interests exist.

Funding Information

This work acknowledges support from the following programs: National Key R&D Program of China (2017YFA0701100), National Natural Science Foundation of China (11822207, U1613202, 11572280, 61633002, 91648120), Zhejiang University, and Dr. Li Dak Sum and Yip Yio Chin Fund for Stem Cell and Regenerative Medicine.

Supplementary Material

References

Rus

, Tolley

. Design, fabrication and control of soft robots. Nature, 2015; 521:467–475.

Polygerinos

, Wang

, Galloway

, et al. Soft robotic glove for combined assistance and at-home rehabilitation. Robot Auton Syst, 2015; 73:135–143.

Trivedi

, Rahn

, Kier

, et al. Soft robotics: biological inspiration, state of the art, and future research. Appl Bionics Biomech, 2008; 5:99–117.

Yang

, Xiao

, Zhang

, et al. A soft artificial muscle driven robot with reinforcement learning. Sci Rep-UK, 2018; 8:14518.

Kim

, Laschi

, Trimmer

. Soft robotics: a bioinspired evolution in robotics. Trends Biotechnol, 2013; 31:287–294.

, Zou

, Mao

, et al. Agile and resilient insect-scale robot. Soft Robot, 2019; 6:133–141.

Fish

, Kocak

. Biomimetics and marine technology: an introduction. Mar Technol Soc J, 2011; 45:8–13.

Villanueva

, Smith

, Priya

. A biomimetic robotic jellyfish (Robojelly) actuated by shape memory alloy composite actuators. Bioinspir Biomim, 2011; 6:036004.

Yeom

, Oh

. A biomimetic jellyfish robot based on ionic polymer metal composite actuators. Smart Mater Struct, 2009; 18:085002.

10.

, Fish

, Chen

, et al. Bio-inspired robotic dog paddling: kinematic and hydro-dynamic analysis. Bioinspi. Biomim, 2019; 14:066008.

11.

Gupta

, Qin

, Wang

, et al. Soft robots based on dielectric elastomer actuators: a review. Smart Mater Struct, 2019; 28:103002.

12.

Cheng

, Li

, Liang

, et al. Untethered soft robotic jellyfish. Smart Mater Struct, 2018; 28:015019.

13.

, Li

, Liang

, et al. Fast-moving soft electronic fish. Sci Adv, 2017; 3:e1602045.

14.

Christianson

, Goldberg

, Deheyn

, et al. Translucent soft robots driven by frameless fluid electrode dielectric elastomer actuators. Sci Robot, 2018; 3:eaat1893.

15.

Shintake

, Cacucciolo

, Shea

, et al. Soft biomimetic fish robot made of dielectric elastomer actuators. Soft Robot, 2018; 5:466–474.

16.

Wang

, Huang

, McCoul

, et al. A soft breaststroke-inspired swimming robot actuated by dielectric elastomers. Smart Mater Struct, 2019; 28:045006.

17.

Bovio

, Cecchi

, Baralli

. Autonomous underwater vehicles for scientific and naval operations. Annu Rev Control, 2006; 30:117–130.

18.

Sahoo

, Dwivedy

, Robi

. Advancements in the field of autonomous underwater vehicle. Ocean Eng, 2019; 181:145–160.

19.

Vicsek

, Zafeiris

. Collective motion. Phys Rep, 2012; 517:71–140.

20.

Mintchev

, Zappetti

, Willemin

, et al. A soft robot for random exploration of terrestrial environments. In: Proceedings from 2018 IEEE International Conference on Robotics and Automation (ICRA), Brisbane Australia. Piscataway, NJ: IEEE, 2018, pp. 7492–7497.

21.

Won

, Kim

, Park

, et al. On-demand orbital maneuver of multiple soft robots via hierarchical magnetomotility. Nat Commun, 2019; 10:1–8.

22.

Wang

, Tnunay

, Zuo

, et al. Fixed-time formation control of multirobot systems: design and experiments. IEEE Trans Ind Electron, 2018; 66:6292–6301.

23.

Montijano

, Cristofalo

, Zhou

, et al. Vision-based distributed formation control without an external positioning system. IEEE Trans Robot, 2016; 32:339–351.

24.

Zhang

, Yan

, Xie

. Consensus-based leader-following formation control for a group of semi-biomimetic robotic fishes. Int J Adv Robot Syst, 2017; 14:1729881417720784.

25.

Bajec

, Heppner

. Organized flight in birds. Anim Behav, 2009; 78:777–789.

26.

Muro

, Escobedo

, Spector

, et al. Wolf-pack (Canis lupus) hunting strategies emerge from simple rules in computational simulations. Behav Processes, 2011; 88:192–197.

27.

Couzin

, Krause

, James

, et al. Collective memory and spatial sorting in animal groups. J Theor Biol, 2002; 218:1–11.

28.

, Godaba

, Zhang

, et al. A soft active origami robot. Extreme Mech Lett, 2018; 24:30–37.

29.

Yan

, Zhan

. Real-time multiple mobile robots visual detection system. Sens Rev, 2011; 31:228–238.

30.

Yuan

, Yu

, Wu

, et al. Precise planar motion measurement of a swimming multi-joint robotic fish. Sci China Inf Sci, 2016; 59:92208.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

163.01 MB

0.00 MB