Multiple unmanned aerial vehicle coordinated strikes against ground targets based on an improved multi-agent deep deterministic policy gradient algorithm

Abstract

With the development of swarm intelligence, multi-unmanned aerial vehicle (UAV) cooperation in uncertain dynamic environments has been able to safely and efficiently realize all-round strikes on ground targets. However, the limited life span of UAV onboard batteries challenges the cooperative strikes against the ground targets. To address this challenge, this paper proposes an improved multi-agent deep deterministic policy gradient (i-MADDPG) algorithm for energy-saving cooperative strikes of ground targets. Introducing a sensor fusion layer and self-attention mechanism to the actor network helps the UAVs collect more comprehensive information and filter the collected data, resulting in more accurate environmental perception. In addition, importing an egocentric state representation mechanism into the critic network contributes to the computation of different Q-values for different UAV observation states, which ensures each UAV can make appropriate decisions based on its own state. Simulation experiments are conducted to validate the performance of the improved MADDPG algorithm. The results show that the success rate of ground target strikes of the proposed algorithm is improved by 29.5%, and the collision rate is reduced compared with the traditional MADDPG algorithm.

Keywords

Multi-agent reinforcement learning sensor fusion layer self-attention egocentric state representation mechanism multiple UAV

Get full access to this article

View all access options for this article.

References

Zhang

Wang

On uav source seeking with complex dynamic characteristics and multiple constraints: a cooperative standoff monitoring mode. Aerosp Sci Technol 2022; 121: 107315.

Zhang

Xing

Cooperative task assignment of multi-uav system. Chin J Aeronaut 2020; 33(11): 2825–2827.

Zhang

Zhao

Wei

, et al. Cooperative trajectory design of multiple uav base stations with heterogeneous graph neural networks. IEEE Trans Wirel Commun 2023; 22(3): 1495–1509.

Nguyen

Nahavandi

Deep reinforcement learning for multiagent systems: a review of challenges, solutions, and applications. IEEE Trans Cybern 2020; 50(9): 3826–3839.

Chang

Chen

Zhang

, et al. Multi-uav mobile edge computing and path planning platform based on reinforcement learning. IEEE Trans Emerg Top Comput Intell 2022; 6(3): 489–498.

Jiang

Deng

Nallanathan

, et al. Reinforcement learning for real-time optimization in NB-IoT networks. IEEE J Sel Areas Commun 2019; 37(6): 1424–1440.

Muñoz

Barrado

Çetin

, et al. Deep reinforcement learning for drone delivery. Drones 2019; 3(3): 72.

Renxian

Liang

Multi-ship collaborative collision avoidance strategy based on multi-agent deep reinforcement learning. Comput Integr Manuf Syst 2024; 30(6): 1972.

Liu

Wang

, et al. Task assignment of uav swarms based on deep reinforcement learning. Drones 2023; 7(5): 297.

10.

Xiao

Pisutsin

Feroskhan

Collaborative target search with a visual drone swarm: an adaptive curriculum embedded multistage reinforcement learning approach. IEEE Trans Neural Netw Learn Syst 2023. DOI: 10.1109/TNNLS.2023.3331370.

11.

Liu

Feng

Pei

, et al. Blockchain-enabled secure data sharing scheme in mobile-edge computing: an asynchronous advantage actor–critic learning approach. IEEE Internet Things J 2021; 8(4): 2342–2353.

12.

Wang

Liu

Hong

, et al. Madrl-based UAV swarm non-cooperative game under incomplete information. Chin J Aeronaut 2024; 37(6): 293–306.

13.

Peng

Berseth

van de Panne

, et al. Terrain-adaptive locomotion skills using deep reinforcement learning. ACM Trans Graph 2016; 35(4): 1–12.

14.

Liu

Tian

, et al. Reinforcement learning based two-level control framework of uav swarm for cooperative persistent surveillance in an unknown urban area. Aerosp Sci Technol 2020; 98: 105671.

15.

, et al. An attention mechanism and adaptive accuracy triple-dependent maddpg formation control method for hybrid uavs. IEEE Trans Intell Transp Syst 2024; 25(9): 11648–11663.

16.

Wang

Guo

, et al. Autonomous collaborative combat strategy of unmanned system group in continuous dynamic environment based on pd-maddpg. Comput Commun 2023; 200: 182–204.

17.

Zhan

Zhang

, et al. Multiple-uav reinforcement learning algorithm based on improved PPO in ray framework. Drones 2022; 6(7): 166.

18.

Grosfeld-Nir

A two-state partially observable markov decision process with uniformly distributed observations. Oper Res 1996; 44(3): 458–463.

19.

Savas

Hibbard

, et al. Entropy maximization for partially observable markov decision processes. IEEE Trans Automat Contr 2022; 67(12): 6948–6955.

20.

Mnih

Kavukcuoglu

Silver

, et al. Human-level control through deep reinforcement learning. Nature 2015; 518(7540): 529–533.

21.

Lillicrap

Hunt

Pritzel

, et al. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015.

22.

Lowe

Tamar

, et al. Multi-agent actor-critic for mixed cooperative-competitive environments. Adv Neural Inf Process Syst 2017; 30: 1–12.

23.

Wang

Gao

, et al. Uav swarm confrontation using hierarchical multiagent reinforcement learning. Int J Aerosp Eng 2021; 2021: 1–12.

24.

Jeon

Lee

Kaliappan

, et al. Multiagent reinforcement learning based on fusion-multiactor-attention-critic for multiple-unmanned-aerial-vehicle navigation control. Energies 2022; 15(19): 7426.

25.

Gulzar

Rizvi

STH

Javed

, et al. Multi-agent cooperative control consensus: a comparative review. Electronics 2018; 7(2): 22.

26.

Gulzar

Ling

Yaqoob

, et al. Realization of an improved path planning strategy. In: 2015 International conference on control, automation and information sciences (ICCAIS), 2015, pp.384–389. New York: IEEE.

27.

Yan

Xiang

Wang

, et al. Pascal: population-specific curriculum-based MADRL for collision-free flocking with large-scale fixed-wing UAV swarms. Aerosp Sci Technol 2023; 133: 108091.

28.

Gym

Sanghi

Deep reinforcement learning with python. Berlin: Springer, 2021.