Multi-agent systems are of ever-increasing importance in a contested space environment—use of multiple, cooperative satellites potentially increases positive mission outcomes on orbit, while autonomy becomes an ever-increasing requirement to increase reaction time to dynamic situations and lower the burden on space operators. This research explores multi-agent satellite swarm Guidance, Navigation, and Control (GNC) using deep reinforcement learning (DRL). DRL policies are trained to provide guidance inputs to agents in multi-agent swarm environments for completing complex, teamwork-focused objectives in geosynchronous orbit. An example scenario is explored for a group of satellite agents maneuvering to triangulate an object that is non-stationary in the relative orbit frame. Reward shaping is used to encourage learning guidance that positions swarm members to maximize triangulation accuracy, using angles-only observations for navigation relative to the target. Results show the policies successfully learn guidance through reward shaping to improve triangulation accuracy by a significant factor.
McCamishSRomanoMYunX. Autonomous distributed control algorithm for multiple spacecraft in close proximity operations. In: AIAA guidance, navigation and control conference and exhibit. Guidance, navigation, and control and co-located conferences, Hilton Head, SC, 20–23 August 2007. Reston, VA: AIAA.
6.
SaundersDE. A framework for autonomous cooperative optimal assignment and control of satellite formations. PhD Thesis, Air Force Institute of Technology, Dayton, OH, 2021.
7.
ZhaiGZhangJZhouZ. On-orbit target tracking and inspection by satellite formation. J Syst Eng Electr2013; 24: 879–888.
8.
KoenigAWD’AmicoS. Robust and safe N-spacecraft swarming in perturbed near-circular orbits. J Guid Contr Dyn2018; 41: 1643–1662.
9.
BernhardBChoiCRahmaniA, et al. Coordinated motion planning for on-orbit satellite inspection using a swarm of small-spacecraft. In: 2020 IEEE aerospace conference, Big Sky, MT, 7–14 March 2020, pp. 1–13. New York: IEEE.
10.
DonoAPliceLMuetingJ, et al. Propulsion trade studies for spacecraft swarm mission design. In: 2018 IEEE aerospace conference, Big Sky, MT, 3–10 March 2018, pp. 1–12. New York: IEEE.
11.
MorganDChungSJHadaeghFY. Model predictive control of swarms of spacecraft using sequential convex programming. J Guid Contr Dyn2014; 37: 1725–1740.
12.
NagSSummererL. Behaviour based, autonomous and distributed scatter manoeuvres for satellite swarms. Acta Astronautica2013; 82: 95–109.
HovellKUlrichS. Deep reinforcement learning for spacecraft proximity operations guidance. J Spacecr Rock2021; 58: 254–264.
15.
ClohessyWHWiltshireRS. Terminal guidance system for satellite rendezvous. J Aerosp Sci1960; 27: 653–658.
16.
MnihVKavukcuogluKSilverD, et al. Human-level control through deep reinforcement learning. Nature2015; 518: 529–533.
17.
WangFZhangJJZhengX, et al. Where does AlphaGo go: from church-turing thesis to AlphaGo thesis and beyond. IEEE/CAA J Autom Sin2016; 3: 113–120.
18.
HeYXingLChenY, et al. A generic Markov decision process model and reinforcement learning method for scheduling agile earth observation satellites. IEEE Trans Syst Man Cyb Syst2022; 52: 1463–1474.
19.
ZhouZZhangWDingJ, et al. Cooperative pursuit with Voronoi partitions. Automatica2016; 72: 64–72.
20.
SargentLHCoverstoneVLRodriguezN, et al. Toward reinforcement learning identification for swarms engaged in cooperative pursuit. In: AIAA SCITECH 2022 forum, San Diego, CA, 3–7 January. Reston, VA: American Institute of Aeronautics and Astronautics.
21.
LeiHH ShubertMDamronN, et al. Deep reinforcement learning for multi-agent autonomous satellite inspection. In: 44th annual AAS guidance, navigation and control (GN&C) conference, Breckenridge, CO, 3–9 February 2022.
22.
KaplanG. Angles-only navigation: position and velocity solution from absolute triangulation. Navigation2011; 58: 187–201.
LiangELiawRMoritzP, et al. RLlib: abstractions for distributed reinforcement learning. arXiv:1712.09381 [cs], 2018.
25.
SchulmanJWolskiFDhariwalP, et al. Proximal policy optimization algorithms. arXiv:170706347 [cs], 2017.
26.
BrockmanGCheungVPetterssonL, et al. OpenAI gym. arXiv:1606.01540, 2016.
27.
AbadiMAgarwalABarhamP, et al. TensorFlow: large-scale machine learning on heterogeneous systems. In: Proceedings of the 12th USENIX conference on Operating Systems Design and Implementation (OSDI’16), 2015, pp. 265–283. USA: USENIX Association.
28.
KirkJTCainSC.Multi-hypothesis test detection for star tracking systems. In: AIAA scitech 2021 forum, Virtual, 19–21 January 2021. Reston, VA: American Institute of Aeronautics and Astronautics.
29.
HarrisATeilTSchaubH. Spacecraft decision-making autonomy using deep reinforcement learning. In: 29th AAS/AIAA space flight mechanics meeting, Maui, Hawaii, 13–17 January 2019, pp. 1–19. AAS Spaceflight Mechanics Meeting, paper No. AAS-19-447
30.
RavaioliUJCunninghamJMcCarrollJ, et al. Safe reinforcement learning benchmark environments for aerospace control systems. In: 2022 IEEE aerospace conference (AERO), Big Sky, MT, 5–12 March 2022, pp. 1–20. New York: IEEE.
