Active non-cooperative target tracking is an important ability for free-floating space manipulator (FFSM) systems. Aiming at the difficulty of pose and posture estimation of non-cooperative targets and the complexity of FFSM planning and control, the paper proposed an end-to-end target tracking algorithm using deep reinforcement learning (DRL) algorithm. The algorithm uses raw camera images as input and directly outputs the manipulator joint velocities. To deal with the partially observable Markov decision processes (POMDP) problem caused by single image input, we combine the soft actor–critic algorithm with recurrent neural networks. Therefore, the velocity information of the target is introduced to transform the task into a fully observable Markov decision-making process (MDP). Our approach is completely data-driven. It avoids the position and posture estimation of non-cooperative target and the complex modeling motion planning process of FFSM. We trained and tested the algorithm by building a simulation environment in CoppeliaSim. The results showed that our approach achieved good results in active non-cooperative target tracking of FFSM systems and demonstrated the great potential of DRL algorithms in solving space tasks.
ArdaensJSGaiasG (2017) Angles-only relative orbit determination during the avanti experiment. In: Proceedings of the 26th international symposium on space flight dynamics (ISSFD), Matsuyama, Japan, 3–9 June, pp. 1–8. Available at: https://issfd.org/
2.
CaoYWangSZhengX, et al. (2023) Reinforcement learning with prior policy guidance for motion planning of dual-arm free-floating space robot. Aerospace Science and Technology136: 108098.
3.
CardenasAQuirozOHernándezR, et al. (2021) Vision-based control of a mobile manipulator with an adaptable-passive suspension for unstructured environments. Journal of Mechanisms and Robotics: Transactions of the ASME13(5): 050908.
4.
ChangKMaDHanX, et al. (2020) Lyapunov vector-based formation tracking control for unmanned aerial vehicles with obstacle/collision avoidance. Transactions of the Institute of Measurement and Control42(5): 942–950.
5.
ChangRJiaQChuM (2022) Stabilization control for spacecraft-manipulator system after capturing tumbling target. International Journal of Control, Automation and Systems20(11): 3706–3717.
6.
Cortés-PérezNTorres-MéndezLA (2021) A mirror-based active vision system for underwater robots: From the design to active object tracking application. Frontiers in Robotics and AI8: 542717.
7.
D’AmicoSArdaensJSLarssonR (2012) Spaceborne autonomous formation-flying experiment on the PRISMA mission. Journal of Guidance, Control, and Dynamics35(3): 834–850.
8.
DelpechMBergesJDjalalS, et al. (2012) Preliminary results of the vision based rendezvous and formation flying experiments performed during the PRISMA extended mission. Advances in the Astronautical Sciences145: 1375–1390.
9.
DeutschBNiemannHDenzlerJ (2005) Multi-step active object tracking with entropy based optimal actions using the sequential Kalman filter. In: Proceedings of the IEEE international conference on image processing 2005, Genova, 14 September, Vol. 3, pp. 105–108. New York: IEEE.
10.
DongYFengSTaiL (2021) Three-dimensional target tracking strategy based on guidance laws and optimal information fusion. Transactions of the Institute of Measurement and Control43(7): 1560–1570.
11.
DuDZhouQQiN, et al. (2019) Learning to control a free-floating space robot using deep reinforcement learning. In: Proceedings of the 2019 IEEE international conference on unmanned systems (ICUS), Beijing, China, 17–19 October, pp. 519–523. New York: IEEE.
HaarnojaTZhouAAbbeelP, et al. (2018a) Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In: DyJKrauseA (eds). Proceedings of the 35th International Conference on Machine Learning, ICML Proceedings of the international conference on machine learning (PMLR), Stockholm, 10–15 July, pp. 1861–1870. Available at: https://proceedings.mlr.press/
14.
HaarnojaTZhouAHartikainenK, et al. (2018b) Soft actor-critic algorithms and applications. Available at: https://arxiv.org/abs/1812.05905
15.
HeYLiangBHeJ, et al. (2017) Non-cooperative spacecraft pose tracking based on point cloud feature. Acta Astronautica139: 213–221.
16.
HuJWangT (2018) Minimum base attitude disturbance planning for a space robot during target capture. Journal of Mechanisms and Robotics: Transactions of the ASME10(5): 051002.
LeiWFuHSunG (2022) Active object tracking of free floating space manipulators based on deep reinforcement learning. Advances in Space Research70(11): 3506–3519.
19.
LiJXuJZhongF, et al. (2020) Pose-assisted multi-camera collaboration for active object tracking. In: Proceedings of the 34th AAAI conference on artificial intelligence, New York, 7–12 February, Vol. 34, pp. 759–766. Washington, DC: Association for the Advancement of Artificial Intelligence (AAAI).
20.
LiangBChenZGuoM, et al. (2021) Space robot target intelligent capture system based on deep reinforcement learning model. Journal of Physics: Conference Series1848: 012078.
21.
LillicrapTPHuntJJPritzelA, et al. (2015) Continuous control with deep reinforcement learning. Available at: https://arxiv.org/abs/1509.02971
22.
LinCShiJLyuY, et al. (2022) Over-flight and standoff tracking of a ground target with a fixed-wing unmanned aerial vehicle based on a unified sliding mode guidance law. Transactions of the Institute of Measurement and Control44(2): 410–423.
23.
LuoWSunPZhongF, et al. (2018) End-to-end active object tracking via reinforcement learning. In: Proceedings of the 35th international conference on machine learning (PMLR), Stockholm, 10–15 July, pp. 3286–3295. Available at: https://proceedings.mlr.press/
24.
LuoWSunPZhongF, et al. (2019) End-to-end active object tracking and its real-world deployment via reinforcement learning. IEEE Transactions on Pattern Analysis and Machine Intelligence42(6): 1317–1332.
25.
NakamuraYMukherjeeR (1990) Nonholonomic path planning of space robots via bi-directional approach. In: Proceedings of the IEEE international conference on robotics and automation, Cincinnati, OH, 13–18 May, pp. 1764–1769. New York: IEEE.
26.
NenchevDNYoshidaK (1999) Impact analysis and post-impact motion control issues of a free-floating space robot subject to a force impulse. IEEE Transactions on Robotics and Automation15(3): 548–557.
27.
Nguyen-HuynhTCSharfI (2013) Adaptive reactionless motion and parameter identification in postcapture of space debris. Journal of Guidance, Control, and Dynamics36(2): 404–414.
28.
NotebornRBodinPLarssonR, et al. (2011) Flight results from the PRISMA optical line of sight based autonomous rendezvous experiment. In: Proceedings of the 4th international conference on spacecraft formation flying missions and technologies, Saint-Hubert, QC, Canada, 18–20 May, pp. 18–20. Available at: https://elib.dlr.de/74574/
29.
RisticBClarkDEGordonN (2013) Calibration of multi-target tracking algorithms using non-cooperative targets. IEEE Journal of Selected Topics in Signal Processing7(3): 390–398.
30.
RohmerESinghSPNFreeseM (2013) CoppeliaSim (formerly V-REP): A versatile and scalable robot simulation framework. In: Proceedings of the international conference on intelligent robots and systems (IROS), Tokyo, Japan, 3–7 November. Available at: https://www.coppeliarobotics.com/
31.
SchulmanJWolskiFDhariwalP, et al. (2017) Proximal policy optimization algorithms. Available at: https://arxiv.org/abs/1707.06347
32.
ShtarkTGurfilP (2017) Tracking a non-cooperative target using real-time stereovision-based control: An experimental study. Sensors17(4): 735.
33.
SunRGilesCL (2001) Sequence learning: From recognition and prediction to sequential decision making. IEEE Intelligent Systems16(4): 67–70.
34.
SutskeverIVinyalsOLeQV (2014) Sequence to sequence learning with neural networks. In: Advances in neural information processing systems 27: Proceedings of the 27th international conference on neural information processing systems, Montreal, QC, Canada, 8–13 December.
35.
ThienelJVan EepoelJSannerR (2006) Accurate state estimation and tracking of a non-cooperative target vehicle. In: Proceedings of the AIAA guidance, navigation, and control conference and exhibit, Keystone, CO, 21–24 August, p. 6802. Reston, VA: American Institute of Aeronautics and Astronautics (AIAA).
36.
UmetaniYYoshidaK (1989) Resolved motion rate control of space manipulators with generalized Jacobian matrix. IEEE Transactions on Robotics and Automation5(3): 303–314.
37.
WangMLuoJFangJ, et al. (2018) Optimal trajectory planning of free-floating space manipulator using differential evolution algorithm. Advances in Space Research61(6): 1525–1536.
38.
WangMLuoJZhengL, et al. (2020) Generate optimal grasping trajectories to the end-effector using an improved genetic algorithm. Advances in Space Research66(7): 1803–1817.
39.
WangSCaoYZhengX, et al. (2022) A learning system for motion planning of free-float dual-arm space manipulator towards non-cooperative object. Aerospace Science and Technology131: 107980.
40.
WangSZhengXCaoY, et al. (2021) A multi-target trajectory planning of a 6-DoF free-floating space robot via reinforcement learning. In: Proceedings of the 2021 IEEE/RSJ international conference on intelligent robots and systems (IROS), Prague, 27 September–1 October, pp. 3724–3730. New York: IEEE.
41.
WangZBapstVHeessN, et al. (2016) Sample efficient actor-critic with experience replay. Available at: https://arxiv.org/abs/1611.01224
42.
WuYHYuZCLiCY, et al. (2020) Reinforcement learning in dual-arm trajectory planning for a free-floating space robot. Aerospace Science and Technology98: 105657.
43.
XiMZhouYChenZ, et al. (2021) Anti-distractor active object tracking in 3D environments. IEEE Transactions on Circuits and Systems for Video Technology32: 3697–3707.
44.
XieYWuXInamoriT, et al. (2019) Compensation of base disturbance using optimal trajectory planning of dual-manipulators in a space robot. Advances in Space Research63(3): 1147–1160.
45.
YaoQ (2021) Adaptive fuzzy neural network control for a space manipulator in the presence of output constraints and input nonlinearities. Advances in Space Research67(6): 1830–1843.
46.
YoshidaKUmetaniY (1993) Control of space manipulators with generalized Jacobian matrix. In: XuYKanadeT (eds) Space Robotics: Dynamics and Control. Boston, MA: Springer, pp. 165–204.
47.
YuZ-WLiuX-FCaiG-P (2016) Dynamics modeling and control of a 6-DOF space robot with flexible panels for capturing a free floating target. Acta Astronautica128: 560–572.
ZhangFDuanGR (2018) Manipulator-actuated adaptive integrated translational and rotational stabilization for spacecraft in proximity operations with control constraint. International Journal of Control, Automation and Systems16(5): 2103–2113.
