In a tactical wargame, the decisions of the artificial intelligence (AI) commander are critical to the final combat result. Due to the existence of fog-of-war, AI commanders are faced with unknown and invisible information on the battlefield and lack of understanding of the situation, and it is difficult to make appropriate tactical strategies. The traditional knowledge rule-based decision-making method lacks flexibility and autonomy. How to make flexible and autonomous decision-making when facing complex battlefield situations is a difficult problem. This paper aims to solve the decision-making problem of the AI commander by using the deep reinforcement learning (DRL) method. We develop a tactical wargame as the research environment, which contains built-in script AI and supports the machine–machine combat mode. On this basis, an end-to-end actor–critic framework for commander decision making based on the convolutional neural network is designed to represent the battlefield situation and the reinforcement learning method is used to try different tactical strategies. Finally, we carry out a combat experiment between a DRL-based agent and a rule-based agent in a jungle terrain scenario. The result shows that the AI commander who adopts the actor–critic method successfully learns how to get a higher score in the tactical wargame, and the DRL-based agent has a higher winning ratio than the rule-based agent.
CaicedoJCLazebnikS. Active object localization with deep reinforcement learning. In: O’ConnerL (ed.) IEEE international conference on computer vision, Santiago, Chile, 7–13 December 2015, pp.2488–2496. NW Washington, DC: ICCV.
2.
OhJGuoXLeeH, et al. Action-conditional video prediction using deep networks in Atari games. In: CortesC (ed.) International conference on neural information processing systems, Montreal, Canada, 7–12 December 2015, pp.2863–2871. Cambridge, MA: MIT Press.
3.
SilverDHuangAMaddisonCJ, et al. Mastering the game of Go with deep neural networks and tree search. Nature2016; 529: 484–489.
4.
LillicrapTPHuntJJPritzelA, et al. Continuous control with deep reinforcement learning. Comput Sci2015; 8: A187.
5.
GuSLillicrapTSutskeverI, et al. Continuous deep Q-learning with model-based acceleration. In: LangfordJ (ed.) International conference on machine learning, New York, USA, 19–24 June 2016, pp.2829–2838. New York, NY: PMLR.
6.
GuoRJSpragueK.Replication of human operators’ situation assessment and decision making for simulated area reconnaissance in wargames. J Defens Model Simulat2016; 13: 213–225.
7.
BojarskiSCongdonCB. REALM: a rule-based evolutionary computation agent that learns to play Mario. In: IEEE Computer Society (ed.) Computational intelligence & games, Copenhagen, Denmark, 18–21 August 2010, pp.83–90. NW Washington, DC: IEEE.
8.
PalSKShiuSCK. Foundations of soft case-based reasoning. Hoboken, NJ: John Wiley, 2004.
9.
AmildeR. Modeling the behavior of a hierarchy of command agents with context-based reasoning. J Defens Model Simulat2018; 15: 369–381.
10.
PokahrABraubachLLamersdorfW, et al. Jadex: a BDI reasoning engine. Multiagent Syst Artif Soc2005; 15: 149–174.
11.
ShenSO’HareGMPCollierR. Decision-making of BDI agents, a fuzzy approach. In: WeiD (ed.) International conference on computer & information technology, Wuhan, China, 14–16 September 2004, pp.1022–1027. NW Washington, DC: IEEE.
12.
FoxWP.Applied game theory to improve strategic and tactical military decisions. J Defens Manag2016; 6: 147.
KrizhevskyASutskeverIHintonGE. ImageNet classification with deep convolutional neural networks. In: BartlettP (ed.) Advances in neural information processing systems, Nevada, USA, 3–6 December 2012, pp.1097–1105. Red Hook, NY: Curran Associates, Inc.
15.
SimonyanKZissermanA. Very deep convolutional networks for large-scale image recognition. Comput Sci2014; 4: 1–14.
16.
GirshickRDonahueJDarrellT, et al. Region-based convolutional networks for accurate object detection and segmentation. IEEE Trans Pattern Anal Mach Intell2015; 38: 142–158.
17.
LiuWAnguelovDErhanD, et al. SSD: single shot multibox detector. In: LeibeBMatasJSebeN, et al. (eds) European conference on computer vision, Amsterdam, The Netherlands, 11–14 October 2016, pp.21–37. Cham, Switzerland: Springer.
18.
WadeBM. Creating surrogate models for an air and missile defense simulation using design of experiments and neural networks. J Defens Model Simulat2019; 9: 1–12.
19.
AmatoCShaniG. High-level reinforcement learning in strategy games. In: van der HoekWKaminkaG (eds) International conference on autonomous agents & multiagent systems, Toronto, Canada, 10–14 May 2010, pp.78–82. Bologna, Italy: AAMAS.
20.
GlavinFGMaddenMG. Adaptive shooting for bots in first person shooter games using reinforcement learning. IEEE Trans Computat Intell AI Game2014; 7: 180–192.
21.
HosuI-ARebedeaT.Playing Atari games with deep reinforcement learning and human checkpoint replay. Los Alamos, USA: arxiv.org, 2016.
22.
KröseBJA. Learning from delayed rewards. Robot Auton Syst1995; 15: 233–235.
23.
PengPPangLYuanY, et al. Continual match based training in Pommerman: technical report. CoRR2018; abs/1812.07297.
24.
WangJZhuTLiH, et al. Belief-state Monte-Carlo Tree Search for Phantom Go. IEEE Trans Computat Intell2017; 10: 139–154.
25.
LiskowskiPJaśkowskiWKrawiecK. Learning to play Othello with deep neural networks. IEEE Trans Game2018; 10: 354–364.
26.
TakadaKIizukaHYamamotoM. Reinforcement learning to create evaluation and policy functions using Minimax Tree Search in Hex. IEEE Trans Game2019; 12: 63–73.
27.
VinyalsOEwaldsTBartunovS, et al. StarCraft II: a new challenge for reinforcement learning. Los Alamos, USA: arxiv.org, 2017.
28.
ArulkumaranKCullyATogeliusJ. AlphaStar: an evolutionary computation perspective. In: IbanezML (ed.) The genetic and evolutionary computation conference, Prague, Czech Republic, 13–17 July 2019, pp.314–315. New York, USA: GECCO.
29.
TianYGongQShangW, et al. ELF: an extensive, lightweight and flexible research platform for real-time strategy games. Red Hook, NY: Curran Associates Inc., 2017.
30.
WangBWangJZhangB, et al. Leader-follower consensus for multi-agent systems with three-layer network framework and dynamic interaction jointly connected topology. Neurocomputing2016; 207: 231–239.
31.
KandathHSenthilnathJSundaramS. Mutli-agent consensus under communication failure using Actor-Critic Reinforcement Learning. In: SundaramS (ed.) 2018 IEEE symposium series on computational intelligence (SSCI), Bangalore, India, 18–21 November 2018, pp. 1461–1465. NW Washington, DC: IEEE.
SuttonRSMcAllesterDSinghS, et al. Policy gradient methods for reinforcement learning with function approximation. In: CortesC (ed.) Proceedings of the 12th international conference on neural information processing systems, Denver, USA, 1 October 2000, pp.1057–1063. Cambridge, MA: MIT Press.
34.
DegrisTPilarskiPMSuttonRS. Model-free reinforcement learning with continuous action in practice. In: SamadT (ed.) 2012 American control conference (ACC), Montreal, Canada, 27–29 June 2012, pp.2177–2182. Denver, USA: ACC.
