LuckyMera: a modular AI framework for building hybrid NetHack agents

2.1 NetHack as AI testbed

As we will see more in depth in Section 2.2, the environment offered by NetHack was widely used to develop and test intelligent agent capable of playing the game. An interesting approach is the one proposed in [7], in which the authors present a solution to explore the levels of the game exploiting the concept of occupancy maps, especially popular in robotics. In RL research, the first example of usage of NetHack is gym_nethack [8, 6], which offers an interface to the game through a Gym [5] environment. However, in this case, the dynamics were heavily modified by removing several obstacles, resulting in a much simpler version of the game.

In this work, instead, we make deep use of the NetHack Learning Environment (NLE) [21]: a Gym environment that leaves the game mechanisms unchanged. It is designed to wrap the entire game, returning all the observation available from the game, i.e. the map of the level, the current agent’s statistics, the textual message showed to the user and information about the inventory. The environment has 93 available actions, divided in 16 movement actions and 77 command actions. NLE is of particularly interesting because it is able to combine a complex environment with a fast simulator, being extremely efficient and computationally lightweight. Since current architectures cannot win the game, MiniHack [27] was released: a framework defined on top of NLE, which proposes a set of simpler environments, together with the possibility to easily design new tasks. The tasks proposed can be mainly divided into navigation tasks, in which the agent has to reach a goal position, and skill tasks, which involve more complex abilities, such as using potions, selecting the appropriate armor and fighting more powerful monsters.

We were deeply inspired by the results obtained in the NeurIPS 2021 NetHack Challenge [15]. Our framework was developed using mainly the challenge task, but it is completely environment-independent, and it work well with all the tasks proposed in NLE and MiniHack.

2.2 First AI Bots for NetHack

From its initial release, there have been several bots addressing the problem of NetHack. One of the first able to achieve significant results is TAEB 4 , a modular framework for designing automatic and semi-automatic players. It uses the publish/subscribe paradigm to perform the communication among the different components; for the pathfinding task, it employs Dijkstra’s algorithm [11]. The first symbolic bot able to "ascend", i.e. win the game, was BotHack 5 . Its architecture is particularly noticeable for the accurate recognition of the kind of floor the agent is exploring, and the use of the A* algorithm [17] for the navigation tasks. Nonetheless, it was able to achieve these results mostly using an exploit present in older NetHack version, which is no longer applicable in the current game. The current best open-source NetHack bot is AutoAscend 6 , winner of the NeurIPS 2021 NetHack Challenge. It implements a set of high-level strategies, each handling a specific behavior and wrapping multiple actions, and selects one based on its priority.

Although these approaches are able to obtain good results at the game, none of them offers a valid research platform, as we do with LuckyMera. Their goal was to create performance-oriented agents to win the game, while our main objective is to provide a development-oriented framework, to train, integrate and test neuro-symbolic approaches.

2.3 Imitation learning approaches

Imitation Learning is a Machine Learning technique in which the agent, to learn an intelligent behavior, instead of relying on the interaction with the environment, is provided with a set of demonstrations from an expert [31]. The agent’s objective is to mimic the expert’s actions, hopefully achieving an optimal policy, following a form of Supervised Learning. The dataset contains trajectories of experiences, made of state-action pairs; in particular, the trajectories will be in the form of τ = { ( s 0 , a 0 * ) , ( s 1 , a 1 * ) , … , ( s n , a n * ) } where an action a i * is assumed to be optimal in a given state s _i . It is critical to notice that, when performing Imitation Learning, the agent should not copy the expert’s behavior unconditionally; instead, it should extract key information from the trajectories, being able to generalize and achieve good performance also in states never seen before.

One of the simplest Imitation Learning algorithms is Behavioral Cloning (BC): given a state-action pair ( s t , a t * ) , the objective is to learn a policy π by minimizing a loss function L ( a t * , π ( s ) ) , assuming that the pairs are i.i.d., which means that the pairs are mutually independent and have the same probability to appear. BC has been shown to achieve good results especially in environments with relatively small state space, so that it can be covered for the most part by the expert’s demonstrations, e.g. autonomous driving [13]. Nonetheless, in most cases it can be quite problematic due to the i.i.d. assumption. An improvement of BC is DAgger [26], that employs an iterative process in which it first performs Supervised Learning to learn a policy, like in BC. It then uses it to produce observations, queries the expert on those observations, and integrates the dataset with these new demonstrations. A different approach to Imitation Learning is represented by Inverse Reinforcement Learning [1], in which the idea is to learn the reward function by observing the demonstrations from the expert, and then use it to find the optimal policy with Reinforcement Learning algorithms. NetHack is particularly convenient to perform Imitation Learning, thanks to the NetHack Learning Dataset [16]. It collects both state transitions from human games, and state-action trajectories generated by the winner of the NetHack Challenge 2021, AutoAscend.

We will review in detail the implementation of the Behavioral Cloning algorithm we offer in LuckyMera in Section 3.2.4.

2.4 Reinforcement learning approaches

Reinforcement Learning algorithms are usually tested in simulated environments, like games. RL approaches have shown superhuman capabilities in classical games, such as Go [29] and Chess [28]; furthermore, there were also works on more complex, multiplayer games, like StarCraft II [30] and Dota 2 [3]. One of the most influential benchmark for Reinforcement Learning agents is the Arcade Learning Environment [2]. It offers an interface to hundreds of video games from the Atari 2600 game console, and it has been widely used in RL research as a testbed to experiment with different approaches [24]. However, the environments proposed are too simple and not adequate for current RL algorithms. Nowadays, AI research needs more challenging benchmarks, to test more elaborate capabilities in complex environments; in this sense, NetHack is perfect to drive the research in the next years. Furthermore, the Arcade Learning Environment simply provides an interface to the games, while LuckyMera builds on top of the game interface to offer a modular framework for designing AI agents.

A comparable framework to our work is OpenSpiel [22], which is a collection of game environments and algorithms, including both search/planning strategies and reinforcement learning, with the possibility to add new games and methods. Their objective is to provide a set of simple and traditional games, enabling researchers to assess the same algorithm across multiple environments. In contrast, we focus on a single yet complex environment, which requires different skills, releasing a framework mainly oriented towards the definition of modular architectures.

Several studies on Reinforcement Learning were conducted on the NetHack environment. The MiniHack suite was used to test the E3B algorithm [18], a method for defining intrinsic exploration bonuses based on learned embeddings of previous states. Chester et al. [10] proposed a hybrid approach, using symbolic planning for low-level actions, and Reinforcement Learning to train a meta-controller; they show that this method surpasses the baseline algorithms in a custom MiniHack environment. Powers et al. [25] presented CORA, a platform for Continual Reinforcement Learning, providing MiniHack as one of the benchmark environments. Using the MiniHack tasks from CORA, Kessler et al. [19] studied a task-agnostic, model-based method for Continual Reinforcement Learning, showing it to be a strong baseline compared to state-of-the-art approaches.

LuckyMera is instead an approach-agnostic research platform; it can be expanded to test any Artificial Intelligence method or paradigm, including Reinforcement Learning ones. It is particularly convenient because of the possibility to train targeted skills, tackling a more feasible problem, and then to integrate them with the other modules offered by the framework.

3.1 Design of the Agent

The LuckyMera framework is released together with the implementation of an AI agent, designed following the modular structure, that represents a useful baseline for further studies. At the highest level of abstraction, the agent’s strategy simply consists of the iterative execution of the highest-priority plannable skill in the current state; the user can easily set the priority of each skill via a configuration file. In Figure 2 there is a high-level representation of the system execution flow.

OPEN IN VIEWER

Fig. 2

Flowchart of the behavior of the LuckyMera agent. Essentially, it iterates through planning and execution. During skill planning, it gets the highest priority skill that can be planned; then, this skill is actually executed.

The agent’s main execution flow is made by the iteration, throughout the duration of the game, of the two principal phases: skill-planning and skill-execution. The planning of a skill begins with the analysis of the game state, derived from the NLE observations. During this phase, the agent verifies if the skill can actually be executed, i.e. its preconditions are satisfied, and performs some preliminary steps, depending on the nature of the skill itself. The module implementing a skill should provide the planning method, in compliance with the framework’s main component interface. Thanks to this abstraction, the system is perfectly compatible out-of-the-box with any symbolic and neural skill implementation, and in general with any AI module; we will discuss this feature more in detail in Section 3.2.2. Once the planning of a skill succeeds, the agent performs the skill-execution phase. Each skill provides the implementation of a series of actions needed to perform its plan.

Our framework offers also the possibility to easily define custom modules. To leverage this extensibility property, we provide simple class templates, that enable the definition of any specific action. An in-depth discussion about the extensibility property of LuckyMera is present in Section 3.2.1 .

In addition to the high-level modules, the architecture also provides low-level solutions for interacting with the game. In particular, it is based on the GameWhisperer class, which deals with the interaction with the NLE framework, offering several refinements to the low-level observations; it also defines methods to encapsulate multiple NetHack commands into single, more abstract atomic commands. On the other hand, to handle the navigation in the game world, the system leverages on the DungeonWalker class. It offers some useful functionalities for pathfinding and exploration. To do so, it employs the A* algorithm [17], using the octile distance heuristic [4] to take into account also diagonal movements.

Here we will discuss some of the main features that LuckyMera offers, which make it an integrated framework for quick testing of new approaches, automatic creation of labeled trajectories and training of Machine Learning models. During the discussion, we will also present some possible applications of the framework, namely a neuro-symbolic approach and our baseline agent LuckyMera-v1.0.

3.2.2 Skills Integration

The main component of the framework offers a simple, straightforward interface, making it easy to integrate skill modules. Each module should inherit from one of the classes representing the Skill concept and, by doing so, define its own planning and execution methods. All the actions that a LuckyMera agent can perform are defined as skills, and are executed following a priority list. The strategy the agent follows is determined by the priority assigned to each module. In the framework, the order of the modules can be easily changed through a configuration file. In our tests, we let the agent adopt a cautious strategy, in order to maximize the score⁷. In particular, the top-priority actions are the ones that can help the bot overcoming dangerous situations, like praying to receive resources, engraving the name “Elbereth” to scare enemies and run. After those actions, the agents checks if it can fight nearby monsters. Otherwise, if it is in a safe circumstance, it can explore the unseen parts of the dungeon, or search for hidden rooms and corridors. The complete list of the currently implemented skills is available in Appendix 5.1. It includes both the symbolic skills and the ones coming from the integration of the neural modules.

Besides these skills, it is possible to integrate any external module compliant with the interface, as presented in Section 3.2.1. In fact, the framework allows for the import of any given model, so that new actions — or new strategies for already defined operations — can be implemented, or trained neural models can be used to perform specific tasks. As an example, the framework was tested with the integration of a neural RL agent, trained using the IMPALA algorithm [12], with the implementation from TorchBeast [20]. Since the task of playing the entire game of NetHack is too difficult for current RL approaches, the training was executed on some MiniHack environments, which offers more controlled and feasible challenges. The environments selected are Room-Ultimate-15x15 and KeyRoom-S5, represented in Figure 3. In this case, the game map is depicted using the pixel observation from MiniHack, which is offered in addition to the standard ASCII interface. To improve the performance of the pure neural agents, the trained models were then integrated with some prior knowledge about the game, in the form of a set of simple, generic rules. More details about the symbolic rules are available in Section 4.1.

OPEN IN VIEWER

Fig. 3

MiniHack environments used for the training of the RL agent.

3.2.3 Trajectory Saving

The architecture we propose comes with the possibility to save the experiences of the bot with the environment, in the form of trajectories of state-action pairs. Given the performance of a LuckyMera bot, its behavior is meaningful and valid in the context of the game, and it could be used as an expert, e.g. in Imitation Learning applications. Within the framework, it is possible to exploit the capabilities of the bot to define a dataset of experiences, which is automatically labeled with the actual action performed by the agent. The trajectory saving mechanism is independent of the type of observation: it is possible to store any element present in the observation space defined by NLE, by defining them at runtime. The framework integrates also the NLE Language Wrapper [14], which translates the non-language observations from NetHack, e.g. glyphs and chars, into corresponding language representations (Figure 4). These language observations can be selected in the saving process as well, and could be useful in the fine-tuning of language models.

OPEN IN VIEWER

Fig. 4

One level of NetHack viewed with different representations. The top image is the standard ASCII interface, while the bottom image shows the language representation provided by the NLE Language Wrapper.

3.2.5 Example of Usage: LuckyMera-v1.0

LuckyMera comes with a set of pre-defined modules, to showcase one of the possible concrete use-cases of the framework. These modules implement some basic actions and the relative reasoning processes, that consider the facts known by the agent and its surroundings to select the best skill to perform. Such actions allow the agent to interact with the NetHack environment and to progress in the game. Indeed, there are modules designed to explore the dungeon, other ones to deal with the enemies present in the game, and other skills for actions such as eating or praying. The complete list of the skills released together with the LuckyMera framework is reported in Table 3.

These skills specify the behavior of a possible agent, referred to as LuckyMera-v1.0. This implementation mainly serves as a practical demonstration of the usage of the framework, emphasizing the intrinsic modularity of our architecture, rather than positioning itself as a fully-fledged AI agent. With the LuckyMera-v1.0 agent, our goal is to establish and release a solid baseline — a starting point for further developments — without the aspiration to surpass the state-of-the-art. Nonetheless, we assessed our approach against the current best-performing solutions, achieving comparable results. Further details regarding this evaluation can be found in Section 4.

OPEN IN VIEWER

Fig. 5

Results of the ablation studies performed on the framework.

4.1 Reinforcement learning approach results

We integrated a Reinforcement Learning approach, based on the IMPALA algorithm. As testing environments, we used the Room-Ultimate-15x15 and KeyRoom-S5 tasks from MiniHack. Initially, we tested the pure neural approach, then we integrated it with some prior knowledge about the problem, in the form of basic symbolic rules, expressed using first-order logic. Those rules were used to increase the probability of crucial actions, like attacking nearby monsters and moving towards the key. In particular, the rules we employed are showed in Table 1.

In Figure 5a, results for the two environments considered are reported. It is clear that, in both cases, the integration of rules led to a increase in the agent’s performance.

The imitation learning approach implemented in LuckyMera is based on the Behavioral Cloning algorithm. This method was tested using the Room-5x5 environment from MiniHack. It was selected for its low complexity, which guarantees relatively fast training processes and low space occupation to store the dataset. The model was trained for five epochs, and then the learned policy was evaluated in a different instance of the environment. In Figure 5b we present a comparison between the performance of a random agent and a trained model. It is clear that the trained agent is always able to solve the task, being also close to the optimal behavior.

4.3 LuckyMera-v1.0 Agent Baseline

As a final validation of our implementation, we designed a LuckyMera agent composed of multiple skills (more details in Appendix 5.1) and tested against the most challenging NetHackChallenge-v0 environment, which represents the complete NetHack game. This was done to compare our approach with the participants of the NeurIPS 2021 NetHack Challenge [15], which represent the current state-of-the-art models tackling the game of NetHack.

Our agent was able to achieve an average score of 1046.96 and a median of 817. In Figure 6, the results of LuckyMera are put against the highest scoring teams from the challenge. The agent is able to virtually reach the 6^th position, overcoming more than 80% of the competitors. The challenge ran for 144 days, with 46 participating teams and 631 overall submissions⁸.

OPEN IN VIEWER

Fig. 6

Results obtained by a LuckyMera agent, compared with the state-of-the-art bots from the NeurIPS NetHack Challenge 2021.