YOLO-MARL: You Only LLM Once for Multi-agent Reinforcement Learning

Advancements in deep multi-agent reinforcement learning (MARL) have positioned it as a promising approach for decision-making in cooperative games. However, it still remains challenging for MARL agents to learn cooperative strategies for some game environments. Recently, large language models (LLMs) have demonstrated emergent reasoning capabilities, making them promising candidates for enhancing coordination among the agents. However, due to the model size of LLMs, it can be expensive to frequently infer LLMs for actions that agents can take. In this work, we propose You Only LLM Once for MARL (YOLO-MARL), a novel framework that leverages the high-level task planning capabilities of LLMs to improve the policy learning process of multi-agents in cooperative games. Notably, for each game environment, YOLO-MARL only requires one time interaction with LLMs in the proposed strategy generation, state interpretation and planning function generation modules, before the MARL policy training process. This avoids the ongoing costs and computational time associated with frequent LLMs API calls during training. Moreover, the trained decentralized normal-sized neural network-based policies operate independently of the LLM. We evaluate our method across three different environments and demonstrate that YOLO-MARL outperforms traditional MARL algorithms. Multi-agent reinforcement learning (MARL) algorithms have proven to be a powerful framework for addressing complex decision-making problems in multi-agent systems. With the rising applications of multi-agent systems, such as mobile robots in warehouses and games requiring complex reasoning and strategy, it is increasingly crucial for individual agents to learn, cooperate, or compete in dynamic environments without a centralized decision-maker (Papoudakis & Schäfer, 2021). In cooperative Markov games, agents are trained to coordinate their actions to maximize the joint rewards. However, existing MARL algorithms face challenges in learning distributed policies for cooperative games.