Multi-agent Continual Coordination via Progressive Task Contextualization

Cooperative Multi-agent Reinforcement Learning (MARL) has attracted prominent attention in recent years [1], and achieved great progress in multiple aspects, like path finding [2], active voltage control [3], and dynamic algorithm configuration [4]. Among the multitudinous methods, researchers, on the one hand, focus on facilitating coordination ability via solving specific challenges, including non-stationarity [5], credit assignment [6], and scalability [7]. Other works, on the other hand, investigate the cooperative MARL from multiple aspects, like efficient communication [8], zero-shot coordination (ZSC) [9], policy robustness [10], etc. A lot of methods emerge as promising solutions for different scenarios, including policy-based ones [11,12], value-based series [13,14], and many other variants, showing remarkable coordination ability in a wide range of tasks like SMAC [15]. Despite the great success, the mainstream cooperative MARL methods are still restricted to being trained in one single task or multiple tasks simultaneously, assuming that the agents have access to data from all tasks at all times, which is unrealistic for physical agents in the real world that can only attend to one task at a time. Continual Reinforcement Learning plays a promising role in the mentioned problem [16], where the agent aims to avoid catastrophic forgetting, as well as enable knowledge transfer to new tasks (a.k.a.