Multi-Agent Reinforcement Learning in Time-varying Networked Systems

In comparison to single-agent reinforcement learning (RL), MARL poses many challenges, chief of which is scalability [49]. Even if each agent's local state/action spaces are small, the size of the global state/action space can be large, potentially exponentially large in the number of agents, which renders many RL algorithms such as -learning not applicable. A promising approach for addressing the scalability challenge that has received attention in recent years is to exploit application-specific structures, e.g., [16, 32, 35]. A particularly important example of such a structure is a networked structure, e.g., applications in multi-agent networked systems such as social networks [6, 24], communication networks [44, 52], queueing networks [31], and smart transportation networks [51]. In these networked systems, it is often possible to exploit static, local dependency structures [1, 14, 15, 29], e.g., the fact that agents only interact with a fixed set of neighboring agents throughout the game. This sort of dependency structure often leads to scalable, distributed algorithms for optimization and control [1, 14, 29], and has proven effective for designing scalable and distributed MARL algorithms, e.g.