A Finite-Sample Analysis of Payoff-Based Independent Learning in Zero-Sum Stochastic Games

Recent years have seen remarkable successes of reinforcement learning (RL) in a variety of applications, such as board games (Silver et al., 2017), autonomous driving (Shalev-Shwartz et al., 2016), city navigation (Mirowski et al., 2018), and fusion plasma control (Degrave et al., 2022). A common feature of these applications is that there are multiple decision-makers interacting with each other in an unknown environment. While empirical successes have shown the potential of multi-agent reinforcement learning (MARL) (Busoniu et al., 2008; Zhang et al., 2021a), the training of MARL agents largely relies on heuristics and parameter-tuning, and is not always reliable. In particular, many practical MARL algorithms are heuristically extended from their single-agent counterparts and lack theoretical guarantees. A growing literature seeks to provide theoretical insights to substantiate the empirical success of MARL and inform the design of efficient, and provably convergent algorithms.