self-play posterior sampling algorithm
A Self-Play Posterior Sampling Algorithm for Zero-Sum Markov Games
Xiong, Wei, Zhong, Han, Shi, Chengshuai, Shen, Cong, Zhang, Tong
While there is a long line of for Markov games (MGs) almost exclusively related works on the theoretical understanding of singleagent build on the "optimism in the face of uncertainty" RL with general function approximation (Jiang et al., (OFU) principle. This work focuses on a different 2017; Sun et al., 2019; Wang et al., 2020; Jin et al., 2021a; approach of posterior sampling, which is Du et al., 2021; Dann et al., 2021), the theory of MARL celebrated in many bandits and reinforcement with general function approximation is substantially less learning settings but remains under-explored for explored. In this paper, we aim to explore this topic in MGs. Specifically, for episodic two-player zerosum the context of two-player zero-sum Markov games (MGs) MGs, a novel posterior sampling algorithm (Shapley, 1953; Littman, 1994). is developed with general function approximation. Theoretical analysis demonstrates that the The goal of learning in a two-player zero-sum MG is to posterior sampling algorithm admits a T -regret learn the Nash equilibrium at which the policy of each bound for problems with a low multi-agent decoupling player maximizes her own cumulative rewards, provided coefficient, which is a new complexity that the policies of other agents are fixed.