Modeling multi-agent systems requires understanding howagents interact. Such systems are often difficult to model because they can involve a variety of types ofinteractions that layer together todriverich social behavioral dynamics.

To test the efficiency of each component, we remove them separately (LG-ODE-no att,7 LG-ODE-no PE) and find the performances drop. This suggests that distinguishing the importance of nodes w.r.t8 time and incorporating temporal information via learnable positional encoding would benefit model performance.9 ForEqn2, we adopt the GNN model in[2]tocapture the interaction among agents.

In contrast, the mathematics needed to analyze such schemes is what forms the focus in Stochastic Approximation (SA) theory [2, 4]. More generally, SA refers to an iterative scheme that helps find zeroes or optimal points of a function, for which only noisy evaluationsarepossible.

Most existing results only focus on online settings, in which agents can interact with the environment during training.

Thenfor N ( log (de H/ ))sufficientlylarge, withprobability1 , wehave (b ;!)= O

We study online learning in two-player uninformed Markov games, where the opponent's actions and policies are unobserved. In this setting, Tian et al. (2021) show that achieving no-external-regret is impossible without incurring an exponential dependence on the episode length $H$. They then turn to the weaker notion of Nash-value regret and propose a V-learning algorithm with regret $O(K^{2/3})$ after $K$ episodes. However, their algorithm and guarantee do not adapt to the difficulty of the problem: even in the case where the opponent follows a fixed policy and thus $O(\sqrt{K})$ external regret is well-known to be achievable, their result is still the worse rate $O(K^{2/3})$ on a weaker metric. In this work, we fully address both limitations. First, we introduce empirical Nash-value regret, a new regret notion that is strictly stronger than Nash-value regret and naturally reduces to external regret when the opponent follows a fixed policy. Moreover, under this new metric, we propose a parameter-free algorithm that achieves an $O(\min \{\sqrt{K} + (CK)^{1/3},\sqrt{LK}\})$ regret bound, where $C$ quantifies the variance of the opponent's policies and $L$ denotes the number of policy switches (both at most $O(K)$). Therefore, our results not only recover the two extremes -- $O(\sqrt{K})$ external regret when the opponent is fixed and $O(K^{2/3})$ Nash-value regret in the worst case -- but also smoothly interpolate between these extremes by automatically adapting to the opponent's non-stationarity. We achieve so by first providing a new analysis of the epoch-based V-learning algorithm by Mao et al. (2022), establishing an $O(ηC + \sqrt{K/η})$ regret bound, where $η$ is the epoch incremental factor. Next, we show how to adaptively restart this algorithm with an appropriate $η$ in response to the potential non-stationarity of the opponent, eventually achieving our final results.

We generate 128,000 images as agents' observations using python's matplotlib library Hunter [2007] V ariational autoencoder [Kingma and Welling, 2014] is used to encode the observations. Input is flatted 30,720-dimensional vector (32 by 320 by 3). Both encoder and decoder have one hidden layer with the dimension size being 1,024. The output (communication message) is a 10-dimensional vector. ReLU is used as the activation function.