After discussions among the reviewers, they all agreed that the paper has enough theoretical contributions. They were also very happy with rebuttal and the answers provided by the authors. Please incorporate the suggested comments and references in the final version.

Summary and Contributions: In this article, the authors present a new method in the field of multi-task Reinforcement Learning. While the method is not restricted to a certain domain, they investigate the method in the experimental part in the application domain of manipulation, using an existing manipulation task benchmark suite (Meta-World). The main issues with multi-task RL that the authors motivate in the introduction and use to motivate their method are: conflicting gradients and balancing optimisation between tasks. They address important issues in multi-task RL that typically hurt the performance gain that we expect in terms of data efficiency and final performance, reported in all major publications in the field. From a high level perspective, there are two main ideas in the paper.

The reviewers agreed that this is a reasonably well-written paper, on an important topic, with excellent empirical results. Given that the level of enthusiasm varied widely across reviewers, I'd recommend revising the final paper for more clarity, especially with respect to the novelty of the ideas.

It would be great to make it more accessible to a more general audience, as the ideas it contains are fairly intuitive at their core. One suggestion would be to include illustrative figures to convey the general intuition, for example for the case of the linear-Gaussian bandit, since the confidence sets have a natural geometric interpretation in terms of the variance of the posterior. The analyses of the examples given (linear bandits, MDPs, factored MDPs) essentially all follow a recipe made possible by the results relating the confidence bounds to the regret. Specifically, they are: 1) Construct a confidence interval based on the mutual information using the characteristics of the problem at hand (linearity/Gaussian noise assumptions for the bandit, specific forms of the prior for MDPs) 2) Bound the sum of the information gain. Combining these two then gives a regret bound.

The paper extends Russo and Van Roy (JMRL2016) work to provide information-theoretical analysis of Thompson sampling and UCB-like algorithms in more general setting. The three reviewers acknowledge the contributions, and the potential impact of connecting information-theoretical concepts to the design of algorithms. Reviewers have suggested ways to improve the manuscript. The authors should follow these directions, and in particular fix notations, include simulation results, and provide explanations about proofs when necessary. The contributions in this paper are "methodological", i.e., it proposes a framework to analyze the regret of certain algorithms.

Let me start with a global comment. I enjoyed very much reading this paper. I found it well written (apart from typos, and some English sentences constructions that are a bit heavy) and interesting. It is related to a modern sub-field of reinforcement learning: multi-agent learning, that lacks theory w.r.t. to single-agent RL. The paper introduces a mean-field analysis of a large population of agents playing simple symmetric matrix games against each others, so that, as the population gets large, each player effectively plays against a single "mean" player.

This paper introduces a mean-field model of multiagent Q-learning in repeated symmetric games. The model assumes that at each time step each agent plays symmetric games with m other randomly chosen agents, and considers the limit of n, m to infinity. Under these settings the authors have derived the Fokker-Planck equation governing the time evolution of the distribution of the agents' Q-values. The review scores exhibited quite a large split. Two reviewers rated this paper well above the threshold, whereas Reviewer #1 rated it negatively.

Although some techniques are analogous to previous work (which is not bad per se, as it allows to apply more general regularisers within previous frameworks such as soft-actor-critic with small changes only), this work differs significantly from previous work and yields new insights how to obtain sparse policies or not. Claims are supported by proofs and experiments confirm that considering more flexible regularizations can be beneficial in different tasks. There are some issues with the continuous time case, see the section on improvements for details. Further the authors claim that trigonometric and exponential functions families yield multimodal policies (line 287). However, it is not clear to me how this is different to say entropy regularisation, and why a softmax policy cannot have multiple modes (unless of course I parameterize the policy with a single Gaussian in the continuous case, but this is a different issue).

The paper contributes valuable insights to understanding the influence of regularization in policy optimization in MDPs. The reviewers reached a consensus about the paper's results having significant merit, and the paper can be accepted for presentation.

Additional Feedback: Update: thanks for the answer, it helped clarify some points. I think the proposed additions will improve the clarity of the paper. While providing a common theoretical ground for general utilities in RL is not a minor contribution by any means, I would have loved to find a discussion on how to build upon these results. Do authors think their work can be leveraged to develop more efficient algorithm in the context of RL with general utilities, or the intended outcome is a deeper understanding of the setting without particular practical upsides? 2. Where the Variational Policy Gradient approach stands in comparison with other policy optimization methods for (specific) general utilities, e.g.