Weaknesses: - The paper's narrative is based around POMDPs, but the experimental evaluation does not really stress the capability of the method in that respect. Evaluation is done on pixel-based control, which is PO of course, but we have know that a lagged observation of a few time-steps can make the state fully observable quickly. Hence, we do not know how the method fares in environments where the state uncertainty has to be actively reduced by the agent. Therefore I think the paper overstates the results. It is easy to get out of this, however, since one can just drop the POMDP claim. For me personally (and the optimal control community) it is obvious that we want some kind of state estimation when we use control, as most–if not all–practical problems are PO.

The method targets a model-based approach to solve POMDPs with high-dimensional observation spaces. This problem is tackle by learning jointly about the dynamics of the POMDP and the optimal policy by maximum likelihood using an "RL as inference" type objective. In more detail, the latent space transitions are predicted by an inference model that is trained to maximise an evidence lower bound. The reviewers are mostly positive about the paper. They mention the theoretical soundness of the approach and the quality of writing as well as the empirical set-up and usefulness of the ablations.

The paper is very well written and organized and its contributions are quite original as it proposes a novel coarse-ID method for robust model-based reinforcement learning in which both exploration AND exploitation are optimized jointly (which was not the case in previous similar works). The method proposed to solve the robust Reinforcement Learning problem is all the more original as it does not rely on Stochastic Dynamic Programming, but rather on Semidefinite Programming. Concerning clarity, the only element that is not clear for me is related to equation (1) in page 2: do you consider in the system model some uncertainty in the measurements of the states x? For example, it is said in the supplemental material that the velocity of the servo-motor of your second experiment is estimated using a high pass-filter, and is hence not perfectly known. If it is modeled, is it included in the process noise w or how do you deal with it?

The paper presents a new technique for robust optimization and balanced exploration in LQR problems. The technique is quite innovative since it leverages semidefinite programming instead of dynamic programming. This is an important algorithmic contribution with solid theory. For the empirical evaluation, the authors are expected to include the new experiments and running times mentioned in the rebuttal. Overall, this is very nice work.

Additional Feedback: some details: p1 - define MOBA in the abstract - such as multi-agent, grammar - attention already reference(s)? Why is this no longer computationally feasible exactly? - Note that there still lacks a ... grammar - [Our] AI [system] achieved ... - How many players is the "top 0.04%"? The paper requires a thourough proofreading effort to make it publishable in the NeuRIPS proceedings. How important is this incorporated expert knowledge? "s.t." is commonly used as definition or optimization constraint.

This paper demonstrates an application of RL and search to a challenging MOBA game-playing task, leading to AI agents able to defeat top professional human players. Three out of four reviewers consider that although this is an application-oriented paper with a strong engineering focus, it is still relevant enough for publication at NeurIPS. Only R2 is advocating for rejection, based essentially on the lack of scientific novelty. I believe that such impressive large scale applications of RL are well worth pushing forward and I am thus recommending acceptance. The general algorithms being used may not be novel, but their instantiation to solve this specific task largely is.

Value-function-based methods have long played an important role in reinforcement learning. However, finding the best next action given a value function of arbitrary complexity is nontrivial when the action space is too large for enumeration. We develop a framework for value-function-based deep reinforcement learning with a combinatorial action space, in which the action selection problem is explicitly formulated as a mixed-integer optimization problem. As a motivating example, we present an application of this framework to the capacitated vehicle routing problem (CVRP), a combinatorial optimization problem in which a set of locations must be covered by a single vehicle with limited capacity. On each instance, we model an action as the construction of a single route, and consider a deterministic policy which is improved through a simple policy iteration algorithm. Our approach is competitive with other reinforcement learning methods and achieves an average gap of 1.7% with state-of-the-art OR methods on standard library instances of medium size.

This paper considers learning in mean-field games (MFG). MFGs take the limit of an infinite number of agents, which are considered indistinguishable. Based on a motivating example consisting of a repeated Ad auction problem, the authors introduce a "general" mean-field game (GMFG), a model-free version of the standard MFG. The authors revisit standard Q-Learning and a soft version of it, and provide convergence and complexity results of such an algorithm. These methods are compared numerically on the auction problem together with a recently proposed approach and show better performance.

Weaknesses: My primary concern is that: 0. The paper seems to propose two ideas: 1) measuring distance between distributions as an expected squared difference between empirical characteristic functions evaluated at points sampled according to some adversarially learned distribution T; 2) the reciprocal training of adversarial autoencoders, i.e. adversarially aligning embeddings of X and Y, while making sure that these embeddings follow the Gaussian distribution and minimize the reconstruction loss. I wonder whether the impact of these two design choices can be evaluated independently: 1) seeing how direct minimization of C_T(X, g(Z)) wrt g performs compared to the model with a dedicated encoder/critic; 2) replacing C_T in Algorithm 1 with MMD / Sliced Wasserstein Distance or another statistical distance (moreover, distance to a Gaussian can often be estimated in closed form); does Lemma 4 hold for other statistical distances? And there are some things that I must have misunderstood. In general, authors discuss in great details possible interpretations of phase and amplitude components of CFs, but cram a lot of content critical to proper understanding of the final model on the first half of page 6. For example, in lines 214-215: "we further re-design the critic loss by finding an anchor as C(f(Y),Z) C(f(X),Z)" - it is still not clear to me what "anchors" authors are referring to.

The reviewers have reached a consensus that this work constitutes a novel and interesting extension of previous work on learning implicit models using GANs and integral probability metrics. The problems identified in the first round of reviewing were largely addressed in the author response, and I therefore can comfortably recommend accepting this paper.