The reviewers enjoyed the paper, although expressed some concerns regarding the novelty (it combines a number of existing ideas). Still, the combination does result in a clear performance increase on a small set of Atari 2600 games. In the discussion the reviewers appreciated the additional experiments provided in the rebuttal, and reiterated the need for the final version of this paper to incorporate these and to be cleaned up. I also want to encourage the authors to report the performance of their algorithm on a larger number of Atari 2600 games -- in particular, how were these 6 games selected? Was there an unconscious bias in this selection?

The paper proposes a method for improving convergence rates of RL algorithms when one has access to a set of state-only expert demonstrations. The method works by modifying the given MDP so that the episode terminates whenever the agent leaves the set of states that had high-probability under the expert demonstrations. The paper then proves an upper bound on the regret incurred using their algorithm (as compared to the expert) in terms of the regret for the RL algorithm that is used to solve the modified MDP. The paper presents a set of experiments showing that the proposed mechanism can effectively strike a tradeoff between convergence rate and optimality. The clarity of the exposition is quite high, and the paper is easy to follow.

The proof of your theory lacks discussion of POMDP settings. Although the framework in focused in solving the Dec-POMDP problem, most parts of the proof are under MDP setting. But there is no more discussion on that phenomenon. The use of weighting is not that convinced. In Section 6.2.3, the performance of the Weighted QMIX method is unacceptable.

I want to thank the authors for preparing the detailed rebuttal. This paper was discussed among all the reviewers during the post-rebuttal discussion phase. Also, given the borderline scores, we requested an additional emergency reviewer for this paper. While the rebuttal helped clarify some of the reviewers' questions, the reviewers shared a few concerns regarding the experimental evaluation, comparisons to SOTA, and the relationship of the proposed approach w.r.t. the relevant literature. Overall, the reviewers have a positive assessment of the paper and appreciated the technical insights to design the weighted QMIX algorithm.

Additional Feedback: After reading the other reviews and the authors' rebuttal, I have increased my score to 7. The additional experiments are greatly appreciated, but I think more details should be provided for them: e.g. I feel that if the policy has all the necessary information and is trained with a model-free approach, it should be able to obtain comparable or better result than a model-based approach (with much worse sample complexity, of course). That being said, the comparison between model-based and model-free methods is not the focus of the work and the experiments with model-based baselines do show good results. I think the paper presents an interesting idea for improving the robustness of model-based rl method to different reward functions. I have a few questions regarding the details of the algorithm, as listed below.

The reviewers were split on this paper, with two advocating for (weak) rejects, and two for (strong) accepts. The primary contention here relates to the baselines. However, partially because additional baselines were added in the rebuttal, and partially because of the novel contribution, this paper should be accepted.

Weaknesses: 1) The task of enhancing the target coverage in Directional Sensor Networks (DSNs) is important and challenging. However, as far as I am concerned, it is not a standard benchmark environment for studying multi-agent reinforcement learning. The proposed method/model design targets at a specific problem, limiting its significance. There already exist some popular environments for multi-agent cooperation. If experiments are conducted on these standard benchmarks, the significance of this work for the machine learning (ML) or reinforcement learning (RL) community can be improved.

Weaknesses: - In multi-agent reinforcement learning research, Schelling diagrams are normally plotted as a function of the number of *other cooperators* (besides the focal agent making the decision), i.e. C - 1, rather than the total number of cooperators, C, as was done here. Either way is certainly correct in principle, Schelling said as much in the original 1973 paper. However, there are several reasons why the C - 1 parameterization is convenient. For instance, it lets you read off game theoretic properties from the diagram more easily. To see if cooperation or defection is favored for a particular number of other cooperators, you simply compare a point on the R_c curve to the point on the R_d curve that is right above it.

The paper is modelling MARL problems under the angle of social dilemma, and tries to tackle the problem of common-pool resource management. The authors do not introduce a novel method, instead this paper is a comparison of a wide range of existing relevant algorithms on a single problem (water management). The experiments are well motivated and in general, the paper is very clear. My understanding is that although the paper focuses on a water management, it is aimed as a more general survey of the quality of current MARL algorithms on common-pool resource management. The authors argue that water management is a good example to study because it is critical and life-supporting, and safety issues are very relevant.

Summary and Contributions: This paper studies reinforcement learning (RL) problems on large state and action spaces that are endowed with a metric. They key assumption is that the optimal state-action value function, Q*, is Lipschitz smooth with respect to that metric. The setting is that of an episodic, H-stage Markov decision process, in which the learner must choose an action for each stage of each episode while achieving low regret against an optimal policy. Previously, an algorithm was proposed for this problem based on learning the Q* function on an adaptive discretization of the state-action space that becomes steadily finer on important regions of the space. The regret of this algorithm was thought to depend on the packing number of the state-action space.