The reviewers appreciated the analysis of the learned attention from the model, and recommend accepting. I would ask that the authors add some of the experiments suggested by R4 in their "key questions" section. If there is not enough space, please add these to the supplemental.

The results are new and important to the field, and the analysis in this setting seems nontrivial. In addition, the paper also develops a new variant of TDC under a blockwise diminishing stepsize, and proves it asymptotically convergent with an arbitrarily small training error at linear convergence rate. Extensive experiments demonstrate that the new TDC variant can converge as fast as vanilla TDC with constant stepsize, and at the same time it enjoys comparable accuracy as TDC with diminishing stepsize. Overall, the paper has both analytical as well as practical value. However, the following issues need to be addressed. Markovian sample path has been studied in e.g., [30,34].

Four knowledgeable referees reviewed this paper. After conducting initial reviews, reading the authors' rebuttal (which resolved some concerns, but not the core concerns of two of the reviewers), and discussing the paper, the reviewers did not agree on an outcome. Two of the reviewers came to the conclusion that this is a ground-breaking paper (simple and elegant). The other two reviewers were perhaps somewhat intrigued, but did not feel the paper was yet ready for publication. For example, during the discussion phase, R4 (a very accomplished and well-respected research in the field) made very valid points about the papers weaknesses: "So all this leads me to suggest that there needs to be a better context, more related work and a better way to situate the paper in related arenas, e.g., provide some sort of a framework to back up the findings. I understand the issue of limited space, but given the amount of literature in this area, I feel that the paper doesnt do a good enough job explaining its findings in context."

Summary and Contributions: Based on rebuttal and discussion: Upon reading all reviews, I recognize that we agree the article is well presented, and I stand by the concerns I raised. Note that I primarily criticized the absence of some relevant context in the original submission (which the authors admit in their rebuttal), rather than the contribution itself (albeit it may be smaller than proclaimed). Their refutation of it being a planning setting is fair. While I maintain that it is a self-play setting, this is implied by CTDE and thus not necessary to state again. A stale flavor remains from overselling their contribution's novelty in the introduction [L36-45].

Originally, there was some disagreement between reviewers on this paper, but after rebuttal and careful discussion between reviewers and AC, all agree that the paper is interesting and has merit and could be proposed for acceptance as poster. One critical reviewer now recognises that the predictability idea is neat and the concern about positioning of the work has been largely clarified. Reviewers agree there is a contribution to joint exploration in MAS, which is one of the bottlenecks that deserve being addressed and discussed.

The paper proposes to use episodic backwards updates to improve data efficiency in RL tasks, furthermore they introduce a soft relaxation of this in order to combat the overestimation that typically comes from using backwards updates when using Neural Network models. Overall the paper is very clearly written. My main concerns with the paper are in the experimental details as well as in the literature review, also when taking into account the existing literature the novelty of the work is quite limited. The idea of using backwards updates is quite old and goes back to at least the 1993 paper "Prioritized Sweeping" by Moore and Atkeson, which in fact demonstrates a method that is very similar to what the authors propose and which the authors fail to cite. Furthermore recently there were quite a few papers operating in a similar space of ideas using a backward view in ways similar to the authors, e.g.: Fast deep reinforcement learning using online adjustments from the past, https://arxiv.org/abs/1810.08163

All reviewers recommend accepting the paper. The authors response did address most of the reviewers' concerns. While the AC recommends accepting the paper, the AC encourages the authors to consider the comments of reviewer 1. Only changing the backup mechanism keeping all other hyper parameters fixed as in the Nature DQN model is indeed a good experimental setup. However, the optimal operation mode for different models might be different (even when sharing architectures and training protocols): for instance we could'afford' a larger learning rate if we have a better back-up mechanism.

Originality: The proposed approach is a novel combination of well-known techniques such as RL and GAN for recommendation. Related work has been adequately cited. It is clear how the proposed approach differs from the existing literature. Quality: The approach appears to be technically sound. The theoretical analysis and the experiments support the claims.

The reviewers overall felt positively about this paper, though scores were somewhat marginal. The reviews, while not glowing, overall lean toward acceptance of the paper: the reviewers feel the work is technically sound, the method is practical and effective, related work is good, and the experiments are convincing. There are some more tentative comments regarding the novelty/originality, being mostly a combination of existing techniques (R3), however this issue seems not to be a dealbreaker, and the difference compared to existing work is clear. There are a few clarifying points that seem to be addressed in the rebuttal.

Summary and Contributions: The goal of the paper is to prove (asymptotic) convergence of asynchronous Q-learning and other variants of Q-learning within a "simplified" common framework. This is done using the (commonly adopted) ODE method and the results in Borkar and Meyn, but crucially, modeling the ODEs as switched linear systems with state-dependent switching policies. This allows the authors to unify the treatment across different instances of Q-learning, and in particular establish convergence of some interesting variants of Q-learning ("averaging Q-learning" which essentially amounts to target tracking with a Polyak average; Q-learning with a linear state-space function approximator). The established theory of switching systems is a key tool in the paper, and its usage in the context of an analysis of RL algorithms may be of interest in its own right. The novelty of the paper stems principally from the use of switching systems as an analysis tool; the authors provide a condition for global asymptotic convergence of Q-learning with linear function approximators that appears to be weaker than in previously published work.