Review for NeurIPS paper: The Mean-Squared Error of Double Q-Learning

Summary and Contributions: The authors provide a theoretical analysis of Double Q-learning, specifically the asymptotic MSE in the case of linear function approximation. Their analysis suggests a way to select the step size and final output to obtain faster initial convergence while maintaining the same asymptotic result, which they then verify empirically in different experiments. The asymptotic analysis relies on a stochastic approximation result, which they apply in a regime where the policy to select the next step Q evaluation is already optimal (see Eq 4-5). They can then transform the q-learning and Double Q-learning updates in a similar form (eq 10 and 12) which allows the result to apply and the asymptotic MSE to be compared between the approaches. The main concern here is that by assuming that the max action is fixed to being the optimal policy, we are already in a regime where overestimation bias isn't present in Q-learning due to the usual reasons (taking expectation after the max), and so Double Q-learning has little to offer and all to lose (since it has split updates and parameters). The asymptotic analysis doesn't seem like the right way to understand the potential benefits of Double Q-learning in this respect.