Review for NeurIPS paper: Leverage the Average: an Analysis of KL Regularization in Reinforcement Learning

TRPO, MPO, SAC, soft Q-learning, softmax DQN, DPP, etc) under one mirror descent framework and provides proofs for KL regularized value iteration. The paper shows that using KL regularization implicitly averages the estimates of the Q function, and using this result it shows a linear dependence of the approximation error of Q on the time horizon, whereas in many previous works with similar assumptions it was quadratic. This is a significant result. In addition, KL regularization ensures convergence in the case of independent and centered errors, which is not the case for standard approximate dynamic programming. The paper also examines how KL regularization interacts with entropy regularization, and presents empirical findings suggesting that KL regularization alone might be sufficient and better then entropy regularization, encouraging a lot of exploration in the beginning and less so as the policy deviates from being uniform.