Review for NeurIPS paper: Leverage the Average: an Analysis of KL Regularization in Reinforcement Learning

Strengths: The paper offers two novel theoretical contributions. The first contribution is the introduction of the novel DA-MPI framework, which the authors prove is equivalent to the MD-MPI scheme (proposition 1). The second contribution is the derivation of two new performance bounds based on this framework, one for a purely KL-regularised objective and another for a KL and entropy regularised objective (theorems 1 and 2). These bounds have been missing from existing analysis and offer an important contribution to the NeurIPS community. The insight offered in the analysis of these bounds is valuable: we see that the first bound demonstrates that using KL-regularisation leads to a linear dependence on the horizon term, unlike the typical quadratic dependence for non regularised form.