They also raise some great points and criticisms, which we respond to below. All reviewers want more experiments, though R1 "understands that the focus of the paper B, E, the nontrivial proof of Prop.8, etc.) and accompanied them with We can show how CI changes with W, as suggested by R1. We can run the experiments on more domains (probably in the appendix, as we want the main text to focus on theory). We have tried many methods to stablize training and documented the working tricks in F.2. We are considering changing "confidence interval" to "value interval" to avoid this issue.

In many real-world applications of reinforcement learning (RL), interactions with the environment are limited due to cost or feasibility. This presents a challenge to traditional RL algorithms since the max-return objective involves an expectation over on-policy samples. We introduce a new formulation of max-return optimization that allows the problem to be re-expressed by an expectation over an arbitrary behavior-agnostic and off-policy data distribution. We first derive this result by considering a regularized version of the dual max-return objective before extending our findings to unregularized objectives through the use of a Lagrangian formulation of the linear programming characterization of Q-values. We show that, if auxiliary dual variables of the objective are optimized, then the gradient of the off-policy objective is exactly the on-policy policy gradient, without any use of importance weighting. In addition to revealing the appealing theoretical properties of this approach, we also show that it delivers good practical performance.