These techniques mainly fall in one of two categories: value-based methods [e.g.,

These techniques mainly fall in one of two categories: value-based methods [e.g.,

Clarity: The clarity of this paper is very poor. Value-based is used to mean different things at different points in the paper. This makes the paper very confusing. There is value based to mean value based methods such as Q-learning or SARSA (although no reference to these algorithms or anything like them is made) in the introduction and then value-based to refer to the policy gradient theorem presented in [3]. As discussed in the correctness section, much of the paper is ambiguous and seems wrong or confused in its claims.

We cast policy gradient methods as the repeated application of two operators: a policy improvement operator $\mathcal{I}$, which maps any policy $\pi$ to a better one $\mathcal{I}\pi$, and a projection operator $\mathcal{P}$, which finds the best approximation of $\mathcal{I}\pi$ in the set of realizable policies. We use this framework to introduce operator-based versions of traditional policy gradient methods such as REINFORCE and PPO, which leads to a better understanding of their original counterparts. We also use the understanding we develop of the role of $\mathcal{I}$ and $\mathcal{P}$ to propose a new global lower bound of the expected return. This new perspective allows us to further bridge the gap between policy-based and value-based methods, showing how REINFORCE and the Bellman optimality operator, for example, can be seen as two sides of the same coin.