The proximal policy optimization (PPO) algorithm stands as one of the most prosperous methods in the field of reinforcement learning (RL). Despite its success, the theoretical understanding of PPO remains deficient. Specifically, it is unclear whether PPO or its optimistic variants can effectively solve linear Markov decision processes (MDPs), which are arguably the simplest models in RL with function approximation.

Sutton et al., 2022], is elegantly captured using the options framework [Sutton et al., 1999].

Part of this work was done during Kang Xu's internship at Shanghai Artificial Intelligence Laboratory.

Exploration is essential in reinforcement learning, particularly in environments where external rewards are sparse. Here we focus on exploration with intrinsic rewards, where the agent transiently augments the external rewards with self-generated intrinsic rewards.

In general, research in distributional reinforcement learning has focused on the classical setting of a scalar reward function.