Accelerating Policy Gradient by Estimating Value Function from Prior Computation in Deep Reinforcement Learning

This paper investigates the use of prior computation to estimate the value function to improve sample efficiency in on-policy policy gradient methods in reinforcement learning. Our approach is to estimate the value function from prior computations, such as from the Q-network learned in DQN or the value function trained for different but related environments. In particular, we learn a new value function for the target task while combining it with a value estimate from the prior computation. Finally, the resulting value function is used as a baseline in the policy gradient method. This use of a baseline has the theoretical property of reducing variance in gradient computation and thus improving sample efficiency. The experiments show the successful use of prior value estimates in various settings and improved sample efficiency in several tasks. Reusing past computations has brought tremendous success in many machine learning fields, including computer vision (e.g., pre-trained models on ResNet) and natural language processing (e.g., large language models like GPT-3). These techniques allow for the creation of practical tools in real-world problems where task-specific data is scarce. Reinforcement learning (RL) provides algorithmic advantages and works on dynamically changing datasets in many cases in contrast to the static datasets used in supervised learning. However, this comes at the cost of requiring many samples. Despite this, RL has shown tremendous breakthroughs in recent years, particularly when large amounts of data (e.g., in simulations or games) are available (Silver et al., 2016; Vinyals et al., 2019). However, using RL in many real-world tasks is challenging, partly due to the scarcity of data (i.e., environment interaction) that most RL algorithms require. Off-policy algorithms provide a mechanism for reusing data.