4 Value function methods

This is the fourth of a series of articles in which I summarize the lectures from CS285 held by Professor Sergey Levine, to whom all credit goes. All images are taken from his lectures. This article I wrote is an introduction to deep reinforcement learning. Actor-critic algorithms build on the policy gradient framework that we discussed in this article. On top, they're also augmented with learned value functions and Q-functions. What if we just learn a value function and then try to use that value function to figure out how to act? The intuition for why this should be possible is that the value function tells us which states are better than others, so if we simply select actions that go into the better states maybe we don't need an explicit policy neural network anymore. Here's the the way to make this intuition a bit more formal.