Temporal Difference Learning in Reinforcement Learning

As we saw for Monte Carlo methods, Prediction refers to the problem of estimating the values of states, a value of a state is an indication of how good is that state for an agent in the given environment, the higher the value of the state the better it is to be in that state. Monte Carlo and Temporal Difference Learning are similar in the sense that they both use real-world experience to evaluate a given policy, however, Monte Carlo methods wait until the return following the visit is known which is after the episode ends is available to update the value of the state, whereas TD methods update the state value in the next time step, at the next time step t 1 they immediately form a target and make a useful update using the observed reward. Updating the state value just after one time step is called one-step TD or TD(0), which is a special case of the TD(lambda) and n-step TD methods which are beyond the scope of our discussion, however, the principles we explore here can be extended to those methods without much complexity. As we can observe that TD(0) bases its update on an existing estimate of the next state value, because of which it is known to be a bootstrapping method. Temporal Difference methods are said to combine the sampling of Monte Carlo with the bootstrapping of DP, that is because in Monte Carlo methods target is an estimate because we do not know the actual expected value rather use a sample return from that particular episode, and in DP that target is an estimate because the value of the next state is not known instead the current estimate is used, and in TD the target is an estimate because of both the reasons, it samples the expected values and it uses the current estimate instead of the true state value.