The successor representation, gamma-models, and infinite-horizon prediction

Reinforcement learning algorithms are frequently categorized by whether they predict future states at any point in their decision-making process. Those that do are called model-based, and those that do not are dubbed model-free. This classification is so common that we mostly take it for granted these days; I am guilty of using it myself. However, this distinction is not as clear-cut as it may initially seem. In this post, I will talk about an alternative view that emphases the mechanism of prediction instead of the content of prediction.