Critic Algorithms using Cooperative Networks

While most reinforcement learning algorithms aim at minimizing the Mean Squared Bellman Error, in function approximation it makes more sense to track the Projected Bellman Error. This is because with function approximation the true optimal of the Bellman Equation might not be representable by the function class. An example would be the true solution not being within the range space of the design matrix when using linear architectures. In such a scenario, one looks at the projected optimal solution onto the range space of the design matrix. This projected optimal solution is the fixed point solution of the Bellman Equation.