HowtoLearnaUsefulCritic?Model-based Action-Gradient-EstimatorPolicyOptimization

However, instead of gradients, the critic is, typically, only trained to accurately predict expected returns, which, on their own, are useless for policy optimization.