Deep Policy Gradient Methods Without Batch Updates, Target Networks, or Replay Buffers

We show that these methods fail catastrophically when limited to small replay buffers or during incremental learning, where updates only use the most recent sample without batch updates or a replay buffer.