General response

Rephrase any claims that seem too strong, add additional reference and discuss more connections to previous works. Paper too long We will reorganize our paper (see general response). The red lines in bars represent median rewards. We improve reward under attacks consistently across runs. Critic attack sometimes improves PPO performance (green lines of Figure 1).