Revisiting the Adversarial Robustness-Accuracy Tradeoff in Robot Learning