Your Policy Regularizer is Secretly an Adversary