Adversarial training with restricted data manipulation