On the Generation and Removal of Speaker Adversarial Perturbation for Voice-Privacy Protection