Adversarial Speaker Disentanglement Using Unannotated External Data for Self-supervised Representation Based Voice Conversion