LASPA: Language Agnostic Speaker Disentanglement with Prefix-Tuned Cross-Attention