PADA: Pruning Assisted Domain Adaptation for Self-Supervised Speech Representations