Efficient Domain Adaptation for Speech Foundation Models