Parameter-efficient Adaptation of Multilingual Multimodal Models for Low-resource ASR