MAGIC-VFM: Meta-learning Adaptation for Ground Interaction Control with Visual Foundation Models