MAGMA -- Multimodal Augmentation of Generative Models through Adapter-based Finetuning