Improving the Efficiency of Visually Augmented Language Models