Context Matters: Data-Efficient Augmentation of Large Language Models for Scientific Applications