Effective Large Language Model Adaptation for Improved Grounding