Knowing When to Ask -- Bridging Large Language Models and Data