On Reality and the Limits of Language Data: Aligning LLMs with Human Norms