Mapping from Meaning: Addressing the Miscalibration of Prompt-Sensitive Language Models