Robustifying Language Models with Test-Time Adaptation