Collapsed Language Models Promote Fairness