Debiasing Vison-Language Models with Text-Only Training