On the Brittleness of CLIP Text Encoders