A Sober Look at the Robustness of CLIPs to Spurious Features