On the Difficulty of Defending Contrastive Learning against Backdoor Attacks