Promoting Counterfactual Robustness through Diversity