Agree to Disagree: Diversity through Disagreement for Better Transferability