When Worlds Collide: Integrating Different Counterfactual Assumptions in Fairness