Mind the Graph When Balancing Data for Fairness or Robustness