To be Robust and to be Fair: Aligning Fairness with Robustness