Should We Attend More or Less? Modulating Attention for Fairness