Fairness without Demographics through Learning Graph of Gradients