Class-attribute Priors: Adapting Optimization to Heterogeneity and Fairness Objective