Fair for a few: Improving Fairness in Doubly Imbalanced Datasets