Alpha and Prejudice: Improving $\alpha$-sized Worst-case Fairness via Intrinsic Reweighting