Mitigating Algorithmic Bias with Limited Annotations