A Experiments Supplement

Since most loss values falls within the range of [0.1, 10], we evaluate how the model accuracy and fairness change w.r.t. Figure 1 shows the change of fairness (equalized odds) under different cutoff value. A.2 Sensitivity of Validation Size We show the effect of validation size on accuracy and equalized odds in Fig.. As shown in the figures, when the validation size is larger than 10% of training size, the model's performance becomes stable in terms of accuracy and fairness. During validation, we freeze the contrastive encoder and train a downstream linear classifier g with parameter ω for classification task. Figure 4: Change of accuracy as validation size varies.