Review for NeurIPS paper: Can I Trust My Fairness Metric? Assessing Fairness with Unlabeled Data and Bayesian Inference

Additional Feedback: This paper proposes to use Bayesian estimates of fairness metrics. It combines this with Bayesian calibration models (one for each protected attribute value in this particular case) in order to use unlabelled data. In light of existing work (Foulds et al 2019) on Bayesian modelling of fairness, the contribution is rather minor and is limited to the case where we have unlabelled data. The approach the authors use, as it is based on calibration, seems limited to rather specific notions of fairness where Bayesian calibration can be usefully applied. Although in l.64 the definition of calibration is correct, in l. 105-107 you write that s_j P_M(y_j 1 s_j) . Since j is a specific example, there should not be any randomness here.