Review for NeurIPS paper: Achieving Equalized Odds by Resampling Sensitive Attributes

Weaknesses: Intuitively randomising sensitive feature should lead to fairer results, however, fairness though unawareness poses a risk of unfairness by proxy as there are ways of predicting protected characteristic features from other features [Ruggieri et all, 2010, Adler et al 2016]. Also a continuous analog of fairness through unawareness [Dwark et al 2012] has been proposed via counterfactual fairness [Matt J. Kusner, et al, Counterfactual fairness, 2017]. In the counterfactual fairness, one has to estimate a dependency structure over the features, i.e. a causal graph, in order to create a counterfactual example when changing/flipping observational sensitive feature. To properly evaluate the contribution of the proposed approach, it has to be compared --methodologically and empirically -- not only to fairness through unawareness, but also to counterfactual fairness approaches. Another concern is that very little information is dedicate to the analysis how to estimate p(A Y).