Review for NeurIPS paper: Achieving Equalized Odds by Resampling Sensitive Attributes

The introduction of randomization tests for the assessment of fairness is very useful, and the proposed method for encouraging fairness in an adversarial learning system is relevant and novel. The discussion phase showed that the paper would benefit in discussing this work in a broader context, beyond parity measures. First, to shortly describe the pros and cons of addressing fairness through these parity measures (also taking the caution words mentioned in the broader impact section). Second, this would allow to better contrast the randomization of the sensitive attribute that is carried out here with "interventions" on the sensitive features. In particular such as (i) a scheme where randomization would be simply used to mask the sensitive attribute, and (ii) a rudimentary assessment of counterfactual fairness that would be obtained by simply flipping the sensitive attribute.