Fair Learning with Private Demographic Data

As algorithmic systems driven by machine learning start to play an increasingly important role in society, concerns arise over their compliance with laws, regulations and societal norms. In particular, machine learning systems have been found to be discriminating against certain demographic groups in applications of criminal assessment, lending and facial recognition (Barocas et al. (2019)). To ensure non-discrimination in learning tasks, knowledge of the sensitive attributes is essential, however, laws and regulation often prohibit access and use of this sensitive data. As an example, credit card companies do not have the right to ask about an individual's race when applying for credit, while at the same time they have to prove that their decisions are non-discriminatory (Commission (2013); Chen et al. (2019)). Apple Card, a credit card offered by Apple and Goldman Sachs, was recently accused of being discriminatory Vigdor (2019). Married couples rushed to Twitter to report that there were significant differences in the credit limit given individually to each of them even though they had shared finances and similar income levels. Supposing Apple was trying to make sure its learned model was non discriminatory, it would have been forced to use proxies for gender and recent work has shown that proxies can be problematic Kallus et al. (2019). We are then faced with what seems to be two opposing societal notions to satisfy: we want our system to be non-discriminatory while maintaining the privacy of our sensitive attributes. Note that even if the features that our model uses are independent of the sensitive attributes, it is not enough to guarantee notions of non-discrimination that further condition on the truth, e.g.