Machine learning algorithms have become prevalent tools for decision-making in socially sensitive situations, such as determining credit-score ratings or predicting recidivism during parole. It has been recognized that algorithms are capable of learning societal biases, for example with respect to race (Larson, Mattu, Kirchner, and Angwin 2016) or gender (Lambrecht and Tucker 2019; Blau and Kahn 2003), and this realization seeded an important debate in the machine learning community about fairness of algorithms and their impact on decision-making. In order to define and measure discrimination, existing intuitive notions have been statistically formalized, thereby providing fairness metrics. For example, demographic parity (Darlington 1971) requires the protected attribute A (gender/race/religion etc.) to be independent of a constructed classifier or regressor Ŷ, written as Ŷ A. Another notion, termed equality of odds (Hardt, Price, Srebro et al. 2016), requires equal false positive and false negative rates of classifier Ŷ between different groups (females and males for example), written as Ŷ A Y. To this day, various different notions of fairness exist, which are sometimes incompatible (Corbett-Davies and Goel 2018), meaning not of all of them can be achieved for a predictor Ŷ simultaneously. There is still no consensus on which notion of fairness is the correct one. The discussion on algorithmic fairness is, however, not restricted to the machine learning domain. There are many legal and philosophical aspects that have arisen. For example, the legal distinction between disparate impact and disparate treatment (McGinley 2011) is important for assessing fairness from a judicial point of view.