Discrimination in machine learning algorithms

A human may discriminate either because of irrational prejudice induced by ignorance and stereotypes or based on statistical generalization: lacking specific information on an individual, he is assigned the characteristics prevalent in the sensitive attribute category he belongs to. For example, in the United States, lacking information on education, a black person may be assumed to have relatively low level since this is the case in general for black people in the country) [7]. When a statistical or machine learning algorithm is used in the decision process, its behavior concerning discrimination depends on the information it is given. In particular, if the sensitive attribute is available to the algorithm (i.e., it is included in the learning data and can be used for predictions), it may discriminate either because the data it is taught contain irrational prejudice (Figure 1(a)) or because the sensitive attribute is associated to an unobserved attribute that is relevant for the prediction of Y, the outcome of interest (Figure 1(b)).