On Learning and Testing of Counterfactual Fairness through Data Preprocessing

The rapid popularization of machine learning methods and the growing availability of personal data have enabled decision-makers from various fields such as graduate admission (Waters and Miikkulainen, 2014), hiring (Ajunwa et al., 2016), credit scoring (Thomas, 2009), and criminal justice (Brennan et al., 2009) to make data-driven decisions efficiently. However, the community and the authorities have also raised concern that these automatically learned decisions may inherit the historical bias and discrimination from the training data and would cause serious ethical problems when used in practice (Nature Editorial, 2016; Angwin and Larson, 2016; Dwoskin, 2015; Executive Office of the President et al., 2016). Consider a training dataset D consisting of sensitive attributes S such as gender and race, non-sensitive attributes A and decisions Y. If the historical decisions Y are not fair across the sensitive groups, a powerful machine learning algorithm will capture this pattern of bias and yield learned decisions Ŷ that mimic the preference of the historical decisionmaker, and it is often the case that the more discriminative an algorithm is, the more discriminatory it might be. While researchers agree that methods should be developed to learn fair decisions, opinions vary on the quantitative definition of fairness. In general, researchers use either the observational or counterfactual approaches to formalize the concept of fairness. The observational approaches often describe fairness with metrics of the observable data and predicted decisions (Hardt et al., 2016; Chouldechova, 2017; Yeom and Tschantz, 2018). For example, Demographic Parity (DP) or Group Fairness (Zemel et al., 2013; Khademi et al., 2019) considers the learned decision Ŷ to be fair if it has the same distribution for different sensitive groups, i.e., P (Ŷ |S = s) = P (Ŷ |S = s