Bias and ethics in artificial intelligence have captured the attention of the public and some organizations following some high-profile examples of it at work. For instance, there has been work that has demonstrated bias against darker skinned and female individuals in face recognition technology and a secret AI recruiting tool at Amazon that showed bias against women, among many other examples. But when it comes to looking inside at our own houses -- or businesses -- we may not be very far along in prioritizing AI ethics or taking measures to mitigate bias in algorithms. According to a new report from FICO, a global analytics software firm, 65% of C-level analytics and data executives surveyed said that their company cannot explain how specific AI model decisions or predictions are made, and 73% have struggled to get broader executive support for prioritizing AI ethics and responsible AI practices. Only 20% actively monitor their models in production for fairness and ethics.

Fairness by decision-makers is believed to be auditable by third parties. In this study, we show that this is not always true. We consider the following scenario. Imagine a decision-maker who discloses a subset of his dataset with decisions to make his decisions auditable. If he is corrupt, and he deliberately selects a subset that looks fair even though the overall decision is unfair, can we identify this decision-maker's fraud? We answer this question negatively. We first propose a sampling method that produces a subset whose distribution is biased from the original (to pretend to be fair); however, its differentiation from uniform sampling is difficult. We call such a sampling method as stealthily biased sampling, which is formulated as a Wasserstein distance minimization problem, and is solved through a minimum-cost flow computation. We proved that the stealthily biased sampling minimizes an upper-bound of the indistinguishability. We conducted experiments to see that the stealthily biased sampling is, in fact, difficult to detect.