Subpopulation Data Poisoning Attacks

Machine learning (ML) and deep learning systems are being deployed in sensitive applications, but they can fail in multiple ways, impacting the confidentiality, integrity and availability of user data [33]. To date, evasion attacks or inference-time attacks have been studied extensively in image classification [60, 20, 7], speech recognition [9, 49], and cyber security [58, 68, 30]. Still, among the threats machine learning and deep learning systems are vulnerable to, poisoning attacks at training time has recently surfaced as the threat perceived as most potentially dangerous to companies' ML infrastructures [52]. The threat of poisoning attacks becomes even more severe as modern deep learning systems rely on large, diverse datasets, and their size makes it difficult to guarantee the trustworthiness of the training data. In existing poisoning attacks, adversaries can insert a set of corrupted, poisoned data at training time to induce a specific outcome in classification at inference time. Existing poisoning attacks can be classified into: availability attacks [4, 67, 25] in which the overall accuracy of the model is degraded; targeted attacks [27, 51, 59] in which specific test instances are targeted for misclassification; and backdoor attacks [21] in which a backdoor pattern added to testing points induces misclassification. Poisoning attacks range in the amount of knowledge the attacker has about the ML system, with white-box attacks assuming full knowledge, black-box attacks assuming minimal knowledge about the ML system, and gray-box attacks assuming partial knowledge, such as the feature representation or model architecture [59]. The threat models for poisoning attacks defined in the literature rely on strong assumptions on the adversarial capabilities. In both poisoning availability attacks [4, 67, 25] and backdoor attacks [21] the adversary needs to control a relatively large fraction of the training data (e.g., 10% or 20%) to influence the model at inference time.