Learning with Multiplicative Perturbations

Adversarial Training (AT) and Virtual Adversarial Training (VAT) are the regularization techniques that train Deep Neural Networks (DNNs) with adversarial examples generated by adding small but worst-case perturbations to input examples. In this paper, we propose xAT and xVAT, new adversarial training algorithms, that generate multiplicative perturbations to input examples for robust training of DNNs. Such perturbations are much more perceptible and interpretable than their additive counterparts exploited by AT and VAT. Furthermore, the multiplicative perturbations can be generated transductively or inductively while the standard AT and VAT only support a transductive implementation. W e conduct a series of experiments that analyze the behavior of the multiplicative perturbations and demonstrate that xAT and xVAT match or outperform state-of-the-art classification accuracies across multiple established benchmarks while being about 30% faster than their additive counterparts. Furthermore, the resulting DNNs also demonstrate distinct weight distributions. 1. Introduction Over the past few years, Deep Neural Networks (DNNs) have achieved state-of-the-art performance on a wide range of learning tasks. However, the success of DNNs has a high reliance on large sets of labeled examples; when trained on small datasets, DNNs plague to overfitting if not regularized properly. For many practical applications, collecting a large amount of labeled examples is very expensive and/or time-consuming. To address this issue, researchers have investigated a host of techniques, such as Dropout [24], A T [4, 25], V A T [14], and Mixup [29], to regularize the training of DNNs.