Fast Distributionally Robust Learning with Variance Reduced Min-Max Optimization

With machine learning systems increasingly being deployed in real-world settings, there is an urgent need for machine learning approaches that can adapt to--or are robust to--changes in the environment. Despite this, the dominant paradigm for supervised learning [Hastie et al., 2009] remains that of empirical risk minimization (ERM) [Vapnik, 2013], wherein a model is trained by minimizing a loss over a fixed set of training data. A key assumption underlying this approach is that the training data is from the same distribution as the test data--i.e., the distribution of the data does not change between training time and deployment. Such assumptions are well known to rarely hold in practice. Indeed, the distribution may change due to sample selection bias, nonstationarity in the environment [Quionero-Candela et al., 2009], or even adversarial perturbations [Szegedy et al., 2013, Madry et al., 2018], leaving machine learning models trained through ERM particularly susceptible to adversarial attacks [Szegedy et al., 2013, Carlini and Wagner, 2017] or to degraded performance from distribution shifts. Distributionally robust supervised learning (DRSL) seeks to address this issue by explicitly optimizing for solutions that are are robust to adversarial distribution shifts.