This paper proposes a distributionally robust approach to logistic regression. We use the Wasserstein distance to construct a ball in the space of probability distributions centered at the uniform distribution on the training samples. If the radius of this ball is chosen judiciously, we can guarantee that it contains the unknown data-generating distribution with high confidence. We then formulate a distribution-ally robust logistic regression model that minimizes a worst-case expected logloss function, where the worst case is taken over all distributions in the Wasserstein ball. We prove that this optimization problem admits a tractable reformulation and encapsulates the classical as well as the popular regularized logistic regression problems as special cases. We further propose a distributionally robust approach based on Wasserstein balls to compute upper and lower confidence bounds on the misclassification probability of the resulting classifier. These bounds are given by the optimal values of two highly tractable linear programs.

We consider Empirical Risk Minimization (ERM) in the context of stochastic optimization with exp-concave and smooth losses--a general optimization framework that captures several important learning problems including linear and logistic regression, learning SVMs with the squared hinge-loss, portfolio selection and more. In this setting, we establish the first evidence that ERM is able to attain fast generalization rates, and show that the expected loss of the ERM solution in d dimensions converges to the optimal expected loss in a rate of d/n. This rate matches existing lower bounds up to constants and improves by a log n factor upon the state-of-the-art, which is only known to be attained by an online-to-batch conversion of computationally expensive online algorithms.

Recently there has been substantial interest in spectral methods for learning dynamical systems. These methods are popular since they often offer a good tradeoff between computational and statistical efficiency. Unfortunately, they can be difficult to use and extend in practice: e.g., they can make it difficult to incorporate prior information such as sparsity or structure.

Trace regression models have received considerable attent ion in the context of matrix completion, quantum state tomography, and compress ed sensing. Estimation of the underlying matrix from regularization-based approaches promoting low-rankedness, notably nuclear norm regularization, hav e enjoyed great popularity. In this paper, we argue that such regularization may no l onger be necessary if the underlying matrix is symmetric positive semidefinite ( spd) and the design satisfies certain conditions. In this situation, simple lea st squares estimation subject to an spd constraint may perform as well as regularization-based app roaches with a proper choice of regularization parameter, which ent ails knowledge of the noise level and/or tuning. By contrast, constrained least s quares estimation comes without any tuning parameter and may hence be preferred due t o its simplicity.

Learning mixtures of linear regressions is a natural generalization of the basic linear regression problem.

Neural Information Processing Systems http://nips.cc/

However, deep learning as a non-linear regression tool based on statistics mostly suffers from the insufficient or non-existing training data, which is the usual case and should be overcome for autonomous learning systems.

Neural Information Processing Systems http://nips.cc/

Neural Information Processing Systems http://nips.cc/