Nonparametric Online Learning Using Lipschitz Regularized Deep Neural Networks

In recent years, deep neural networks have been applied to many off-line machine learning tasks. Despite their state-of-of-the-art performance, the theory behind their generalization abilities is still not complete. When turning to the online domain even much less is known and understood both from the practical use and the theoretical side. Thus, the main focus of this paper is exploring the theoretical guarantees of deep neural networks in online learning under general stochastic processes. In the traditional online learning setting, and in particular in sequential prediction under uncertainty, the learner is evaluated by a loss function that is not entirely known at each iteration [8]. In this work, we study online prediction focusing on the challenging case where the unknown underlying process is stationary and ergodic, thus allowing observations to depend on each other arbitrarily. Many papers before have considered online learning under stationary and ergodic sources and in various application domains. For example, in online portfolio selection, [19, 16, 17, 42, 26] proposed nonparametric online strategies that guarantee, under mild conditions, convergence to the best possible outcome. 1