Random Function Descent

Classical worst-case optimization theory neither explains the success of optimization in machine learning, nor does it help with step size selection. In this paper we demonstrate the viability and advantages of replacing the classical'convex function' framework with a'random function' framework. With complexity \mathcal{O}(n 3d 3), where n is the number of steps and d the number of dimensions, Bayesian optimization with gradients has not been viable in large dimension so far. Specifically, we use a'stochastic Taylor approximation' to rediscover gradient descent, which is scalable in high dimension due to \mathcal{O}(nd) complexity. This rediscovery yields a specific step size schedule we call Random Function Descent (RFD).