Guided Learning of Nonconvex Models through Successive Functional Gradient Optimization

Our original motivation was functional gradient learning of additive models in gradient boosting (Friedman, 2001). In This paper presents a framework of successive our framework, essentially, training proceeds with repeating functional gradient optimization for training nonconvex a local search, which limits the searched parameter space to models such as neural networks, where the functional neighborhood of the current parameter at each training is driven by mirror descent in a function iteration, instead of searching the entire space at once as space. We provide a theoretical analysis and empirical the standard method does. This is analogous to ε-boosting study of the training method derived from where the use of a very small step-size (for successively this framework. It is shown that the method leads expanding the ensemble of weak functions) is known to to better performance than that of standard training achieve better generalization (Friedman, 2001).