Reviews: Regularized Gradient Boosting

Gradient boosting (GB) has been extensively studied in the past, both theoretically and experimentally. Recently, with the advent of big data, several accelerated versions of vanilla GB have been proposed (in particular the well known XGBoost), and while the experimental evaluations of these methods have been abundant, the same cannot be said for the theoretical analysis. In this paper, the authors tackle this important problem. The main contribution of this paper consists in casting the various accelerated GB methods in a regularized gradient boosting setting. Indeed, by introducing a regularization term in the usual minimization objective of GB, it is possible to recover most, if not all, of the various accelerated gradient boosting approaches (XGboost included), while at the same time opening up several interesting and exciting possibilities for deriving new/novel acceleration strategies.