Gradient Boosting from scratch – ML Review – Medium

Although most of the Kaggle competition winners use stack/ensemble of various models, one particular model that is part of most of the ensembles is some variant of Gradient Boosting (GBM) algorithm. Take for an example the winner of latest Kaggle competition: Michael Jahrer's solution with representation learning in Safe Driver Prediction. His solution was a blend of 6 models. 1 LightGBM (a variant of GBM) and 5 Neural Nets. Although his success is attributed to the new semi-supervised learning that he invented for the structured data, but gradient boosting model has done the useful part too. Even though GBM is being used widely, many practitioners still treat it as complex black-box algorithm and just run the models using pre-built libraries.